Won’t Get Fooled Again: Answering Questions with False Premises
Year: 2023

Abstract:
Pre-trained language models (PLMs) have shown unprecedented potential in various fields, especially as the backbones for questionanswering (QA) systems. However, they tend to be easily deceived by tricky questions such as “How many eyes does the sun have?”. Such frailties of PLMs often allude to the lack of knowledge within them. In this paper, we find that the PLMs already possess the knowledge required to rebut such questions, and the key is how to activate the knowledge. To systematize this observation, we investigate the PLMs’ responses to one kind of tricky questions, i.e., the false premises questions (FPQs). We annotate a FalseQA dataset containing 2365 human-written FPQs, with the corresponding explanations for the false premises and the revised true premise questions. Using FalseQA, we discover that PLMs are capable of discriminating FPQs by fine-tuning on moderate numbers (e.g., 256) of examples. PLMs also generate reasonable explanations for the false premise, which serve as rebuttals. Further replaying a few general questions during training allows PLMs to excel on FPQs and general questions simultaneously. Our work suggests that once the rebuttal ability is stimulated, knowledge inside the PLMs can be effectively utilized to handle FPQs, which incentivizes the research on PLM-based QA systems. The FalseQA dataset and code are available at https://github.com/thunlp/FalseQA.

heading: 1 Introduction
text: Recent advances in pre-trained language models (PLMs) (Lewis et al., 2020; Raffel et al., 2020; Brown et al., 2020; Roller et al., 2021; Han et al., 2021) have achieved significant performance gains for various types of tasks, even surpassing human levels on language ability benchmarks (Wang et al., 2018, 2019; Srivastava et al., 2022). The
∗ Corresponding author: Huadong Wang (huadw2012@163.com)
unprecedented ability of PLMs lays the foundation for various practical applications. For example, PLMs that exhibit general world knowledge and commonsense knowledge have the potential to serve as backbones for general-purpose questionanswering models (Tafjord and Clark, 2021; Guu et al., 2020).
However, these PLM-based question-answering models have an intriguing paradox. On the one hand, they achieve high performance on normal questions raised by humans. For example, UNIFIEDQA (Khashabi et al., 2020) achieves state-ofthe-art performance on many question-answering tasks. MACAW (Tafjord and Clark, 2021) can perform multi-angle question-answering and answer 75% of the question in the Challenge300 dataset (Tafjord and Clark, 2021) correctly. On the other hand, they are vulnerable to tricky questions (see Table 1). For example, MACAW answers one
1AllenAI’s Blog. 2Blog Giving GPT-3 a Turing Test.
5626
out of nine tricky questions correctly, while other models including GPT-3 (Brown et al., 2020) fail all of them (Tafjord and Clark, 2021). InstructGPT (Ouyang et al., 2022) also reports that it fails to identify instructions with false premises. These questions are easy to rebut for humans but pose an undeniable obstacle for PLMs3. The inability to rebut also results in the misalignment (Kenton et al., 2021) of language models to human expectations.
Without careful investigation, this paradox could easily lead to the conclusion that PLMs lack the world or commonsense knowledge to rebut these questions. Although it’s crucial for the PLMs to embed as much general knowledge as possible, we provide a pilot experiment to find out that the PLMs already possess the knowledge required for the tricky questions which they fail (see Section 3.2). As a consequence, we hypothesize that the knowledge in current PLMs is enough for handling a large portion of tricky questions. However, this knowledge is not activated.
To support our hypothesis, we take a close look at these tricky questions. Most of these tricky questions contain false premises. For example, in the question “How many eyes does the sun have?”, the questioner must presume that “the sun can have eyes” in order to make the query about the quantity meaningful. These questions are called False Premise Questions (FPQs). Such false premises always violate human knowledge or logic and rarely appear in the natural text, thus leading to an out-ofdistribution generalization gap for the PLMs.
Targeting to fill the gap between the natural text and FPQs, we present the first specialized dataset of FPQs, dubbed as FalseQA dataset. Specifically, we first systematically categorize the false premises to ensure the coverage of the dataset. Then we ask human annotators to manually compose the FPQs, as well as explanations for the false premises. The annotators are also asked to edit the false premise questions into true premise questions (TPQs) using minimal modification, with which the PLMs are less prone to learn shortcuts from the format of FPQs.
Based on FalseQA dataset, we first conduct systematic experiments on the PLMs’ discrimination
3Although most PLMs fail, we found ChatGPT (OpenAI, 2022) satisfactorily answers these questions. Their training data is manually written by annotators and continuously updated using user queries, which might contain such questions. However, their data is not public. Our work provides the same possibility for general PLMs, even the much smaller ones.
and rebuttal ability of FPQs. We reach three essential conclusions: (1) PLMs of different types and scales can distinguish the FPQs from TPQs, and scaling effect (Kaplan et al., 2020) also holds for FalseQA. (2) PLMs can give reasonable explanations for the false premises, which can serve as rebuttals. (3) The number of FPQ examples needed to activate the PLM’s rebuttal ability is moderate. For example, 256 FPQs can result in more than 70% accuracy for models larger than 1B. And for some larger PLMs, in-context learning with a few examples can also activate the ability. Then we consider the practical scenario where the models need to handle both FPQs and general questions. We demonstrate that a simple but effective data replay method can help mitigate the catastrophic forgetting of general questions, where the model discriminates 86.7% FPQs in FalseQA and only rebuts 1.4% general questions. These results lead to optimism that PLMs can be used as the backbones of a practical question-answering system that is robust to tricky questions.

heading: 2 Related Work
text: Three groups of research are related to our work: direct question answering dataset, question unanswerability, and question premise verification.
Direct Question Answering Dataset. For the most practical scenario of the question-answering system, the candidate answers are absent. Therefore, direct question answering (DQA), as a counterpart to extractive QA (Rajpurkar et al., 2016) or multiple-choice QA (Lai et al., 2017) has received increased attention. Natural Questions (Kwiatkowski et al., 2019) collects the queries sent to the google search engine. ARC-DA (Bhakthavatsalam et al., 2021) proposes modifying a reasoning-based multiple choice QA into DQA format. Tafjord and Clark (2021) manually compose Challenge300 dataset which is still challenging to powerful models such as GPT-3 and MACAW. Our dataset can be seen as a direct question-answering dataset with explanations. However, the question distribution is radically different from the questions in natural corpora, serving as an adversarial scenario for DQA models.
Question Unanswerability. Tricky questions are unanswerable questions. Previous works (Raina and Gales, 2022; Rajpurkar et al., 2018; Asai and Choi, 2021; Davis, 2020) confirm the existence of unanswerable questions in existing benchmarks, in-
cluding SQuAD (Rajpurkar et al., 2016), Natural Questions (Kwiatkowski et al., 2019), VQA (Antol et al., 2015), etc. Most unanswerable questions in these benchmarks are due to missing information in the context provided to the questions. However, FalseQA contains questions that are out of natural text distribution, and are unanswerable due to misleading false premises.
Question Premise Verification. Answering FPQs has been studied before the deep learning era (Kaplan, 1978). In recent PLM-based questionanswering research, relevant efforts use external knowledge to verify the correctness of the question premise. For example, Kim et al. (2021) studies the FPQs in Natural Questions (Kwiatkowski et al., 2019). A concurrent work (Min et al., 2022) further gathers the 8400 Reddit questions and annotated the false premises among 25% of them. The correctness of the premises in their datasets requires expert knowledge or context to determine. Therefore, they use retrieval-augmented language models (Krishna et al., 2021) or external knowledge base to provide information for the premise classification, and both reach the conclusion that discovering and explaining those prepositions that require expert knowledge is challenging. However, it remains elusive whether PLMs without external assistance can discover and rebut the tricky questions that require only general knowledge and are straightforward for humans. We propose the first manually written dataset for FPQs and support our hypothesis through experiments that the inability of PLMs for FPQs can be mitigated when giving them examples.

heading: 3 Preliminaries
text: In this section, we introduce the definition of FPQ and the pilot experiment on PLMs about FPQs.

heading: 3.1 False Premise Questions
text: When questioning, humans usually assume that some facts are shared and endorsed by the questioner and the answerer. Such facts are the premises of the question. For example, in the question “How many eyes does the sun have?”, the target of the question is the number of eyes, which assumes the correctness of the fact “The sun has eyes”.
In general, a fact can be expressed by relational triples, where each relational triple takes the form of <subject, predicate, object>. A question is asking for the missing part in one relational triple. For example, the above question can be expressed as nested triples as <triple, quantity, ?>, where triple = <sun, has_property, eye>. We define the complete relational triple as the support triple. Then a false premise problem is one whose support triples are not correct. In the above example, <sun, has_property, eye> is false under real-world background, thus any question that builds on this triple contains false premises. By this definition, “Does the sun have eyes?” is not an FPQ, since it does not assume <sun, has_property, eye> to be true. In fact, PLMs know the authenticity of such triples well. However, they can’t answer FPQs built upon these triples.

heading: 3.2 PLMs’ original responses to FPQs
text: We begin with a pilot experiment that confirms current PLMs’ responses to FPQs are not satisfactory despite their knowledge. We query the PLMs with the questions taken from FalseQA test split (see
Section 4). We use the large PLMs whose API is publicly available, including Bloom (Scao et al., 2022), OPT (Zhang et al., 2022), Jurassic-1 (Lieber et al., 2021), GPT-3(text-davinci-003) (Brown et al., 2020) (as known as InstructGPT). We use the prompt “Question: Answer:”, where the blank is filled by the question text. We provide the generated answers of these models in Table 2. We also provide our model’s answer (See Section 5) as comparisons. As we can see, all models fail on these simple FPQs. However, in the column “Ablation”, we are surprised to find that all models give the correct responses to the questions that ask directly about the correctness of the premises. This motivates us to hypothesize that the inability of current PLMs to handle FPQs is due to distribution mismatch, instead of missing knowledge. Therefore, we need a dataset specializing in FPQs.

heading: 4 Dataset
text: To build a dataset on FPQs, there are potentially two approaches. An approach is to collect them from natural corpora. However, false premise questions rarely appear in natural corpora, which makes the question collection process laborious. Second, even if we collect false premise questions, the false premises are made by humans and thus are hard to be detected by humans, which doesn’t fit with the motivation of this paper. In fact, Min et al. (2022) have done pioneering work using this approach. On the contrary, our approach is to manually write
such false premise questions. To ensure the quality of our dataset, we expect FalseQA dataset to have the following key features: broad coverage, high quality, few shortcuts, and detailed explanations for the false premises. Below we introduce the annotation steps that ensure these features.

heading: 4.1 Categorization of FPQs.
text: People ask questions in a wide variety of contexts and formats. Increasing the coverage of questions is proven to be beneficial (Khashabi et al., 2020). However, asking annotators to write FPQs freely does not guarantee the coverage of the questions. Therefore, the authors manually think up 29 initial FPQs (see Appendix A.1). Then we categorize these FPQs in terms of error types, and question format. We summarize the categories in Table 3. In total, there are eight error types covering commonsense errors, logical errors, etc., and six question formats covering factual questions, descriptive questions, etc. Although we try to collect as many examples as possible into the initial set, the categorizations are far from exhaustive. Therefore we include an “Others” option to encourage creativity.
Writing FPQs. We recruit twenty human annotators to think up questions that contain false premises. To make the creative process easier, we provide source words to the annotators to compose sentences. We use the subject word of GenericsKB (Bhakthavatsalam et al., 2020) as the source word since they have broad coverage and each
word is paired with a short illustrative sentence that can also inspire the annotators. However, we don’t require the annotated sentence to contain the source word. Moreover, the annotators have the freedom to skip the source words that are not easy to brainstorm. We then ask the annotators to categorize the questions into the above categories. The annotators are required to keep a balanced distribution (see Appendix A.2) over categories when they finish their part. For the quality of the written FPQs, we require them to be correct in syntax and contain obvious false premises.
Revising into TPQs. Previous studies (Du et al., 2021) point out that PLMs are skilled at finding shortcuts in datasets and do not really understand the task. Since the FPQs are created manually, it’s easy to fall into the fixed writing style of the annotators. To alleviate the problem, we annotate a comparison set for these FPQs. Specifically, we ask annotators to edit each FPQ with minimal modifications to make it a problem with true premises (TPQ). The resulting pairs of questions differ only in the correctness of the premises, ensuring that the model learns the essentials of the task.
Writing Detailed Explanations/Answers. Humans usually reply to FPQs with an explanation of why the premise is false (Kaplan, 1978). Generating the explanation also helps check whether the model truly understands the FPQs. Therefore, we ask the annotators to write an explanation for each FPQ. For quality control of the explanations, we require the explanation to be more than the negation of the false premise. For the training set and validation set, we require one explanation per question, for the test set, we require two explanations per
question. For symmetry, the annotators also write answers to the TPQs. The full annotating process is demonstrated in Figure 2.

heading: 4.2 Dataset Statistics
text: The final dataset, dubbed as FalseQA, contains 2365 question pairs. A snapshot of the FPQ dataset is in Table 5. We randomly split the dataset into train, validation, and test splits, with a ratio of 5:2:3. The summary of statistics is shown in Table 4.

heading: 5 Experiments
text: Our experiments are divided into two main parts. To begin with, we conducted extensive experiments to demonstrate that PLMs have the ability to discriminate and rebut FPQs with moderate training data. Next, we propose a practical method to handle both FPQs and general questions well.

heading: 5.1 Models and Settings
text: PLMs are usually divided into three main architectures, namely, encoder-only, decoder-only, and encoder-decoder language models. Since the encoder-only language model can not be used as the QA model, we select typical PLMs from the latter two for experiments.
For decoder-only models, we choose OPT (Zhang et al., 2022), which is a series of open-source pre-trained models aligned to OpenAI GPT-3 (Brown et al., 2020). For the encoder-decoder models, we use T5 (Raffel et al., 2020) and MACAW (Tafjord and Clark, 2021). T5 (Raffel et al., 2020) models are trained with the massive unsupervised pre-training corpus and a mixture of supervised tasks, making them very capable of solving various downstream tasks. MACAW is fine-tuned from T5 models on QA tasks. They achieve state-of-the-art performance on direct QA dataset ARC-DA (Bhakthavatsalam et al., 2021) and perform satisfactorily on most categories
of the demanding dataset Challenge300 (Tafjord and Clark, 2021) except for the FPQs.
Unless specified, all experiments are repeated three times with different random seeds. For each result, we report the mean and standard deviation. The detailed hyperparameters for each experiment are in Appendix B.

heading: 5.2 Discriminating FPQs
text: We first train the PLMs to classify the question in FalseQA into FPQ and TPQ. To mitigate the gap between pre-training and fine-tuning, we adopt the prompt learning paradigm (Schick and Schütze, 2021; Liu et al., 2023) to do the classification. We report the accuracy of the classification. Besides, we report the recall and precision for FPQs since we emphasize the FPQs.
From Table 6, we can see all the models can achieve non-trivial performance on the binary classification. (1) The most powerful model MACAW11B, can achieve 86.6 accuracy. (2) Across all the models of the same type, performance boosts when the size of the model increases. We hypothesize that the scaling effect is because larger models both contain more knowledge and are easier to be activated to understand the task. (3) There is a slight improvement from T5 to MACAW, showing that the ability to identify FPQs can be enhanced by fine-tuning on a corpus of normal questions.

heading: 5.3 Impact of Training Data Size
text: Then we study the PLMs’ performance to discriminate FPQs with fewer training data. We
randomly sample 32, 128, 256, and 512 pairs of FPQ and TPQs as the training data and plot the performance under each data scale in Figure 3. We can see that the accuracy of classifying FPQs and TPQs grows almost linearly as the number of pairs grows exponentially. With only 256 pairs of questions, models larger than 2.7B, i.e., OPT-2.7B, MACAW-3B, MACAW-11B, all achieve more than 70% accuracy, while the smaller models need more data to achieve non-trivial performance. The trade-off between model scale and data scale hints that larger models might be activated with even fewer training data. However, as we have noticed, the gap between human performance and model performance remains large, as an average person can almost completely classify such problems.
The above results already allow us to design a primitive QA pipeline that can handle FPQs. For example, if the model predicts that a question is FPQ, then it refuses to answer such questions, while for other questions it generates the answer.

heading: 5.4 Answering FPQs with Explanations
text: Next, we train the PLMs to discriminate and generate explanations for the FPQs at the same time. Since we need to start from models that already have zero-shot QA ability, we choose only MACAW for the encoder-decoder models. For the decoder-only model, we follow similar approaches to Tafjord and Clark (2021) to train OPT models with a fraction of UnifiedQA dataset (Khashabi
eration. Better results are shown in green .
et al., 2020) in order to steer the model into QA mode 4 without injecting much additional knowledge. We select the model size that can achieve non-trivial performance using 256 pairs of data for this experiment.
To discriminate and generate explanations jointly, we let the models generate the discriminating tokens: “tricky question” or “true question” first. Then the model continues to generate the explanation to FPQs or the answer to TPQs. Since the numbers of tokens responsible for discrimination and generation differ dramatically, we add an additional binary loss on the discriminating tokens. The ratio between the binary loss and the generation loss is 1. We conduct experiments on three training data sizes, i.e, 32, 256, and 1187 question pairs.
In evaluation, if a generated answer contains “tricky question”, we consider the question classified as an FPQ, otherwise, it is classified as a TPQ. Similar to the previous section, we report the recall, precision of predicting FPQs, and accuracy of the binary classification. In addition, we evaluate the quality of the generated explanation by computing the maximum ROUGE-L (Lin, 2004) score between it and the two ground-truth explanations. Note since we focus on the explanation of FPQs, the evaluation does not include the TPQs.
From Table 7, we have three observations. (1) The models jointly predict the question and generate answers successfully. (2) When training data is limited, e.g., 32 question pairs, the accuracy is significantly higher than conducting classification
4We will release the checkpoint.
alone (See in Figure 3), which shows that the explanations of the FPQs help the model to quickly adapt to the task. (3) Adding binary loss boosts the model’s performance on classification. For the generated explanations, the best ROUGE-L achieves 42.0, showing that the explanations are close to humans’. The quality of explanations also gets higher as the model size and data size increase. We provide the model-generated explanation for 10 randomly sampled FPQs in Appendix C.2. We can see the explanations are reasonable.
ent numbers of examples. Better results are in green .

heading: 5.5 In-context Learning
text: We proceed to study the performance of larger models, e.g., GPT-3(175B) on FalseQA. The large PLMs are tuned by in-context learning with frozen model parameters. We select OPT-66B (Zhang et al., 2022), Jurassic-1 (Lieber et al., 2021), and GPT-3(001) and GPT-3(002) 5. We present the results in Table 8. We can see that OPT-66B and Jurassic-1 perform poorly. Therefore, we conclude that due to the distribution mismatch of FPQs to normal questions, it is still hard to activate the rebuttal ability using a few examples for these models, which we leave to future work. GPT-3 can be activated with 2 or 4 pairs of examples, however, its performance is lower than the much smaller finetuned models in Section 5.4. Surprisingly, GPT3(002) has far better performance than GPT-3(001). We hypothesize that they more easily understand the rebuttal task since they are trained with instruction tuning (Ouyang et al., 2022).

heading: 5.6 Performance w.r.t. Category
text: To better understand which kind of FPQs is harder to be discriminated against, we draw the accuracy of each category in Figure 4. In spite of the inconsistency between PLMs, index error is generally
5text-davinci-001, and text-davinci-002 checkpoints.
hard to classify while logic and causality error is easy. For question types, selective questions are hard to classify while factual questions are easy. These observations can guide the future improvement of our dataset.

heading: 5.7 Answering FPQs and General Questions
text: QA models are originally used to answer general questions, e.g., questions in ARC-DA (Bhakthavatsalam et al., 2021) 6 dataset where the distribution is different from FalseQA. Therefore, training purely on FalseQA may lead to catastrophic forgetting. To produce a model that handles both FPQs and general questions, we use a simple data replay technique (DR) (Chaudhry et al., 2019). Specifically, during training on FalseQA dataset, for each iteration over batches, we add a batch of the data samples from the ARC-DA. In order to use as little ARC-DA data as possible, we keep the ARC-DA samples to be the same within 30 batch iterations. The aforementioned binary loss is used no matter with or without DR. The concrete numbers of general questions used in each setting and training details are in Appendix B.5.
In Table 9, we summarize the performance of the raw model before training on FalseQA, the model tuned on FalseQA, and the model tuned on FalseQA with DR. For the original models, since they do not generate the “tricky question” or “true
6Short for AI2 Reasoning Challenge-Direct Answer.
question”, we manually read the generated answers for 100 randomly sampled questions pairs to determine whether it contains any rebuttals. As we can see, before fine-tuning on FPQs, the models perform well on the ARC-DA dataset. However, they fail substantially on FalseQA. After tuning on FalseQA, though the models’ rebuttal ability is activated, ROUGE-L and F1 scores on ARC-DA drop considerably. The false prediction rate (FPR), i.e., the fraction of ARC-DA questions that are incorrectly labeled as tricky questions, is non-negligible. Fortunately, when we apply the DR technique, models not only have small FPRs and the improved quality of generated answers on ARC-DA but the same or even better performance on FalseQA. We also find the questions in ARC-DA that PLM still rebuts (see Appendix C.3) are also reasonable to rebut for humans. The result gives us a promising direction for building QA systems that perform well on general questions and FPQs.

heading: 6 Conclusion
text: In this paper, we investigate using PLMs to answer FPQs, which are simple for humans but deceive most PLMs. We present the first human-written dataset of FPQs. Using the dataset, we successfully activate the discrimination and explanation ability of PLMs and produce PLMs that are both capable of general questions and robust to FPQs. For future directions, we think that more advanced techniques
can be used together with FalseQA to fully activate the model’s ability, e.g., reinforcement learning with human feedbacks (Ouyang et al., 2022). Incorporating more knowledge into PLMs is also beneficial for PLMs to answer FPQs.

heading: Acknowledgement
text: This work is supported by the National Key R&D Program of China (No. 2020AAA0106502) and Institute Guo Qiang at Tsinghua University.
Limitations
There are several limitations in our work. (1) Although we think that PLMs’ rebuttal ability is activated in our experiments, the performance has a large space for improvement. For a binary classification problem, the most powerful PLM in our experiment reaches 87.1% accuracy at most. (2) Since it’s hard to probe what the PLMs truly know, we didn’t further investigate whether PLMs still fail on some FPQs due to a lack of relevant knowledge or other reasons. (3) A third limitation is that we notice that the newly announced model ChatGPT (OpenAI, 2022) handles such questions satisfactorily. However, since their training data and details are not open-sourced, we are unable to investigate how the ability of these particular models is activated. (4) In this paper, we standardize the expected responses to FPQs as rebuttals, which takes a conventional perspective. However, sometimes we can react with a more creative response, such as a rhetorical question. This can be future work.
Ethical Statement
In the construction of the dataset, we forbid the annotators to compose any sentence that is offensive, harmful, or contains personal information. The annotated data is manually checked to ensure safety. We pay our annotators a competitive salary relative to market rates. The annotated dataset is helpful to encourage models “think” before they provide a response, thus being safer in practical deployment.

heading: A Annotation Details
text: A.1 Initial FPQs
We provide the annotators with 29 FPQs in the annotation guide. These questions are original references provided for annotators to brainstorm questions. We list the questions and their error types in Table 10. We didn’t provide FPQs for each question format since the question format is much easier to determine without examples.
A.2 Distribution Balance Criterion
We expect our dataset to have a richer and more uniform distribution of FPQs. We achieve this goal with the help of constraints on the FPQ types. For the eight error types, each type of FPQ should account for at least 5% of the overall data, and the maximum category should not exceed 30%. And for the six problem formats, each type of FPQ
should account for at least 10% of the entire data, and the maximum category should not exceed 30%. All balance criteria do not take into account the “other" category.

heading: B Experiment Details 7
text: B.1 API Calls for Pilot Experiments
We summarize the APIs used in Section 3.2 in Table 11. We will also provide the screenshot of using these APIs in our final reproducible code.
B.2 Details of Discriminating FPQs
For the experiments in Table 6, we use the prompt learning (Schick and Schütze, 2021) paradigm. We use “true” and “false” as the label word for FPQ and TPQ, respectively 8. For T5 models, following the usage of T5 (Raffel et al., 2020) in their original paper, we append “potential tricky question:” to identify the task. MACAW models are multi-angle QA models, to use their direct question angle, we follow their paper and use “$answer$ ; $question$ = ” as the prefix. For OPT models, we train them in a vanilla input-output format. We list the hyperparameters for each experiment in Table 12. For MACAW-11B, we use half-precision acceleration and do not find performance degradation compared to full-precision computation. For the experiment in Figure 3, we use the same input-output format mentioned before. Our hyperparameters used in this section are listed in Table 12.
B.3 Details of Answering FPQs
Since fine-tuned models in few-shots (e.g. 32 question pairs) sometimes may not generate “tricky/true question” at the beginning of sentence 9, and a normal answer hardly has “tricky/true question” in it, we count whether “tricky question” or “true question” appears in outputs for classification evaluation to get the recall, precision, and accuracy scores. When evaluating the generated explanation, we remove “tricky question” and “true question”. We list our hyperparameters used in this section in Table 13 and keep them the same when adding the binary loss.
7We choose random seeds 4, 13, and 34 in all experiments. 8Since our target is to classify whether it has a false premise, we set True for FPQs and False for TPQs. 9Some seeds in OPT models sometimes produce “this is a tricky question”.
B.4 Details of In-context Learning
In-context learning, introduced in GPT-3 (Brown et al., 2020), has been a successful way of adapting extensive language models. In in-context learning, we provide a textual prefix p of the task and one or a few training data samples before sending the input questions. We adopt the QA prefix in the GPT-3 demo for all the PLMs tested. Specifically, the prefix is:
p = I am a highly intelligent question answering bot. If you ask me a question that is rooted in truth, I will give you the answer. If you ask me a question that is nonsense, trickery, or has no clear answer, I will say “tricky question.” first and give the reason, otherwise I will say “true question.” first and give the reason.
A few pairs of FalseQA samples {(qiF , aiF ), (qiT , aiT )} can be concatenated to the textual instruction. Therefore the full prefix before the input question has the following form:
p+ Q:qiF + A:a i F + Q:q i T+ A:aiT + ...+ Q: + A:
where + indicates string concatenation, and the input example is filled into the blank.
We list our hyperparameters for in-context learning in Table 14.
B.5 Answering FPQs and General Questions
We list our hyperparameters in this section in Table 15. We count the number of general questions when using the data replay technique in Table 16.

heading: C Additional Results
text: C.1 More Raw PLM’s Responses to FPQs
We present three more examples of PLM’s responses to the FPQs and their responses to the corresponding questions that directly ask about the correctness of the premises in Table 17. We can see that in most cases PLMs identify whether the premises are true or false successfully, however, they fail on the FPQs.
C.2 Model-generated Answers and Explanations
We present randomly sampled FPQs in the test split and the corresponding references, discrimination results, and explanations/answers in Table 18. We use MACAW-11B trained with full training data while binary loss is added in this demonstration. We can see that in most cases, the explanation generated by the model is close to the reference. However, there are cases that the generated explanation is counterfactual. For example, “A spider’s shell is not helpful to its breath” is incorrect.
C.3 The Questions in ARC-DA that MACAW-FPQ Rebuts
We show the problem that the model still rebuts after data replay. Specifically, we show the model results for the MACAW-11B model after training
on the full training data as well as the replayed data. Since our experiments have three seeds, we show the problem that the model refutes in all seeds. We also show the explanations generated by our model, we randomly pickle one explanation from the three seeds. As we can see in Table 19, the correctness of the premises of these questions is not very clear. As a human, these questions can also be seen as questions containing false premises.
The question in Table 19 “How is a skin cell from a mouse similar to an amoeba?” can be seen as a question that contains a false premise “A mouse’s skin cells, like amoebas, are single-celled organisms.”, as a human, we may also rebut this presupposition. For the question “Volcanoes are considered constructive because they”, generally, the volcanoes are considered destructive unless we want a creative answer. If a user truly wants the creative answer, he might provide explicit instructions to the PLM to trade robustness for creativity, which can be future work.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: page 9 Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: page 9 Section Ethical Considerations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 0, Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 Dataset and Section 5 Experiment

heading: 3 B1. Did you cite the creators of artifacts you used?
text: scientific artifacts
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The dataset is either public or created by ourselves.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 Dataset and Section 5 Experiment

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 4 Dataset

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4 Dataset
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 Dataset
C 3 Did you run computational experiments? Section 5 Experiment

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5 Experiment
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 Experiment and Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5 Experiment
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4 Dataset
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? The instructions are not in English.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4 Dataset

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 4 Dataset
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not yet, but we will try to get one.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We will provide the details only after publication. Currently, providing such information might potentially reveal our identity.

heading: Future Work
text: If a user truly wants the creative answer, he might provide explicit instructions to the PLM to trade robustness for creativity, which can be future work. ACL 2023 Responsible NLP Checklist