The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models
Authors: Aviv Slobodkin
Omer Goldman
Avi Caciularu
Ido Dagan
Shauli Ravfogel
Venue: EMNLP 2023 Main (2023)
Keywords: LLMs
explainability
answerability
hallucinations

Abstract: Large language models (LLMs) have been shown to possess impressive capabilities, while also raising crucial concerns about the faithfulness of their responses.  A primary issue arising in this context is the management of (un)answerable queries by LLMs, which often results in hallucinatory behavior due to overconfidence. In this paper, we explore the behavior of LLMs when presented with (un)answerable queries. We ask: do models \textit{represent} the fact that the question is (un)answerable when generating a hallucinatory answer?
Our results show strong indications that such models encode the answerability of an input query, with the representation of the first decoded token often being a strong indicator. These findings shed new light on the spatial organization within the latent representations of LLMs, unveiling previously unexplored facets of these models. Moreover, they pave the way for the development of improved decoding techniques with better adherence to factual generation, particularly in scenarios where query (un)answerability is a concern.

The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models
Aviv Slobodkin', Omer Goldman', Avi Caciularu 12 Ido Dagan', Shauli Ravfogel'
Bar--lan University ?Google Research
{lovodkin93, omer golldng,,,,, shauli321 shaulil21]@gmail... fegmail.com avica@google. com daganêcs. biu.ac. il
Abstract
Large language models (LLMs) have been shown to possess impressive capabilities, while also raising crucial concerns about the faith- fulness of their responses. A primary issue arising in this context is the management of (un weneealleeeele queries by LLMs, which of- ten results in hallucinatory behavior due to overconfidence. In this paper. we explore the behavior LLMs when presented with (un)answerable queries. We ask: do mod- els represent the fact that the question is (un)answerable when generating hallucina- tory answer? Our results show strong indica- tions that such models encode the answerability of an input query, with the representation of the first decoded token often being a strong indica- tor. These findings shed new light on the spatial organization within the latent representations of L.Mss, unveiling previously unexplored facets of these models. Moreover, they pave the way for the development of improved decoding tech- niques with better adherence to factual gen- eration, particularly in scenarios where query (un) (un)answerability is concern.
(a) SQUAD
(b) NQ
(c) MuSiQuc
Unaroweratie
Unanswerable
Answerable
(elentifee))
(unidentifed)
fidertifias)
(uridemifus)
Figure 1: 3D PCA projection of the last hidden layer's embedding of Flan-UL.2 on each of the three bench- marks. The left images show the embeddings with the regular prompt, and the right ones with a hint- including prompt. Blue and red dots are examples correctly detected by the model answerable and (un)answerable, respectively. while the pink dots are for (un) jaswerable examples that the model provided answers The figures show the good separability be- tween the three groups.
1 Introduction
Modern large language models (LLMs) have been tantalizing the NLP community in the last couple of years (Brown et al., 2020; Chen et al.. 2021; Chung et al., 2022), demonstrating great potential for both research and commercial use, but these models are of course not problem-free. Among their unfavorable behaviors it is possible to find toxicity (Welbl et al., 2021; Deshpande et al., 2023), bias (Nadeem et al., 2021; Abid et al., 2021), and hallucination (Manakul al., 2023; Ji et al., 2023). One of the settings in which LLMs are notori- ously prone to hallucinate is when presented with (un)answerable questions (Sulem et al., 2021; 12;1;A4 Asai
and Choi, 2021; Amayuclas al., 2023). Recent works in this setting, which is the focus of this work, suggested using models' confidence as an indication of answerability (Yin al., 2023), and some suggested further finctuning to enhance the probability of detecting (un)answerable questions
'Ou code is publicly available at Lovodkin93 /nanswerabbllity
com/

---

(Jiang et al., 2021; Kadavath et al., 2022). We, how- ever, ask whether models already represent ques- tions' (un)answerabliiity when producing answers, and find strong evidence for a positive answer. Specifically, by experimenting with three QA datasets (Rajpurkar et al., 2018; Kwiatkowski et al., 2019; Trivedi al., 2022), we observe a substan- tial increase in performance for (un)answerable questions (up to 80%) simply by incorporating to the prompt the possibility of (un)aaswerabiilty. We further show that, even in the absence of guidance in the prompt, the fact that the ques- tion is (un)answerable is decodable from the model's representations. This is done by two meth- ods: first, we find that the beam of decoded re- sponses for (un)answerable queries often contains a response recognizing their (un)answerability; second, we demonstrate that the fact that the question is (un)answerable is easily decodable from the model's representations and that there is a linear separation between representations of (un)answerable and (un)answerable questions (see Figure 1). The existence of the answerability subspace is largely independent of the specific QA dataset used, in the sense that an answerabil- ity classifier trained over representations of ques- tions from one dataset can successfully classify (un)answerable questions from other datasets as well. In addition to providing illuminating insights into the internal mechanics of LLMs, these find- ings also open up new avenues for better decod- ing methods (Meister et al., 2020; Wiher et al.. 2022) to improve performance in general and on (un)answerable questions in particular.
of (un)answerable questions, in which supporting paragraphs have been intentionally removed from the context. Our experiments use these datasets to demonstrate the effectiveness of our approach to identify (un (un)answerability.
Jnnaawsweryyl capabilities in LLMs were mainly studied by using few-shot prompting (Kand- pal al., 2022; Weller et al., 2023). Moreover, sev- eral works have recently shown that LLMs become easier to steer with natural language prompts either as they become larger (Mishra et al., 2022a; Kand- pal et al.. 2022; Carlini et al., 2023) or as they are exposed to larger instruction tuning data (Mishra et al., 2022b; Chung et al., 2022; Wan et al., 2023a), and as consequence, it might improve the (un Janwerbbibyyyy capabilities of the model. Specif- ically, in this work, we utilize prompt manipulation in order systematically reveal to the model the option of avoiding answering hard questions. Auto- matic prompt tuning can be also used for improving (un (un)annwerbbility capabilities, without the need for manual handcrafting prompts. Liao et al. (2022) in- troduced prompt tuning-based strategy to mitigate (un) auneabbseeeble questions, by mapping questions into their proper. specific templates.
Other works tried to manipulate the model pre- dictions towards better (un)answerabbilty via using data augmentation (Zhu et al., 2019), and Asai and Choi (2021) provided an in- in-depth analysis of the ability to detect (un)answerability in LMs, where the case study is the data which is fed to the model.
Furthermore, recent studies have suggested uti- lizing recent advances in white-box model inter- pretability (Geva et al., 2022b; Li et al., 2022; Mallen et al., 2022; Mickus et al., 2022: Meng et al., 2023; Geva et al., 2023) and probing (Adi et al., 2017; Conneau et al., 2018; Voita et al., 2019; Slohodkin et al., 2021) for manipulating the model predictions and analyzing when LLMs struggle answer questions. Recent works also tried to use beam search decoding to manipulate the generated outputs by using the information encapsulated in several beams (Meister et al., 2020; Leblond et al., 2021; Slobodkin et al., 2023; Wan et al., 2023b). Finally, early exiting in language models (Schwartz et al., 2020; Schuster et al., 2022; Din et al., 2023) and model prediction calibration (Desai and Dur- rett, 2020; Jiang et al., 2021; Dhuliawala et al., 2022; Geva et al., 2022a) are strongly related to our work, as they suggest to analyze and improve the model predictions and output distribution.
2 Related Work
In previous research. (un)answerable questions were used to evaluate reasoning capabilities (Ra-
jpurkar et al., 2018; Ferguson et al., 2020;
Kwiatkowski et al., 2019; Trivedi et al., 2022).
It was SQuAD v2 (Rajpurkar et al., 2018)
that provided the first reading comprehension dataset for validating models' ability to deal with (un)answerability, by introducing questions that cannot be addressed from the given context. Kwiatkowski et al. (2019) followed the same line and included about third of (un)answerable ques- tions in their NATURAL QUESTIONS (NQ), an annotated open-domain QA dataset. Recently, MuSiQue (Trivedi et al., 2022) was introduced as a challenging mupti-hop QA benchmark that consists

---

First, we ask whether the model's ability to iden- tify (un)answerable questions is sensitive to the exact wording of the prompt. Specifically, we ask whether merely raising the option of unasnwerabil- ity makes the model less susceptible to hallucina- tion. To that end, we experiment with two types of prompts. The first type is designed to merely guide the model towards addressing question. The second type, however, is more instructive in its approach. Besides guiding the model, it provides an advised course of action for scenarios where the question at hand (un) (un)answerable, hence indi- rectly hinting at the potential for (un) Jun)answerabiliyy Winwdd Lytonseersillyy
amine whether we can endow this algorithm with bias towards unanwerability. Focusing on the zero-shot setting, we gradually increase the beam size. Then, instead of automatically choosing the fighest-rrobabilig answer from the final set of k options, we search for a reply within the final & op- tions that signifies (un) (un)answerabilltt (Appendix A), If such an answer is discovered, we substitute the top-heam answer with "unanswerable".
Queen Milwiin gueton
Iasidon witeet
Sieett Ealowing 200 Xp and Penegg (PPasages)
(Jawestonn)
Gua ket (1Osstion))
Glenn the bakwving paraage and quarder answar
Gieen the bilowing sarcage and
I1i do inscees and saad on passage
Pecoren [Pansagi) Prowey Jinseos
Poseage (Prossrr)
v81
Giventhe bellwing zasaage quntion pandor
Givan lallving
suedioon
((aasape))
(11
bead
Ques ((Questlar))
Quertoon ((Osettor))
Figure 2: Combinations of prompt variants in this work. In addition to some basic instructions, our prompts can also have "hint" to the possibility of (un)answerrbbliity as well as 2 exemplars.
3.3 Identifying an Answerability Subspace In a subsequent set of experiments, our objective is to find evidence for (un)answerability encod- ing directly in the embedding space of the mod- els, by probing the models' last hidden layer. For each task, each model is prompted with a bal- anced trainset comprising 400 answerable and 400 (un aanwerable examples. Then, for each instance, take the embedding from the final hidden layer of the first generated token and train a linear clas- sifier, using logistic regression, to predict answer- ability.? Subsequently, we assess the performance of each classifier on the corresponding test set. As baseline, we also conduct similar experiments using the initial (non-contextual) embedding layer, which should not encode whether the question is answerable or not. Our core objective within this experimental setup is to ascertain whether basic linear classifier, trained on a modestly sized dataset, suffices to effectively discriminate between answer- able and (un)answerable queries.
3 Method
We posit the hypothesis that, despite the inclination of LLMs to produce answers to (un (un)answerable queries, they do encode the (un)answerability of such queries within their latent representations. We examine this hypothesis by undertaking three dis- tinct experimental approaches: (1) prompt manipu- lation, (2) beam scrutiny, and (3) probing (includ- ing identification and erasure of an answerability subspace).
3.1 Prompt Manipulation
3.4 Erasing the Answerability Subspace
Upon identifying a linear subspace that corre- sponds to (un)answerability, natural question to ask is whether that subspace has a behavioral rele- vance, i.e., whether it is being used by the model when producing text. Importantly, this is differ- ent than mere encoding of the information, as the information can be present in the representation and at the same time be irrelevant to the model's 's behavior (Hewitt and Liang, 2019; Elazar et al., 2021; Ravfogel et al., 2021). Recent work on lin- ear concept erasure (Ravfogel et al., 2020, 2022b; Belrose et al., 2023) have proposed set of meth- ods to erase arbitrary linearly-encoded concepts from neural representations, following the intuition that by erasing a subspace that encodes the concept and examining the effect on the model's output,
Our experimental setup encompasses both zero- shot and few-shot prompts, with the latter in- volving the integration of two exemplars in the prompt. In the standard prompt setup, both exem- plars are answerable. However, within the hinting prompt framework, one exemplar designed to be (un)answerable. Figure 2 demonstrates all variants.
3.2 Beam Relaxation
We also experimented with averaging across all the gen- erated tokens' embeddings, which was found to yield inferior performance on a designated development set.
Recall that the output of LMs is usually decoded with algorithms such as beam search. We aim to ex-

---

Answerable
(Unnaanwwrabbee
ditionally, for the probing experiments involving the training of linear classifiers on the models' em- beddings, sample 1000 instances from each bench- mark's s trainset, evenly distributed between answer- able and (un)answerable instances. Of these, we reserve 800 instances for the training of classifiers, with the remaining instances forming the develop- ment set for these classifiers. Below, we describe how we associate asnwerable questions with para- graphs that contain the answer, and how we as- sociate (un)answerable questions with challenging paragraphs (that do not contain the answer, but may be topically similar to the question).
SQuAD
5928
5945
NQ
3489
7719
MuSiQue
1950
1316
Table 1: Number of extracted answerable and (un)answerable questions per dataset test set.
we can verify that the subspace we identify is be- haviorally meaningful, opening an avenue for per- forming interventions in that subspace in order to modify the model's behavior. These methods start with the original representations alongside binary labels (e.g., representations of the text alongside binary gender annotations for each text), and re- turn new representation which is linearly guarded in the sense that any linear classifier trying to re- cover the concept from the representation will fail. While linear erasure has its limitations (Ravfogel et al., 2022a), it has been proven to be an effective method for intervening in the latent representations of black-box box models.
Our first benchmark is SQuAD 2.0 (Rajpurkar et al., 2018), a reading comprehension dataset, com- posed of manually-curated question-anse ansser pairs alongside (un)answerable questions, each derived from single paragraph.
The second benchmark we explore is NATURAL QUESTIONS (NQ; Kwiatkowski et al., 2019), a dataset accumulated from user-generated queries on the Google search engine. Each item within the dataset consists of question, retrieved article, a selected paragraph from the article (referred to the "long answer"), and a short answer inferable from the paragraph. Despite its potential to test QA systems with a retrieval component, our interest lies exclusively in the question-answe answering setting, hence we utilize the "long answer" as the context, assuming an oracle retrieval system. For the formu- lation of answerable instances, we select cases with both long and short answer, using the former as the context and the latter as the response. For the (un)answerable questions, we pair each query with a paragraph from the sourced passage that has not been annotated as the "long answer". In order to create challenging dataset, we select the para- graph that is closest in meaning to the question. To achieve this, we encode both the question and all potential paragraphs using (Sentence-Bere Reimers and Gurevych, 2019) and select the paragraph that exhibits the highest cosine- similarity score.
We use the recently proposed method of Bel- rose et al. (2023) which provides closed-form solution for the concept-erasure objective. Con- cretely, given a binary concept (answerabilityy, the method provides a projection matrix that minimally changes the representations (in the Le sense) while at the same time guarantees the inability to lin- early predict the answerability from the modified representations. We fit the method over the last- layer representations of the training instances from Flan-UL2, particularly when these instances prompted with regular queries from the SQuAD benchmark (refer to $3.3). Then, during inference, the concept-erasing projection matrix is applied in the first generation step, specifically for the test set within the same model-dataset pairing. Our goal is to inspect whether removing the linear separation that exists in the latent space of the model between answerable and (un)answerable questions changes the behavior of the model.
4 Experimental Setup
Our experiments focus on several language models and on three benchmarks.
Our final benchmark is the MuSiQue dataset (Trivedi et al., 2022), a multi-hop dataset featur- ing both answerable and (un)answerable questions. Each instance consists of a question, several candi- date paragraphs, an answer, and decomposition of the question into its single-hop sub-questions. Additionally, each sub-question is paired with a paragraph that has its answer, with all those align-
Benchmarks We consider three QA benchmarks, incorporating (un)answerable questions, in read- ing comprehension setting where models are tasked with responding to a question within a given con- text. For each benchmark, we use the entire de- velopment set to construct our testing dataset. Ad-

---

SQuAD
NQ
MuSiQue
model did not provide an answer as an instance of un-answerability prediction. Note that this evaluation does not consider the correctness of the answers provided. The metric associated with this task is the F1 score, with "unanswerable" consid- ered the positive label. Linear classifiers ($3.3) are also evaluated over the (un)answerabillty classifi- cation task, as the classifier is trained to predict whether not the question is answerable, based on the hidden representations of the LM.
Regular
37.3
5.7
8.2
(11B)
+hint
91.5
85.6
74.7
FLan-UL2
Regular
46.3
13.8
6.8
(20B)
+hint
92.3
83.9
67.3
OPT-IML
Regular
43.9
21.8
17.1
(30B)
+hint
85.4
85.3
54.1
(a) Zero-shot
SQuAD
NQ
MuSiQue
TSAs (11B)
Regular
52.6
8.6
13.6
(11.7)
(2.1)
(4.9)
Additionally, in order to make sure that our meth- ods do not hinder the performance of the models over their primary task, we evaluate them over the QA task as well, using the splits provided by the tasks' designers. We report the commonly used metries: exact match (EM) and (token-wise) F1 scores (Rajpurkar et al., 2016).
hint
91.2
85.8
75,4
(1.0)
(0.5)
(1.4)
Stan-UL2 (20B)
Regular
67.7
20.6
14.5
(4.0)
(1.8)
(4.9)
hint
92.5
83.7
72.1
(0.1)
(0.1)
(0.6)
OPT-IML (30B)
Regular
10.0
11.1
4,9
(1.2)
(2.1)
(1.0)
hint
79.3
85.0
27.5
Language Models We evaluate three instruction- finetuned large language models: namely, the 'xxl' variant of the Flan-T5 model (Flan-T5xxi Chung et al., 2022), the Flan-UL2 model (Chung et al., 2022), and the OPT-IML model (lyer et al., 2023).
(0.3)
(1.5)
(8.0)
(b) Few-shot
Table 2: F1 scores over the (un)answerabilliy classifica- tion task in both zero-shot and few-shot setting. Each model is prompted with a regular prompt and with a prompt that hints the possibility of (un)answerability ("+hint"). In the few-shot setting, results are averaged across three variations of in-context-learning examples (with standard deviation in brackets). Bold marks the better prompting method.
Results
5.1 Prompt Manipulation
5.1.1 Zero-Shot Scenario
Table 2a presents the results the zero-shot set- ting (the model was not provided with question- answer examples). It shows that the detection of (un annwerbleeee questions is substantially improved upon the integration of a hint towards the possibility of (un auneariieyyyy into the prompts, with gains as high as 80 points. can also be observed that, without a hint, the ability to discern (un)answerable queries tends to be superior in larger models. In- terestingly, the introduction of the hint appears to mitigate the impact of model size, evidenced by the smaller Flan-T55xl surpassing its larger coun- terparts in two out of three benchmark evaluations. Additionally, Table 3a displays the models' ex- act match and token-wise Fl scores over the QA task where the model tasked with both detecting aanwerrbeee questions and provide a correct an- swer to the answerable ones. It reveals a notable enhancement in the quality of generated responses when prompted with hint, in some cases resulting in improvements of over 50 points (on both met-
ing paragraphs concatenated and used as context, Conversely, for the (un)answerable queries, the ab- sence of such alignment for some sub-questions is observed. For these (un)answerable instances, identify the paragraph most closely linked to each of the unanswered single-hop questions, using process akin to the approach with the NQ bench- mark. These identified paragraphs are then aggre- gated, together with the paragraphs corresponding to the other single-hop questions, to form the con- text for the (un)answerable queries. Table details the full statisties of our test sets across all three benchmarks. These test sets are obtained from the development set of each respective benchmark.
Evaluation The main task over which we eval- uate models is the (un)answerabillit classifica- tion task. When evaluating QA models over this task we only examine whether they tried to an- swer, i.e., we count every example for which the model provides an answer as an instance of answer- ability prediction, and each example for which the
After analyzing the models' responses, we curated a list of answers that signify abstaining from answering. See Ap- pendix A for further details.

---

SQuAI)
NQ
MuSiQue
boan
Flan-T55a
Regular
55.4
58.6
22.5
27.0
41.2
48.5
(IIB)
+hint
86.5
89.6
73.3
77.0
63.5
70.5
Flan-UL2
Regulur
59.5
62.3
21.7
27.3
35.3
43.4
(20B)
+hint
87.8
90.5
70.5
74.5
55.3
62.2
OPT-IML
Regular
57.8
60.6
29.0
33.0
37.3
44.4
(30B)
thint
47.7
54.5
Zero-shot
Flan- 15ot
Flar-UL2
OPT-IML
SQuAD
NQ
MuSiQue
SQuAD
EM
P5an (IIB)
Regular
61.8
65.2
23.8
28.2
41.3
48.9
DENT
(6.1)
(5.7)
(1.4)
(1.3)
(2.9)
(2:9)
hint
62.9
(1.6)
(1.4)
(0.4)
(0.3)
(2.7)
(2.6)
Flan-UL2 (20B)
Regular
69.8
73,0
26.1
31.2
40.0
47.5
30
(3:4)
87.9
90.7
70.7
74,9
60.1
67.4
20
(0.4)
(0.3)
(0.2)
(0.2)
(0.2)
(0.2)
10-
OPT-IML (30B)
Regular
(0.6)
(0.6)
(0.7)
(0.7)
(1.3)
(1.4)
0
hint
74,5
76.6
73.6
75.9
36.7
44.3
Flem- Flen-15ur
Flan-UL2
OFT-MIL
(2.3)
NQ
(b) Few-shot
beart Scc
30
Table 3: Exact match (EM) and (token) F1 scores over the QA task in zero-shot and few-shot setting. For each model, there are two prompt variants: regular and with a hint of the possibility of (un)answerabblityy In the few-shot setting, results averaged across three vari- ations of in-context-leaming examples (with standard deviation in brackets). Bold marks the better prompting method.
0-
Flon-T5ur
Flen-UL2
OPT-IML
MuSiQue
Figure 3: over the (un)answerability classification task with beam relaxation. In this setting, the models were considered successful reply acknow acknowledging the (un)answerability of question was found any anywhere in the heam. The horizontal lines show the Fl for the usual metric, i.e., successful classification only the correct reply was the top of the beam.
rics). The improvement over the QA task arises in large part from models giving the correct response to (un)answerable questions. We observe age drop of 8.3% in F1 and 7.1% in exact match over answerable questions in zero-shot setting when providing the hint (see Appendix B for all the results over answerable questions).
Appendix D for a comparison of the two possible hints: in the instructions, and as an (un)answerable example.
5.1.2 Few-Shot Scenario
Table 2b provides an exhaustive overview of the results in the few-shot setting. In order to miti- ex-
gate the impact of the chosen examples, periment with three variants of in-context exam- ples for each benchmark, and report the average results, as well as the standard deviation. 4 Mirror- ing the trend seen in the zero-shot scenario, when the prompts encapsulate a hint towards the poten- tial of (un)answerabilliy. there is a significant im- provement in the identification of (ur)answerable queries. This trend is further corroborated by Ta- ble 3b, which reports the exact match and token- wise F1 scores of the models over the QA task. See
5.2 Beam Relaxation
Figure illustrates the models' ability to detect (un)answerable queries, when gradually increas- ing the beam size. Although the increase in beam size yields negligible impact on the final, most probable response (as depicted by the horizontal lines in Figure 3). it shows better recognition of (un)answerability. This illustrated by consis- tent increase in the presence of (un)answerabilit- acknowledging responses within one of the beams
Please refer to Appendix A for comprehensive overview of such responses.
"See Appendix F for further details.

---

Notably, we conjecture that the observed de- crease in performance on the NQ and MuSiQue benchmarks, compared to SQuAD, can be at- (a) SQuAD
(a) SQuAD
(b) NQ
Nb)
(c) MuSiQue
Insrowerralee
Unammarraiio (unidentited)
Ansuerahie
Anosershie
(dentifee)
lisertitee]
(undentteee
Figure 4: 3D PCA projection of the last hidden layer's embedding of the Flan-T5.v model each of the three benchmarks. The left images show the embeddings with the regular prompt, and the right ones with the prompt with hint.
(c) MuSiQue
Unarowerable
Answerable
(slentified)
(unidcntificl
lidentifical]
(urideniifes
Figure 5: 3-D PCA projection of the last hidden layer's embedding of the OPT-IML model on each the three benchmarks. The left images show the embeddings with the regular prompt, and the right ones with the prompt with hint.
(signified by the height of the bars). This observa- tion underscores the notion that beneath the facade of overconfidence expressed by these models, the models do encode their inability to respond to cer- tain queries. Importantly, find that this approach has very little negative impact on the answerable questions, with only slight degradation in the exact match and scores (see Appendix for further details).
queries.
In addition, the MuSiQue dataset introduces a significant challenge by requiring multi-hop reason- ing. There are limited datasets available on which models can be trained for such complex tasks, and even fewer with (un)answerable questions. This scarcity, coupled with the demand for multi-hop reasoning, amplifies the difficulty of MuSiQue. This high complexity is further highlighted by the diminished performance of models, even when re- sponding to answerable questions, as evident in Tables and 9 in Appendices B and E, respectively. This drop in performance shows how challenging these benchmarks are, especially when compared easier ones like SQuAD.
tributed to two main factors: distribution shift and a more challenging task environment. One con- tributing factor is the non-conventional format of queries in NQ; unlike the typical question format found in datasets like SQuAD, NQ queries do not always adhere to this pattern. Language models (LLMs) primarily trained on question-answering datasets, like SQuAD, might struggle with this dis- tribution shift, leading to a decline their perfor- mance when faced with non question-formaalee
5.3 Identifying an Answertability Subspace
We report the performance of the lincar classifiers in Table 4. Notably, when considering the stan-

---

SQuAD
NQ
MuSiQue
Model
1
last
last layer
layer
layer
layer
layer
layer
89.9
74.9
75.0
Flan-T5sd
regular
40.1
89.9
23.0
86.1
47.4
77.5
(IIB)
+hint
40.0
89.4
26.6
86.2
38.6
77.3
70
Flan-UL2
regular
39.4
90.4
42.2
87.3
15.1
78.3
60
(208)
39.6
89.9
41.5
78.3
NO
84.1
86.1
76.1
OPT-IMI
regular
48.4
82.8
40.8
85.5
45.6
75.5
50
(30B)
thint
48.4
83.9
45.3
86.2
45.6
84.9
40
Musiquee
Table 4: Fl scores of (un)answerability classification of the linear classifier trained for each model-dataset pair, once with regular prompt and once with the prompt that hints at the possibility of (un)answerabillty, For each model, we classify once based on the first layer and once based the last layer of the first generated token.
71.5
65.5
77.5
SQUAD
NQ
MuSiQue
Flan Flan-T5xxl
SQUAD
90.4
85.6
64.7
dard prompt, the Fl of the probe is above 75% for all models and datasets. Furthermore, we find that hinting the possibility of (un)answerabillty only marginally improves the ability to correctly classify queries from the representations within the models. These suggest the existence of an (un)ansserability" linear subspace.
NO
82.4
87.3
82.5
50
40
MusiQque
73.4
73.5
78.3
+30
SQUAD
NO
MuSiQue
Flan- UL2
Visuazliation. To examine this hypothesis, perform a PCA projection of the embedding of the final hidden layer of the first generated token onto a 3-D plane. Figures 1, 4, 5 display the re- sults for the UL2, Flan-T5.x and OPT-IML models, respectively. Consistent with our hypothe- sis, it can be observed that (un)answerabbe queries, which were correctly identified model (depicted by red dots in the figures), are distinctly separate from the answerable queries (represented by blue dots in the figures). This separation becomes especially pronounced in the context where the prompt incorporated hint (as illustrated in the right subfigures). Importantly, we find that (un)answerable questions, which the models failed recognize such and instead generated hallucinated response (indicated by pink dots in the figures), appear reside within a separate linear subspace. This finding demon- strates that, notwithstanding the overconfidence exhibited by these models, they intrinsically pos- sess the capacity to distinguish (un)answerable queries. This intrinsic capability is particularly evident given that the subspace corresponding hallucinated (un)answerable questions (pink) seems be positioned between that of the answer- able queries (blue) and that of correctly identified (un)answerable queries (red). This positioning is suggestive of the models' inherent uncertainty.
40
SQUAA
82.8
78.0
17.4
NO
84,4
85.5
41.5
40
such by the
Musilque
60.0
75.5
30
-20
SQUAD
NQ
MusiQue
OPT-IML
Figure 6: Fl scores of (un)answerability classification, as determined by linear classifier trained for each model-dataset pair, and tested on the other benchmarks. Within each heatmap. the column designates the dataset used for training, while the illustrates the dataset on which the classifier was tested.
Trasnfer Between Datasets. In Figure we present the transferability of the (un)answerability classifier trained on a given dataset to other datasets. While performance deteriorates. the Fl scores still well above the F1 scores calculated over the uncontextualized first layer. This suggests that, to a large degree, the probes identify an abstract (un)answerability subspace beyond dataset-specific shallow features.

---

All
Answerable
k-beam Type
EM
F1
EM
F1
Regular
w/o crasure
60.2
63.8
87.1
94.1
with crasure
50.2
55.1
82.0
91.8
Relaxed
w/o crasure
60.9
64.4
87.1
94.1
with crasure
50.8
55.7
82.0
91.8
Table 5: Exact match (EM) and Fl scores of all ques- tions and of answerable questions in the zero-shot ting for the Flan- UL2 model on the SQuAD benchmark with a beam size formance before and after the application of the concept crasure, for the regular k-beam decoding approach, and the relaxed variant.
Before Erasure
After Erasure
Unarowerabic
Unanowerable
Answerable
Answerable (undenniiis)
(ddnttffed)
(unidentifidl
fidd)
The results demonstrate the per-
Figure 7: 3-D PCA projection of the last hidden layer's embedding of the FLan-UL2 model on the SQuAD dataset, without performing erasure (left) and after (right).
5.4 Erasing the Answerability Subspace
the problem answered (un)answerable questions can be mended with either better prompting, better decoding, or simple auxiliary classification models.
Recall that if the subspaces we found are causally related the predictions of the model, we expect that erasing them would deteriorate the model's Iss per- formance in the answerability task. Indeed, when linearly erasing the answerability subspace from the first token representation of Flan-UL2, we see the Fl score over the (un)answerabiliic classifica- tion task decreasing from 50.1 1 to 31.2 with regular beam, and from 65.4 32.7 with beam relaxation. This trend is also evident from the results on the QA task presented Table 5, as well as when pro- jecting the embeddings on a 3-D plane using PCA, as depicted in Figure 7. This suggests that the answerability subspace is influencing the model's behavior in the context of the answerability task.
Limitations
We focus few datasets and models. Despite the effort experiment with several models, future work should experiment with different models, and in particular, examine the relation between the abil- ity to encode (un) Junaaweerrbyl and model scale. We also do not compare the different approaches explored in this paper, which we leave as an in- teresting future research direction. Our focus on linear probing and linear erasure stems from the availability of existing methods from this family, but deep LMs are highly nonlinear and may encode the information we are interested in a nonlinear manner. As such, our results should only be inter- preted as a lower bound for the identification of (un (un)answerability Lastly, our experiments focused on (un) (un)answerability in given context. Future work should also explore the phenomenon in the open-domain setting.
6 Conclusion
We found ample evidence for LM's ability en- code the (un)answerability of questions, despite the fact that models tend to be over-confident and gen- erate hallucinatory answers when presented with (un)answerable questions. We also showed that this discrepancy between model output and its hidden states is mitigated by simply adding the option of (un (un)answerability the prompt.
8 Ethics Statement
The evidence we found includes the existence reply acknowledging the (un)answerability in a beam of decoded answers, meaning that even though the models' best-assessed answer is halluci- natory, the true answer is not lagging too far behind. We also showed that the models' representations after encoding the question and before decoding the answer are highly influenced by the answerabil- ity of the question or lack thereof, with answerable and (un)answerable questions being lincarly sep- arable the embedding space. We conclude that
Model hallucination. in general, can have real- world implications when models are incorporated in, e.g., search engines or other applications. Our study focuses on the ability to discern a specific type of hallucination in selected set of models and datascts. It should not be taken as general solution the problem of hallucination in the QA setting, but rather as preliminary research on po- tential techniques for mitigating the problem of hallucination.

---

References
Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Nlya Sutskever, and Wojeiech Zaremba. 2021. Evaluating large language models trained on code.
Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298-306.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representa- tions (ICLR).
and Yoav Goldberg. 2017.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro- Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction -innttuuon-nenteeeddd language models.
Alfonso Amayuelas. Liangming Pan, Wenhu Chen, and William Wang- 2023. Knowledge of knowledge: Ex- ploring known-unknowns uncertainty with large lan- guage models.
Galen Andrew and Jianfeng Gao. 2007. Scalable train- ing of L1 L--reuularized log- log-linear models. In Proceed- ings the 24th International Conference on Machine Learning, pages 33-40.
Akari Asai and Eunsol Choi. 2021. Challenges in information-sceking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 1492-1504, Online. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you can cram into single S&.#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 2126-2136, Melbourne, Aus- tralia. Association for Computational Linguistics.
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. Leace: Perfect linear concept erasure in closed form. arXiv preprint wxxh::2366.08817
Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 295-302, Online. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpuro- hit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned lan- guage models.
Shehzaad Dhuliawala, Leonard Adolphs. Rajarshi Das. and Mrinmaya Sachan. 2022. Calibration of machine reading systems at scale. In Findings of the Asso- ciation for Computational Linguistics: ACL 2022, pages 1682-1693, Dublin, Ireland, Association for Computational Linguistics.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural lan- guage models. In The Eleventh International Confer- ence on Learning Representations.
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. 2023, Jump to conclusions: Short- cutting transformers with linear transformations. arXiv preprint ar arXiv:2303.094.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try. Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings. Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel HerberttVosss William Hebgen
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral expla- nation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160- 175.
James Ferguson, Matt Gardner, Hannanch Hajishirzi. Tushar Khot, and Pradeep Dasigi. 2020. IRC: A dataset of incomplete information reading compre- hension questions. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language

---

Processing (EMNLP), pages 1137-1147, Online. As- sociation for Computational Linguistics.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. arXiv preprint 2.dd::2211004111
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir
Globerson. 2023. Dissecting recall of factual asso- ciations in auto-regressive language models. arXiv preprint 147777730.
Tom Kwiatkowski, Jennimaria Palomaki. Olivia Red- field, Michael Collins, Ankur Parikh. Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wo Wei Chang. Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Com- putational Linguistics, 7:453-466
Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoav Goldberg. 2022a. LM-debugger: An interactive tool for inspec- tion and intervention in transformer-hased language models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 12-21, Abu Dhabi, UAE, Association for Computational Linguistics.
Rémi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Lespiau lean-Baptiste, Ioannis Antonoglou, Karen Simonyan, and Oriol Vinyals. 2021. Machine translation decoding beyond beam search. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 84 8410-8434, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold- berg. 2022b. Transformer werr-f layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. pages 30-45, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
John Hewitt and Percy Liang. 2019. Designing and in- terpreting probes with control tasks. Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)), pages 2733-2743.
Belinda Li, Jane Yu, Madian Khahsa, Luke Zeutlemoyer, Alon Halevy, and Jacob Andreas. 2022. Quantify- ing adaptability in pre-trained language models with 500 tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages Association for Computational Linguistics.
Seattle, United States.
Srinivasan Iyer, Xi Victoria I Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christo- pher Dewan Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2023. Opt-iml: Scaling language model instruction meta learning through the lens of generalization.
Jinzhi Liao, Xiang Zhao, Jianming Zheng, Xinyi Li. Fei Cai, and Jiuyang Tang. 2022. Ptau: Prompt tuning for attributing unanswerable questions. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '22 page 2229, New York, NY, USA. Association for Computing Machinery.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1 38.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Inves- tigating effectiveness and limitations of paramet- ric and non- parametric memories. arXiv preprint 2121v:2212.10511.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig, 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Associa- tion for Computational Linguistics, 9:962-977.
Potsawee Manakul, Adian Liusie, and Mark J. Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hattield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage. Tristan Hume, Anna Chen. Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know.
Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. If beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 2173-2185, Online. Association for Computa- tional Linguistics.
Kevin Meng. Arna Sen Sharma, Alex Andonian. Yonatan Belinkov, and David Bau. 2023. Mass- editing memory in a transformer. In The Inter- national Conference on Learning Representations (ICLR).

---

Timothee Mickus, Denis Paperno, and Mathieu Con- stant. 2022. How to dissect a Muppet: The struc- ture of transformer embedding spaces. Transactions of the Association for Computational Linguistics, 10:981-996.
crasure. In International Conference on Machine Learning, pages 18400-18421, PMLR.
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Swaroop Mishra. Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022a. Reframing instructional prompts to GPTk's language. In Find- ings of the Association for Computational Linguisties: ACL 2022, pages 589 612, Dublin, Ireland. Associa- tion for Computational Linguistics.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani. Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Met- zler. 2022. Confident adaptive language modeling. In Advances in Neural Information Processing Sys- tems (NeurIPS).
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. 2022b. Cross-task generaliza- tion via natural language crowdsoureing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties ( (Volume Long Pupers), pages 3470- -387,, Dublin, Ireland. Association for Computational Linguistics.
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta. Jesse Dodge, and Noah A. Smith. 2020. The right tool for the job: Matching model and instance complexitie.. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640-6651, Online. Association for Computational Linguistics.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Mcasuring stereotypical bias pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 5356-5371, Online. Association for Computational Linguistics.
Aviv Slobodkin, Avi Caciularu, Eran Hirsch, and Ido Dagan. 2023. Dont add, dont miss: Effective content preserving generation from pre-selected text spans.
Aviv Slobodkin, Leshem Choshen, and Omri Abend. 2021. Mediators in determining what processing BERT performs first. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 86 93, Online. Asso- ciation for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't 't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisticy (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Elior Sulem, Jamaal Hay, and Dan Roth. 2021. Do we know what we don't know? studying unanswerable questions beyond SQuAD 2.0. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4543 4548, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-239) Austin, Texas. Association for Computational Linguisties.
Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg, 2020. Null it out: Guard- ing protected attributes by iterative nullspace projec- tion. In P Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-u up evolution of representations in the trans- former: A study with machine translation and lan- guage modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natu- rul Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLE) pages 4396 09664406, Hong Kong. China. Association for Computational Linguistics.
Shauli Ravfogel, Yoav Goldberg, and Ryan Cotterell. 2022a. Linear guardedness and its implications. arXiv preprint arXiv:2210. 10002.
Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. Counterfactual interventions re- veal the causal effect of relative clause representa- tions on agreement prediction. In Proceedings of the 25th Conference on Computational Natural Lan- guage Learning, pages 194-209.
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023a. Poisoning language models during instruction tuning. arXiv preprint arXiv:2305.000)
David Wan, Mengwen Liu, Kathleen McKeown Markus Dreyer, and Mohit Bansal. 2023b. Faithfulness- decoding strategies for abstractive summarization. arXiv preprint arXiv:2303.0322 03278.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. 2022b. Linear adversarial concept

---

Johannes Welbl, Amelia Glaese, Jonathan Uesato. Sumanth Dathathri, John Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Cop- pin, and Po-Sen Huang. 2021. Challenges in detox- ifying language models. In Findings of the Associ- alion for Computational Linguisticx: EMNLP 2021, pages 2447-2469, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Orion Weller, Mare Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2023. according to..." prompting language mod- els improves quoting from pre-training data. arXiv preprint arXiv:2305 1325.
Gian Wiher, Clara Meister, and Ryan Cotterell. 2022. decoding strategies for neural text generators. Transactions of the Association for Computational Linguistics, 10:997-1012.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. large language models know what they don't 't know?
Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu. 2019. Learning ask unanswer- able questions In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 4238- 4248, Florence, Italy, Association for Computational Linguistics.
machine reading comprehension.

---

A (UnlanswerabiityyRoggiiii Responses
SQuAD
NQ
MuSiQue
EM
F1
Flan-T5xa
Regular
88.0
94.3
65.5
80.0
66.1
78.4
(IIB)
+hint
86.2
92.3
56.3
68.2
61.2
72.9
After analyzing the responses generated by the dif- ferent models in this work, we curated list of answers that signify abstention from answering, which used to identify responses that signify (un)answerability. This includes: "Unanswerable", "N/A", "I don't know", "IDK", "Not known", "An- swer not in context", "Unknown", "No answer", "It is unknown", "None of the above", "None of the above choices", "The answer is unknown", along with their corresponding versions in lowercase.
Flan-UL2
Regular
89.0
94.7
53.1
71.2
56.7
70.1
(20B)
+hint
87.8
93.2
54.3
67.0
52.1
63.6
OPT-IML
Regular
87.6
93.2
66.1
78.9
56.1
67.9
(30B)
+hint
80.2
84.9
53.9
52.0
(a) Zero-shot
SQuAD
NQ
MuSiQue
EM
F1
FM
F1
EM
F1
Flan-T3xxl (IIB)
Regular
87,4
94.2
66.3
80,3
64.2
76.9
(1.9)
(1.1)
(1.9)
(1.6)
(3.1)
(3.0)
hint
84.6
91.1
55.5
67.3
57.4
(2.8)
(2:4)
(5.1)
(5.5)
(5.4)
(5.3)
Han-UL2 2 (20B)
Regular
88,3
94.6
58.3
74.7
61.7
74.2
(0.3)
(2.1)
(3.8)
B Performance on the Answerable Instances
+ hint
87.1
92.7
59.0
72,4
56.4
68.6
(1.0)
(0.6)
(0.1)
(0.2)
(0.8)
(0.7)
OPT-IML (30B)
Regular
83.0
89.6
65.2
(0.5)
(0.4)
(0.7)
(0.8)
(1.8)
(2.0)
Table show the exact match and Fl scores of each model over the QA task only on the answer- able questions of each benchmark, in the zero-shot setting and the few-shot setting. Note that although the addition of the hint hinders the models' per- formance, the drop over the answerable questions is small for the most part and outbalanced by the improvement of detection of (un)answerable ques- tions, leading to the overall improvement shown in Table 3.
+ hint
69.6
73.8
48.8
56.3
50.3
63.0
(5.0)
(5.7)
(3.2)
(4.3)
(1.3)
(1.2)
Few-shot
Table 6: Exact match (EM) and F1 scores over the QA task only for answerable questions in zero-shot and zero-shot setting. For each model, there are two prompt variants: regular and with a hint of the (un)answerability. In the few-shot setting, results averaged across three variations of in-contett-leaminng examples (with stan- dard deviation brackets). Bold marks the better prompting method.
C Prompt Variant Tuning
SQUAD
NQ
MuSiQqee
our work, we experiment with three variants of the prompt containing a hint of the possibility of (un)answerability. These are:
Model (LIB)
Prompt Variant
Cxact
F]
Cxact
F]
Exact
F]
71.5
60.5
67.2
52.5
61.2
37.5
44.9
78.0
84.2
61.5
69.6
67.0
73.1
Flan UL2 (208)
84.0
90.0
60.5
69.0
56.0
63.5
74.5
80.5
55.0
63.6
48.5
56.2
1. Given the following passage and question, an- swer the question. If it cannot be answered based on the passage, reply "unanswerable".
OPT-IML (30B)
83.0
86.6
69.0
74.2
46.5
52.9
59.0
65.2
52.5
58.3
39.5
46.3
63.9
Table 7: Exact match and (token) Fl scores in the zero- shot setting for three variants of the prompt containing a hint of the possibility of (un)answerability, on our development set.
2. Given the following passage and question, an- swer the question. If you don't know the an- swer, reply "IDK".
3. Given the following passage and question, an- swer the question. If there is correct an- swer, reply "N/A".
(on all models), and the third variant in our experi- ments on the MuSiQue dataset (on all models).
We run all three variants a separate develop- for each
D Impact of Hint Placement
ment set decide which prompt model and dataset. Table shows the results of all three variants on our development set. Based on these results, we decide to use the first variant in our experiments on the SQuAD and NQ datasets
To gain deeper understanding of the circum- stances under which the addition of hint in the few-shot scenario is most beneficial, two ablations on the prompts. In one gave hint only in the instructions with all examples answer- able, while in the other we gave special instructions but one of the two examples was (un Junaanwwrable.
conduct
"The development set consists of 200 answerable and 200 (un)answerable instances, extracted from the train set of each respective benchmark.

---

SQuAD
NQ
MuSiQue
to its regular counterpart. Like in Appendix B, these results point to the fact that any improvement achieved in $5.2 indeed stems from better treatment of (un)answerable questions.
Regular
52.6
8.6
(11B)
(11.7)
(2.1)
(4.9)
thint (E&I)
91.2
85.8
75.4
(1.0)
(0.5)
(1.4)
thint (I)
91.1
85.0
74.2
(1.0)
(0.2)
(1.3)
F in-Context-Learning Variants
+hint (E)
88.7
81.9
51.3
(2.1)
(0.8)
(7.2)
In order mitigate the effect of the chosen in- context examples in the few-shot setting, ex- periment with variants of in-context examples, and average their scores. Figure 8, Figure 9, and Figure 10 show the different in-context examples variants for the SQuAD, NQ, and MuSiQue tasks, respectively.
UL2
Regular
67.7
20.6
14.5
(20B)
(4.0)
(1.3)
(4.9)
+hint (E&I)
92.5
83.7
72.1
(0.1)
(0.1)
(0.6)
+hint (I)
92.3
82.3
71.3
(0.1)
(0.1)
(1.0)
thint (E)
92.0
79.7
59.0
(0.1)
(0.6)
(1.7)
OPT-IML
Regular
10.0
11.1
4.9
(30B)
(1.2)
(2.1)
(1.0)
+hint (E&1)
79.3
85.0
27.5
(0.3)
(1.5)
(8.0)
+hint (I)
75.0
81.9
(1.8)
(1.1)
(4.2)
+hint (E)
58.1
61.0
(1.4)
(6.9)
(3.7)
Table 8: Fl scores over the (un)answerability classifica- tion task in few-shot setting. Each model prompted with regular prompt, and with three types of hint- including prompts: only the instructions ("+hint (I)"), only in the exemplars ("+hint (E)") and in both ("+hint (E&I)"), Results are averaged across three variations of in-context-leamiing examples (with standard deviation in brackets). Bold marks the better prompting method.
As per the data presented in Table 8, it is evident that the inclusion of hint within the instructions is considerably more advantageous compared to its addition within the exemplars. Indeed, once hint is incorporated within the instructions, further inclusion within the exemplars has minimal im- pact on the results, with OPT-IML being the sole exception.
E Impact of the Relaxed Beam Search Decoding on the Answerable Queries
Table 9 associated exclusively with each model on answerable questions, evaluated under the frame- work of our beam inspection experiments (see Sec- tion $3.2). Two decoding approaches were em- ployed: a conventional beam-seardh decoding and its relaxed variant where the top answers are sup- planted by an sponse if it emerges within the beam. Our findings suggest a marginal impact of the adapted beam- search on answerable queries, with a maximal re- duction of 7.7 and 8.7 points observed the exact match and Fl scores respectively, when compared

---

SQuAD
NQ
MuSiQue
Model
Beam size
Regular
Relaxed
Regular
Relaxed
Regular
Relaxed
Flan-T5w.
1
88.0
88.0
65.5
65.5
66.1
66.1
(11B)
3
81.0
80.7
60.5
59.3
59.8
58.9
5
81.1
80.4
59.6
57.1
60.3
58.2
7
81.8
80.5
60.4
56.3
61.2
58.6
Flan-UL2
1
89.0
89.0
53.1
53.1
56.7
56.7
(20B)
3
79.4
77.4
48.4
47.8
53.4
49.1
5
79.2
76.6
48.0
46.0
55.8
50.7
7
79.7
75.9
48.1
44.4
57.3
51.4
OPT-IML
1
87.6
87.6
66.1
66.1
56.1
56.1
(30B)
3
82.4
79.4
64.7
63.0
50.6
48.0
5
83.0
76.8
64.6
60.2
51.3
46.5
7
83.6
78.6
64.4
56.8
52.4
44.6
(a) Exact Match
SQuAD
NQ
MuSiQue
Model
Beam size
Regular
Relaxed
Regular
Relaxed
Regular
Relaxed
Flan-T5xal
1
94.3
94.3
80.0
80.0
78.4
78.4
(1IB)
3
91.4
91.1
77.6
76.3
74.3
73.3
5
91.6
90.8
76.9
74.1
74.9
72.6
7
91.7
90.3
77.3
72.7
75.0
71.9
Flan-UL.2
1
94.7
94.7
71.2
71.2
70.1
70.1
(20B)
3
90.4
88.3
67.2
66.6
70.7
65.8
5
90.3
87.5
67.1
64.7
72.2
65.4
7
90.6
86.4
66.8
62.2
72.9
65.5
OPT-IML
1
93.2
93.2
78.9
78.9
67.9
67.9
(30B)
3
90.3
87.3
77.3
75.5
63.0
60.2
5
90.6
84.2
76.9
72.0
63.3
57.8
7
91.0
85.7
76.7
68.1
64.2
55.5
(b) F1
Table 9: Exact match (top table) and (bottom table) scores answerable questions zero-shot setting for different beam sizes. The results demonstrate the performance of two decoding approaches: the conventional beam-search method ("regular"). and a modified relaxed beam-search variant (" ("wltextbackslash relax."). "). the latter technique, the highest-ranking response is substituted by (un (unyannsweabillyyyrroonnnnngggg response, in where such response is present within the breadth of the beam.

---

Answorable Esamslal;
Paosage: Natoens salaased the Natarial dirl clathing lime, which she dosigned with her deighter, I narrdea, The 1980s imsired clothing line, barrowed frox Hadornal's 's punkegirl style when she rose to fare In the 1958s, nis relcased under the Hacyl's label. Madonna also opened serina of fi tress centers arnand the world rward Hand Carty Fitneox. Neweeeer 2811, Medorew and No lemn womunned the re hose of called Truth
underclothins, and accessories. She directed Kechishian. VIT Nallix Simpson; was comerittee Critical conercciel response to the riln wss negative. Modonna contributed the bollad "Nasterplece" the film''s soundtrack, which
fashian
Nadonna to include King
festure file, K.C., biographie
affair
son her Golden Globe waard for Best Original Sonp.
Questian: Naterial Girl clothing Line is relesses under which brand?
Anower: Hacy s.
anle Examolo2:
Passage: In 1919 Father James Burns becare president of Natre Dare, and in three years he produced acadensic revolution that brought the school national standards adopting the elective syster and aoving eaphasis, By
the university"'s traditional scholastic and classicsl relustant
lesiit callagea, bastions acadente cosservatise, continued to gnow over years, adding vore colleges, programs,
syatem electivea. Their
shut out larvard School for that reason, Notre and sports twoms, by 1925, with the addition the Col Tege of Commeree, hotre fiw had groan from a aull lollege to university with five colleges and professtonal law school. The university continued to expand and add new residence halls and butldings with each subsecuant president.
graduates
Question: Over naw many years did the change national standards undertaken at hotre Dane in the sarly 28th century take place?
Andwer: three years.
Un-anseerable Example:
Reseeee language custors to synthesize unious "Neraan" culture local people, bllending their religien Old language with Catholicise Frankish serittage nor th of France. hornan language was forged by the adcottion the indigenous langue d'ail branch of Rossnce by a Norse-spaoke spinking ruling class, and it developed into the regionsl langunge that survives todny. Questian: What was replaced with the horse religton?
Passage: descondants Bollo''s Vikings and their Frankiah wives sould replace
(Chrixtianity)
Answer: unanswerable.
Variant1
Answorable Examplel:
Passage: In November 2013 MEN and the Hotlory estate forrally settled the issue with Denjon, U.C-siste coepeny of ton Productions HiH and of the characters associated with it. With acaulsition continuity, SPECTRE acronyn discarded organisation
acquiring the full copyright r'ln to the organisation's re-intreduction
file rights
resraginedd as "Spectre".
Quration: which teo partins settled the Issue Novamber 20837
Answer: MGM and the HcClory estate.
Anowerabll Faweplat:
Passage: Gencee corposition is used to describe the make up of contents of haplotd genore, which should include genone size, proportions of mevorppttitie IMA itpetitive lhA in detaills ny comparinng ing the grnces coppoittios betwoen genones, scrient ints becter underatind the evolutionary history given
genone can Answer: evolutionary history.
Question:
compositions help resparchers learning about?
Unnaniserraaa ablo Esamgle:
Passsage:: story focmues in wer iew protagonist link who triew to present Hyrull from being engel Feed by corrapted parallel dimonsioe known an the Twilightt Reals. To de so, he takes the Form of both a Hyllan and a wolf, and is assisted by mysterious creature named Hidra. The game takes place handreds of years after Drarins of Time and Negora'ss Maok, in an alternste timeline from The Wind Waker.
Qusation: what land does Ocarlina serve to protect?
Answer: unanswrabble.
Variant2
Answerable Examalel:
Parsager: Neseann returned Spectre's composer.
composing
past- cest-proouction on,
worked during filming. The theatrical trailer released Zuly 2015 contained a rendition John Barry's On her Najesty's Secret Service those, Hindes revealed that final film would have nare than one hundred eirutes of rusic. soundtrack albus nas released on Octaber 2015 in the IK and Neverber 2015 in the USA on the Decca Records labe1. Quistion: who wrote the rusic. for spectro?
Andwar: Thomss Nevean.
Ansserabll Faampla?:
Passage: Oetween 64 and 184 iajorr aftershocks, ronging magnittude from 4.8 6.1, recorded within 72 hours of the main quske, According ranged
chi nese official pounts, "by 12:00
had 40, aftersheric,, M5; the strongest aftershock neasured 6.4 MS." latest aftersheck exceeding
V5. fron 5.0 to 5.9 and from 6.8
occurred on August 5, 2008.
Question: what do the Chinese say is the total number of shocks after the quake?
Answor: 42,719.
answerabll Example:
Passage: Both the number of base pairs and the nutber of genes vary widely fron one species to another, and there is only a rough correlation betssen the two (an observst ion known the Cruslue parwios). At present, the highest known number of in around 56, ,80, for the eukaryotic genceos). alrost three tires as in the ganaro.
Question: protozoan causing trichonanlasis
List species?
highest known Answer: unanswerable.
Variant3
Figure& 8: The three variants of in-context examples for the SQuAD prompts. For the regular prompts, we use the two answerable examples, whereas for the prompts hinting at the possibility of (un (un)aaswerability, we use the first answerable example and the (un)answerable examples. Additionally, for the other prompt variants, we replace the "unanswerable" answer of the (un)answerable example with "IDK" and "N/A", accordingly.

---

Anseerable Examplel:
Paosage: itppocia di-ffars. fros hyposenia and aosseia in that hypocia refees to a state in which neygen supply is insufficient aheneas hyposesia and aroxesia refer specifically to states that have low zero artenial oxygen supply. Hypooda in which there is complete deprivation of saplyy referred to as anacin deficiency not total Jock of oxygen
Question: nedical which hyposia.
Answerable Examole2:
Passage: South Africa have played six of the eight Righy World cup tournarents having been unable to corpeto
first tourranints due to sports buycott curing the apartheid erw fallowing the end of spartheid they hosted the 1995 luugh World cup and wan the tournarent and wire champlara again at the 2007 touraaint in France With two tournament wins they are one of the three best perforwing teses along with iastralia who have also von twice and New Zealand with three wins the only teom to do better
Garstion: whin did south africa first win the rughy world oup
Answer: 1985.
Un-answerablle Example:
Passage:
settle succession the Englissh Protestant in Line chrone Electress Soptila of Hanover arrdduughtor of Janes of Sootland and Tngland After her the crmans wnuld desceed only to nor-fooma Catholic heira
Settlement is of the Farliarent England that was passed crowns on Protestants only.
Questian: The noxt Roman in 1ine to the threns ?
Arower: unmnwerable.
Variantl
Answorable Enamplol:
Pawnsager: Louiien Srrmen born 25 Joly 1978 ) is an Figgt fish waran known for being the First huran have been born after concept ion by in vitro fertilisation IVF
Quenstion:
first
baby
Answer: 25 July 1878.
Answor ablo Examole2:
Passage: The 2018 College Football Playotf National Champtonship was a college foctball bosl gare that determined the national charpton in the
NDAA Division I Footbill Boel Subdivision for the 2017 season The Alabara crinson Tide defeated the Georgia Bulldogs 26 23 In overtive
AAlabana overcare 13 dericit at hairttine Tua Tagovatioa and Ca'Ron Payne were respectively naned the offensive and defencive
players of the same
Question: who son the callege football national choepionehip tonight ?
Orinsinn
Un-anseerable Example:
Passage: Ranch is Anerican conedy web television series starring Ashton Kutcher , Danny Masterson Debra Minger , Elisha Cuthbertt , and San Elliott that debuted in 2015 on Netfllx The show takes place on the fictionsl Iron River Ranch in the ficticious small town of Carrison Colorado detailing the life of the Benretts a dystunctional family consisting of two brothers , their rrancher father and his divorced wire and local bar owner While the opening sequence shows scenes from Duray Colorade and surrounding Ouray county The Ranch is filwed on sound stage in Tront of live aucience in Burbaank Califernia.. Each season consists of 28 opisodes brokon up into two each containing 10 spisedes
Questian:
series
rarch out
unsnnerabll..
Variant2
Answerable Examplet:
Pacssage: "Fool ( If You Think It. Over ) the sitle of popular wong original ly iely released in 1978 by Brisish winger songeriter Chris Rea Rea clse wrote the words and composed the ruste of the song which appears on his 1978 cebut albux Nhotever Iisppred to Benny Santini ? The singln is charting succeess in the USA sarned hin Grawy nosinstion as Bost hew Artist in 1979
Question: who sang fool if you think it over ?
Anawar:: Chris Rea.
Andwerable Examoled:
Misaissigai Freadio Donocratic Party
1964 branch of the populist Freedon Anerican political, party was organized by rivil Rights
Demooratie organization
Mississizpi froe Hi Hississiopi to challange legitinacy the regular Hississippi Democratic Party. , which allowed participation only by whites shen Africcn Americans aode up se percent of the state popul atiion
Quistion: why did the aissiasipii freedon desceratic party onerge at the democratic party convention in 1964 ?
Answer: to lenge the legitimacy of the regular Pissiesippi Democratic Party, which allowed participation only by whites when Africcn Americans nade 40 percent of the state population.
Un-ansserabll bble Example:
Passage: Orinng
which the United There also and Northern Ireland fore guiding way into being. there logally defined to the UK Minister the Urited Kingdes and the UX Statistics Authority describe the Uei bed kingson as heing sade up four countria les one these being Worthern Trelied Other pages on the same teb refer Northern Ireland specifically as a province as do publications of the IK Statistics authority The website of the Worthern Ireland Seatisties and lesearch Agency whso refara to bortheen Irwland as being province as does the webxite of the affice of Public Sector Information and other apenctes within Northern Ireland Fuelications of HM Treasury and the Dipartient of Finance and Personnel of the korthern Taeland Exenutive
the Northern Ireland
Northern
amongst
geverneent For exampl. , the wobsitos of the office the
the other hand describe Nor thern Ireland being a of the UK The UK 's submissioo the United Nations Conference the Standerdisstion Geographical province Northern Iraland
deflnes
up of two countries England and
Scotland , principality Males Question: why northern ireland not part ireland Answer: unanswerable.
Variant3
Figure 9: The three variants of in-context examples for the NQ prompts. For the regular prompts, we use the two answerable examples, whereas for the prompts hinting at the possibility of (un (un)answerabiilty, we use the first answerable example and the (un)answerable examples. Additionally, for the other prompt variants, we replace the "unanswerable" answer of the (un)answerable example with "IDK" and "N/A", accordingly.

---

Ansnerable Examplel:
Passage: Paragraph I: South Africa have played at six of the eight Rubby Norld Cup tournanents, having been unable to conpeco in the First two inurnasents dua sparts boyoots during the aoartheid are. Fallowing the end of apartheid, they hosted the 1695 hughy World Oup and the
Paragraph with two tournament wirs, South Africa thrse
of the three parforking toams, slong with australis who do
Zeal Questian: Hav sany times did the winner of the 1995 Righy World Cup win in total?
also
only
Anower: tao timex.
anseerable Examole2:
Paesager Paragraph 1: Btewek Ohoen in an imerican politicion who served as the 44th president of the United States Fros 2000 to 2017. Pargoraph 2: Oboma sarried Michelle on October 3, 1992, after being engaged for stmest a year. Paragraph 3: Barack Obana was born in Honolule, Hawail. After gradaating froe Colunbia university in 1983, he verked comsunity organizer in Chicago.
Question:
of wire of the American president who born Hawail?
Nichulle.
Un-answer able Example:
Passage: Paragr oph 1: Barack Ohiea an American pelitician served as the 44th president of the United States From 2000 to 2017. Pargaraph 2: Obown sorried Michelle on October 5, 1992, after being engaged for almost a year
Qusstion: Mhat ia the nare of the wife of the karrican president who was barn in hiw York?
Answer: unanswerable.
Variant1
Ansnorable Examplel:
Passage: Paragraph 1: teast well-knoe snack Singapore, Kays toast prepared with (coconut jan), topping of coconut eass,
and sonctiiess nargarine butter. Koya ganerally considered breakfast staple, remains popular in Singapore, The dish is soretines dipped into soft-booledd egs with little dark soy appointed by the
also scnetines crackers. It is
sauce white pesper. Faragraph A sustice
peace in Singapore derives his powers from statuss las.
President of the ispublic of Singupore, under the provisioos of section 11 (1) the Suhordi nate Courts Act (Cap. 371). The Presioent may
reveko the agpointaent of any justice of the peace. A nexly appointed justice of the peace is required by soction 17 of the Subordinate
Courta Act, to take the oath of office and al allegiance as aet out in the schedule to the Schordinate Churts Act, before exercising my the funccions of his office.
becose jistiee of prace in the Answer: aspeinted by the President
ions low
where Keya toost ix popular?
Republic Singapore.
Anowerable
Passage: Parograph 1: Nount, Henry Located the Loris Range, Glacser National, Pork the U.S. state of Montana. Mount Henry is just south Applstoki Pealc
Two Medicine region of the park. Parayr aph K3RZ-LP (185.3 FPC was a radio station in Libby, Montana. It wis owned and operated by the Libby Area Chanber of Comeerce. Faragrash 3: The Londs Range is nountain range located in the Rocky Hountains of northern Montana, United States and extrore southern Alber ta, Canada, It was foraid as a result of the Levis Dvar yoarthratt, a goologic thrust fault resulted in the over lyirlying of younger Cretacemus rocks by older Proterorate racks The range is located within Waterson Lakes National Park in Alberta, Canada and Glacier National Park and the Bob Harshall Nilderness Complex in Hontana, deited States. The hightst peak is Pount Elevelard at
group the range which Henry froe Answer: Rocky Nountaiis.
Question: In
KIR7-IP is lonated is part?
Un-andwerable Exanple:
Passage: Porograph 1: wODS (183.3 3 Miz) known on-air as 103.3 MPP RadLo is a comerctal IM radio station in Boston, Passachuretts. WODS airs a Top 40 (CHR) radio format, and is owned by Entercos. cos. Its studios and offices are located on Les N. Birninggan Parkwary in Br Brighten.
Paragrach 2: The Frbwwaay of the United States to the Repiblic of Indonosia is loosted in lakarta just south of the Monas at lal an Moden Kerdoka Selatan. Par aaragrach 3: Nestainster Dolloge is private Liberal arts college located in the Sugar House reighborneod of Salt Like City, Utah, united States. The anl loge porprises four scheol at the School of Arts Seiemons, the Bill and Vieve forw School of fusinness, the School Education,
Health Sciences. It is the only accredited liberal lindd Thorpson Ridge, horlet County, New York, United States. It late 18th-century building Later modifled an the Greek Revival style.
School of hursing
college the in Orange
Leah. Paragraph
of
Question: What is the business category of Crowford Houue, located in the some city as R0DS and the some state as Nel Wellesley College in Mona Liss Swile?
Anower: unanswerable.
Variant2
Answorable Examplel:
Passage: Paragraph 1: Moot He In St. Louis is a rusical filn nade Netro Goldwyn Mayer and released In 1944. Divided into series of vignettes, starting with openinx the Louisiana Purchase Exposition
1983, relates the story year the Life the Srith fanily
spring Louis, leading The picture
comsnnly referred Brewer, Drake, Leen the Worla''s Morjorie Main, Paragraph 2: Gracie is 2007 Assrican sports drasa fils directed by lavix Giganeain, It atars tarly Schronder as Gracie Boamn Dermet Pulrorey as tryan Oowen, Elissbeth shue as Lindsay Dowen, Jesse Lee Soffer as Joheny Bownn, and Andrew Shue as Coach Owen clark. Poragraph 3: He was born thilip Davis Guggenhein in st. Louts, Nissouri, United States, the son of Harion Dovis and fiin cirecter and producer Chorles Guggenheie. ilis father was Jexish, whereas his wother was Eniscopolion. lee graduated from the Potorac School (Ncteon, Virginia) (1573), from Sideell Friends School (1982), and fron Brown University (1886). Question: when Meet in the birthplace Gracse's director take place? with
Judy Garland, Hargaret Brirre,, Mery astor,
Laddart,
Carroll,
1983.
Faampla?:
Passage: Paragraph Is The city has a Mayor and one of the 16 cities towe in England Wales have a coramental sherlff who acts a deputy for the Nayor. The current and 10rrd Meyor of Southeppoon ix inds Horris Cather inn Heiwing is the current and 578th sherriff. The town crier froe 2014 until his death in 2014 was John Heicdy, who acted as muster of cerenontes in the city and who possessed a cry of 1e4 decibels. Is. Paragraph 2: John May (heen 78 Septeeber 1845 in Southoepton, loepehire; date of deach unkonar) was an Enggi cricketer. May sas a right-handed batsran who was right-are fast bowler.
Question: who the current rayor the birthplace of John Hay?
Andwer: Linda
Unnanswer abla Esample:
Passage: Parograph 1: Innan ihan has held the office of Prine Minister since August 2018. followinng the outcome of nationnide goneral
Group of Homuenta at heppi, is (NESCO world ttge Site located in east - central Karrataka, India, It become the centre of the lind Vijayaragas Impire crpital in the 14th century. Chronicles left by Persian and Furopein trmellara, particularly the Por ougase, state happi aas prosparous, walthy and grand city mar the Tungsbharra River, with nuserous temples, farms and trading sarkets. by 1588 0E, Hampl Vijoyangaass was the wer ler'd's second largest redteval- - era city after Beifing, and probably India''s richest at that time astracting traders free Persia and Portugal. The Empire defeated Hanpl renaiied ruins, Faragrash Ohcer Kinister and Deputy Ohler Minister. Paragraph Thekkody (lousk) district) is the location the Periyar National Park, which is an iiportant tourist attraction in the Korala state of India.
elections held on 25 July 2015. Paragraph Hampi, also reforred to
coalition of Muslin sontonstes; its copital of
conquered, pillaped destroyed sultenate amites the Goverment Karnataka consists 27 nintsters including
1955, after
Question: An 2818, who is the winixter of the state where hampi tourist place is Tocated?
Answor: unansworable.
Variant3
Figure 10: The three variants of in-context examples for the MuSiQue prompts. For the regular prompts, we use the two answerable examples, whereas for the prompts hinting at the possibility of (un (un)answerability, we use the first answerable example and the (un)answerable examples. Additionally, for the other prompt variants, we replace the "unanswerable" answer of the (un)answerable example with "IDK" and "N/A", accordingly.


Reviews:
[{"review_id": "RiWNTUE8ho", "reviewer": "Reviewer_t5Sd", "summary": "", "questions": "1. Why do we want to erase the answerability subspace..?\n2. In Figure 3: although the trend persists across all datasets, I can observe relatively poor performance in both NQ and MuSiQue in comparison to the Squad dataset. What could be the reason for this? \n3. In line 149, 150, you wrote: “three experimental approaches: i) prompt manipulation, 2) beam scrutiny, 3) probing”, but this does not seem to align with the following content (probing is missing). Is this a mistake? \n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive feedback and appreciation of our work. \nHere are our responses to the reviewer’s questions and comments:\n\n***Why are the erasure experiments needed?*** \n\nThe probing experiments alone show that *there exists* a subspace in the representation space that encodes answerability. This, however, does not necessarily show that the subspace is being utilized by the model when deciding whether or not to output an answer. In other words, the information can be present in the representation and at the same time be irrelevant to the model’s behavior [1]. By erasing this subspace and examining the effect on the model’s output, we can verify that the subspace we identify is behaviorally meaningful, opening an avenue for performing interventions in that subspace in order to modify the model’s behavior. \n\n[1] Elazar, Yanai, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. \"Amnesic probing: Behavioral explanation with amnesic counterfactuals.\" Transactions of the Association for Computational Linguistics 9 (2021): 160-175.\n\n***Dividing Method section into subsections***\n\nWe agree with the reviewer’s comment that such a division might be useful. However, as highlighted in lines 491-494, an answerability classifier *can*, in fact, be used to filter unanswerable questions, in a reliable manner which also transfers between datasets (Figure 7). As such, we do not see it as a mere analysis tool. Nevertheless, in light of the reviewer’s comments, we would consider a better layout of the sections for the final version of the paper.\n\n\n***Figure 3  -relatively poor performance in both NQ and MuSiQue compared to Squad:***\n\nWe hypothesize that the trends towards decreased performance are due to distribution shift and a more challenging setting:\n* NQ’s queries do not always appear in a question format. LLMs trained on QA datasets, were mostly introduced to unanswerable questions via datasets like Squad - where questions are in a question format. This distribution shift may cause decreased performance.\n* MusiQue dataset’s requirement for multi-hop poses a significant challenge - theren’t many such datasets on which models could have been trained, and even less so with unanswerable questions. This is evident also in tables 2&3.\nFurthermore, this decreased performance can be observed in the model’s decreased performance even over answerable questions (tables 6&8), which further underscores the challenging nature of these datasets, compared with Squad.\n\n***three experimental approaches vs. 4 in the paper:***\n\nBy probing we refer to sections 3.3 and 3.4. We will clarify this towards the final paper version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. **Important Problem:** Hallucination is an important and timely problem that needs to be solved for the safe deployment of LLMs. I especially value the fact that the authors are addressing hallucination from the unanswerability perspective. \n2. **Effectiveness of proposed method:** Although prompt manipulation and beam relaxation methods are both simple, the performance improvement is drastic without harming the original performance.\n3. **Comprehensive experiments:** Experiments are conducted on three different LLMs, and on three different datasets. \n4. **Interesting insights:** This paper offers a detailed analysis of LLM from unanswerability that could be beneficial to the community.\n", "reasons_to_reject": "There are no major concerns I have regarding this paper. \nHere are some minor comments:\n\n1. I do not really get what we can learn from “erasing the answerability subspace”.\n2. Dividing Method section into subsections. Based on my understanding, prompt manipulation and beam relaxations go beyond just providing analysis of LLM behaviour towards unanswerability. They can be direct solutions to improve the problem of hallucination when presented with unanswerable questions. On the other hand, the other two subspace explorations are merely for understanding the LLM representation from unanswerability aspects. I believe separating them into two parts and emphasising the usefulness of prompt manipulation & beam relaxation method would strengthen this paper. \n", "questions_to_authors": "1. Why do we want to erase the answerability subspace..?\n2. In Figure 3: although the trend persists across all datasets, I can observe relatively poor performance in both NQ and MuSiQue in comparison to the Squad dataset. What could be the reason for this? \n3. In line 149, 150, you wrote: “three experimental approaches: i) prompt manipulation, 2) beam scrutiny, 3) probing”, but this does not seem to align with the following content (probing is missing). Is this a mistake? \n\n", "paper_topic_and_main_contributions": "This paper investigates LLMs' behaviours when presented with unanswerable questions, and explores different ways of improving LLMs in responding to these unanswerable questions. Authors explore four different approaches to either investigate LLM’s awareness toward “unanswerability” or to improve LLM’s ability to handle unanswerable questions: i) prompt manipulation by explicitly providing extra instruction “If it cannot be answered based on the passage, reply “unanswerable””. ii) “beam relaxation” which refers to looking for any cues of unanswerability in the beam candidates (i.e., keywords such as “unanswerable, IDK, I don’t know, N/A, none of the above, etc…). iii) training a linear classifier to see if answerability can be classified based on LLM representation. iv) erasing answerability subspace from LLM embedding space. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- There are grammar/typo issues in this paper. I recommend doing another pass through the paper to ensure these issues are resolved. For instance, “tried to use utilise beam search…” (line 131), “affective methods” (line 224) —> effective methods.\n\n"}}, {"review_id": "j8iTvuR2vr", "reviewer": "Reviewer_CavA", "summary": "", "questions": "In language generation, one problem is that the beam search based approach would make the top k generated text look identical. The reviewer wonder if authors also observe the same phenomena in beam relaxation?\n\nWhile, in this paper, it looks that the (un)answerability prediction is much affected by the beam size. From Line 191, authors mention that the final layer of the first generated token is used for prediction. Does that mean the beam size will much change the first generated token? Appreciate if authors can elaborate more on this, and also please correct reviewer if the understanding is wrong.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive feedback and appreciation of our work, as well as for spotting typos, which we will correct in the final version of the paper. \nHere are our responses to the reviewer’s questions and comments:\n\n***gray-box testing***\n\nWe agree with the reviewer that our analyses, especially those combining the prompt-engineering with the subspace exploration, can be seen as gray-box testing. We also hope that our work will lead to further such explorations in relation to unanswerability, to better understand and control its effects.\n\n***Identical responses in beam-relaxation***\n\nUpon examining the final k candidates across instances, we did observe similar answers. Specifically, when a candidate indicated unanswerability (refer to Appendix C), we observed variations of unanswerability-indicating responses, while remaining candidates were similar. Alternatively, in cases when none of the candidates signified unanswerability, all the candidates resembled other candidates.\n\n***Relation between the answerability prediction and beam-search***\n\nAs the reviewer noted, for (un)answerability prediction, we made use of the last hidden layer of the first generated token. Since beam search impacts only post-first-token generation (because until the first generation, there hasn't been yet a separation into beams) and our predictions utilize embeddings that precede the generation of the first token, beam search does not influence our subspace exploration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) study the answerability problem from different perspectives\n\n(2) hint prompt helps to predict the (un)answerability of the question\n\n(3) visualized embedding space gives readers better illustration\n\nNot necessarily negative points, more or less some discussion\n\nPrompt engineering is important and useful. If pure prompt engineering is somewhat like black box testing. The LLMs are too big, which makes the white box testing too difficult. With the hint prompting, and analysis on the first generated token, is it somewhat like grey-box testing? Maybe this is a question for the community to answer together. \n\nPlease correct the reviewer if anything is missing or incorrect.", "reasons_to_reject": "\n", "questions_to_authors": "In language generation, one problem is that the beam search based approach would make the top k generated text look identical. The reviewer wonder if authors also observe the same phenomena in beam relaxation?\n\nWhile, in this paper, it looks that the (un)answerability prediction is much affected by the beam size. From Line 191, authors mention that the final layer of the first generated token is used for prediction. Does that mean the beam size will much change the first generated token? Appreciate if authors can elaborate more on this, and also please correct reviewer if the understanding is wrong.", "paper_topic_and_main_contributions": "For question answering, this paper is to study the research problem of predicting question (un)answerability, given the pretrained language model, especially for LLMs. \n\nAuthors study this problem mainly from 3 perspectives, including (1) prompt manipulation, (2) beam relaxation and (3) embedding space probing, which would cover the major factors that influence the QA task in LLMs. Experiments are conducted properly with 3 LLMs (Flan-T5, Flan-UL2, and OPT-IML) on 3 benchmark datasets (Squad v2, NQ, and MuSiQue). \n\nFrom the experiments, authors present their findings in each of the direction. (1) hint in prompt would much help to predict the unanswerability of the question; (2) larger beam size would help to recognize the unanswerable questions; (3) authors mainly analyze and visualize the embedding space of the first token for answer generation. The visualized analysis shows that the embedding space would be easier to be separated given hint prompt compared with regular prompt. Meanwhile, concept erasing tech is adopted to verify the influence of the answerability subspace.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 224: affective -> effective?"}, "extra_text": {}}, {"review_id": "YbyEDaMK8K", "reviewer": "Reviewer_9CW2", "summary": "", "questions": "See reasons to reject (2 and 3).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive feedback and appreciation of our work. \nHere are our responses to the reviewer’s questions and comments:\n\n***Closed-domain setting***\n\nWe thank the reviewer for pointing out this issue, and would adapt the final version of the paper to clarify the fact that our work focuses solely on the closed-domain setting. We also agree that an open-domain setting could be an interesting follow up research, but it is outside the scope of our current work.\n\n***Comparison between methods***\n\nWe would like to note that our main goal was to demonstrate, via different approaches, LLM’s inherent understanding of unanswerability, even when they in fact hallucinate responses. However, following the reviewer’s comment, we would try to compare the approaches for the camera-ready.\n\nRegarding the reviewer’s concern about the few-shot setting performing worse than the zero-shot setting in many cases, we suspect that this is due to the fact that in-context-learning examples are selected at random and sometimes they are dissimilar from the target example enough to hinder performance. For the final version, we would try to address this phenomenon, by using several ICL examples and averaging over their results.\n\n***Beam relaxation and hints***\n\nIn our setting, the beam relaxation is done without the hint. We will clarify this in the final version of the paper.\n\n***Unanswerability keywords***\n\nIndeed, the identification of answers indicating unanswerability depends on the keywords listed in Appendix C. As we mentioned in the footnote at the bottom of page 5, as well as in Appendix C, these keywords were collected based on an extensive analysis of the models' outputs (~500 of the unanswerable instances for each model). However, we agree with the reviewer's observation regarding the limited discussion in the paper of this analysis and will address this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper studies an important problem and performs a thorough analysis with three approaches and on three datasets.\n\n2. The linear separation method and its visualization is very interesting.\n\n3. Their approaches could be used (probably with some improvements) to remove some hallucinations from the output.\n\n4. Their processed datasets (if made available), and their setup could be used by others to improve the task.", "reasons_to_reject": "1. While at the first glance, one might think the paper also considers open-domain settings, the setting is constrained to cases where the answer can only be found in the given context (but not in the already existing knowledge in the LLMs). I think this should be clarified.\n\n2. The different approaches are not compared with each other. Also, it's not clear why few-shot settings do worse than zero-shot in many cases (see bold numbers in table 2b vs 2a).\n\n3. I'm not sure if beam relaxation is done with the hint or without the hint?\n\n4. The results of the first two approaches depend highly on the used keywords (Appendix C), but not much is said about it in the paper. Maybe a simple classifier using these initial keywords could improve the reliability of the results.", "questions_to_authors": "See reasons to reject (2 and 3).", "paper_topic_and_main_contributions": "This paper studies how/whether LLMs understand if they don't know the answer to a question given an input context (passage). They test three different approaches:\n\n1. Prompt manipulation, where they provide a hint that the question could be unanswerable, and compare results with the case without the hint. They show that providing the hint significantly increases the results (up to 80%) in both zero-shot and few-shot settings.\n\n2. Beam Relaxation, where they look for cues in the full beam indicating that the question is unanswerable (based on the existence of certain words in the decoded outputs). They show that increasing the beam size improves the task, when they consider a match in the full beam. However, the performance of the highest-ranking output doesn't increase by increasing the beam size.\n\n3. Linear classification (using SVM) on the final hidden layer of the first generated token: Interestingly, they show that it is indeed possible to linearly separate the answerable and unanswerable questions. They do visualization using PCA which supports their claim. They also show that models learn this notation beyond a dataset (by training on one dataset and testing on others). They further show that if the linear separability is erased, the good results don't hold anymore.\n\nThese approaches perform very well on the three datasets they've used (Squad 2, Natural Questions and MuSiQue, where the 2nd and 3rd datasets are adapted for the task), where accuracy of unanswerable question detection is around around 75% to 92%, while the accuracy of the main task doesn't decrease because of applying these models. This indicates that LLMs do have the capability of detecting unanswerable questions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Strengths**:

1. Paper studies a critical problem of detecting hallucination in LLMs from an unanswerability perspective.

2. 3 methods are proposed: (i) prompt engineering to check for unanswerability (ii) beam relaxation: checks for unanswerability cues in the generated candidates using cue phrase list. (iii) linear SVM classifier on first generated token to check for answerability.

3. Experiments with three different LLMs, and three different datasets.

**Weaknesses**:

1. Proposed methods are very simple.

2. Not clear to me even from the rebuttal as to how erasing experiments can help in performing interventions in that subspace in order to modify the model’s behavior.

3. There is not enough analysis: The different approaches are not compared with each other. Also, it's not clear why few-shot settings do worse than zero-shot in many cases

**Suggestions**:

1. Please handle typos. There are too many.

2. Explain motivation for erasing experiments more clearly.

3. "relatively poor performance in both NQ and MuSiQue compared to Squad" -- please include your explanation in the main paper.

4. Limitations section should include: (a) expts on QA with context only. (b) lack of comparison across methods. (c) Small cue phrase list (d) 3 datasets and 3 models. Need more expts to generalize to other datasets/models.
Decision: Accept-Main