Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering
Authors: Zhihua Wen
Zhiliang Tian
Zexin Jian
Zhen Huang
Pei Ke
Yifu Gao
Minlie Huang
Dongsheng Li
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model
knowledge boundary
question answering
Primary area: natural_language_processing
TL;DR: We explore the knowledge boundary of LLMs by investigating their responses to semi-open-ended questions, using an auxiliary model to uncover ambiguous answers and highlighting LLMs’ challenges in recognizing their knowledge limits.

Abstract: Large Language Models (LLMs) are widely used for knowledge-seeking purposes yet suffer from hallucinations. The knowledge boundary of an LLM limits its factual understanding, beyond which it may begin to hallucinate. Investigating the perception of LLMs' knowledge boundary is crucial for detecting hallucinations and LLMs' reliable generation. Current studies perceive LLMs' knowledge boundary on questions with concrete answers (close-ended questions) while paying limited attention to semi-open-ended questions that correspond to many potential answers. Some researchers achieve it by judging whether the question is answerable or not. However, this paradigm is not so suitable for semi-open-ended questions, which are usually ``partially answerable questions'' containing both answerable answers and ambiguous (unanswerable) answers. Ambiguous answers are essential for knowledge-seeking, but it may go beyond the knowledge boundary of LLMs. In this paper, we perceive the LLMs' knowledge boundary with semi-open-ended questions by discovering more ambiguous answers. First, we apply an LLM-based approach to construct semi-open-ended questions and obtain answers from a target LLM. Unfortunately, the output probabilities of mainstream black-box LLMs are inaccessible to sample more low-probability ambiguous answers. Therefore, we apply an open-sourced auxiliary model to explore ambiguous answers for the target LLM. We calculate the nearest semantic representation for existing answers to estimate their probabilities, with which we reduce the generation probability of high-probability existing answers to achieve a more effective generation. Finally, we compare the results from the RAG-based evaluation and LLM self-evaluation to categorize four types of ambiguous answers that are beyond the knowledge boundary of the target LLM. Following our method, we construct a dataset to perceive the knowledge boundary for GPT-4. We find that GPT-4 performs poorly on semi-open-ended questions and is often unaware of its knowledge boundary. Besides, our auxiliary model, LLaMA-2-13B, is effective in discovering many ambiguous answers, including correct answers neglected by GPT-4 and delusive wrong answers GPT-4 struggles to identify.

Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering
Zhihua Wen Zhiliang Tian | Zexin Jian', Zhen Huang',
Pei Ke², Yifu Gao', Minlic Huang", Dongsheng Li
College of Computer, National University of Defense Technology. Hunan, China Tsinghua University, Beijing, China
(zhwen, tianzhiliang, jianzexin21, gaoyifu, huangzhen, dsli] edu. cn
kepei1106@outlook com, aihuang@tsinghua. edu.
Abstract
Large Language Models (LLMs) are widely used for knowledge-sceking purposes yet suffer from hallucinations. The knowledge boundary of an LLM limits its factual understanding, beyond which may begin to hallucinate. Investigating the perception of LLMs' knowledge boundary is crucial for detecting hallucinations and LLMs' reliable generation. Current studies perceive LLMs' knowledge boundary questions with concrete answers (close-ended questions) while paying limited attention to semi-open-ender questions that correspond many potential answers. Some researchers achieve it by judging whether the question is answerable not. However, this paradigm is not so suitable for semi-open-ended questions, which are usually "partially answerable questions" containing both answerable answers and ambiguous (unanswerable) answers. Ambiguous answers are essential for knowledge-secking, but they may go beyond the knowledge boundary of LLMs. this paper, we perceive the LLMs' knowledge boundary with semi-open- questions by discovering approach to construct semi-open-endee questions and obtain answers from a target LLM. Unfortunately, the output probabilities of mainstream black-box LLMs inaccessible to sample for low-probability ambiguous answers. Therefore, we apply an open-sourced auxiliary model to explore ambiguous answers for the target LLM. We calculate the nearest semantic representation for existing answers to estimate their probabilities. with which we reduce the generation probability of high-probability existing answers to achieve a more effective generation. Finally, we compare the results from the RAG-based evaluation and LLM self-evaluation to categorize four types of ambiguous answers that are beyond the knowledge boundary of the target LLM. Following our method, perceive the knowledge boundary for GPT-4. We find that GPT-4 performs poorly on remi-open-ended questions and is often unaware of its knowledge boundary. Besides, our auxiliary model, I.LaMA-21330 13BB, is effective in discovering many ambiguous answers, including correct answers neglected by GPT-4 and delusive wrong answers GPT-4 struggles to identify.
ambiguous answers. First, apply LLM-baser
construct dataset
Work done during internship at the CoAl Group. "Corresponding Authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Large language models (LLMM) have revolutionized interactions with AI, enabling users to acquire knowledge by posing questions in natural language [12] 6). However, LLMs are prone to hallucination and generate non-factual responses, hindering the development of trustworthy AI.
One main cause of LLM hallucination is its unfamiliarity with the long-tail knowledge that appears less frequently than common-senss, knowledge in the training data. To alleviate this issue, many researchers collect more nomain-sseciine training data [27] or incorporate external information [66] 29] via aatrrivvalauugmette the perception of knowledge boundaries for L.Mss, which indicates the extent of knowledge that the I.LM can grasp well, beyond which it may begin to hallucinate [18] Studying the perception of knowledge boundaries for LLMs helps alleviate hallucinations in many ways. For example, 1) It helps detect the hallucinations of target LLM and the extent and scope of its factual knowledge [6 45)
generation (RAG) during inference. Another line of work investigates
2) It helps align LLMs for more honest generation 43
Existing studies on the perception of knowledge boundaries primarily in the form of Question- Answering (QA). Their methods mainly aim to judge whether a question is answerable or unan- swerable and regard their border the knowledge boundary. An answerable question refers to when the LLM is capable of generating a response matching the ground truth, and conversely, an unanswerable question means unable to answer correctly. These studies can be categorized into two groups. Prompt-based perception employs prompt engineering 45 14] to assess whether the LLM can answer the question via LLM self-evaluation. They question whether the LLM knows the answer 33 8
46 or needs external knowledge to answer the question [36]. As LLMs tend to
be overconfident 22 32
17 49 more researchers explore representation-basee perception. These
studies optimize different representations for answers with different answerability 14 10) 36 49 or
extract representations from a fixed encoder to train classifier [23].
However, directly discriminating questions into answerable and unanswerable ones may not apply to some partially answerable questions. In many scenarios, the questions relatively open- open-ended (i.e. having a list of correct answers) that may include (1) a subset of easy answerable answers, and (2) a subset of hard and unpopular answers, which may be unanswerable. These questions (referred to as "semi-open-ended questions") are particularly challenging and knowledge-extensin Investigating the ambiguous answers these semi-open-endee questions in various fields benefits knowledge-seeking Ambiguous answers often go beyond the knowledge boundaries of LLMs and could lead to misinformation (see App. G Therefore, we argue that investigating these questions with their ambiguous answers can augment the perception of the knowledge boundaries for LLMs.
In this paper, we propose to perceive the knowledge boundary for target LLM with semi-open- ended questions by discovering pieces of unfamiliar knowledge where the LLM learns badly. Particularly, We first construct dataset with semi-open-ended questions on the open domain and query the target LLM for their corresponding answers. We define the low-prod wwwrrobability correct answers and delusive incorrect answers are the ambiguous answers corresponding to the LLM's unfamiliar knowledge.
A challenge is that obtaining LLMs' low-probability answers needs accessing LLMs' output proba- bilities (or violently sampling LLMs' outputs many times to approximate the probabilities), which is inaccessible (or expensive) for mainstream black-box LLMs, i.e. GPT-4 Therefore, we approximate the generation probabilities of the target LLM with open- speresourced auxiliary model. We use the Pseudo-inverse of model embedding to estimate the nearest semantic representation for the existing answers. Consequently, we obtain the probability distribution of existing answers and repetitively filter the existing answers (and their semantic-related counterparts) to obtain answers with low- probabilities. Finally, we recognize answers beyond the knowledge boundary of the target LLM by comparing its self-evaluation results against the ground truth answers obtained from RAG-based evaluation.
Empirically, we use our method to construct a dataset of approximately Ik samples and evaluate GPT-4's performance. We find that GPT-4 makes mistakes in 82.90% of questions and 40.14% 15% of its ambiguous answers generated are unqualified. Besides, GPT-4 also makes inaccurate self-evaluation 28.77% of the time, indicating that these are beyond the knowledge boundary of GPT-4. Moreover,
For example, when asked to "Tell me about some exercise habits that are easy to overlook but are good for your health." there are many correct answers, yet the question remains constrained by the context of "exercise habits", "easy overlook" and "good for your health".

---

we find nearly 50% of the candidate answers discovered by our auxiliary model, LLaMA-2-13B, are also beyond the knowledge boundary of GPT-4. including both factual answers that GPT-4 fails to produce and delusive wrong answers GPT-4 evaluates incorrectly.
Our contributions ended questions to the perception of knowledge boundaries for LLMs, (2) We propose an ambiguous answer discovery strategy that discovers many ambiguous answers with pieces of knowledge that are beyond the LLM's knowledge boundary. (3) Experimental results show the poor performance of an advanced LLM, GPT-4, on remi-open-ended questions and the effectiveness of our ambiguous answer discovery method in finding more pieces of knowledge which the LLMs are unfamiliar with
threefold: (1) We the first investigate the importance of semi-open-
Related Work
2.1 Perception of Knowledge Boundaries for LLMs
Existing studies on the perception of knowledge boundaries for LLMs can be categorized into prompt-based perception and pattern-based perception. Prompt-based perception perceives the knowledge boundary by querying the target LLM. Many researchers instruct the LLM before and
after response generation, asking whether it can correctly answer the questions [32] 36
g 46 and
if the generated answers accurate [32 43
In addition, Yin et al. (2024) seek the optimal
prompt for benchmarking LLM knowledge boundaries, [Amayuelas et al. (2023) study the LLMs'
ability to understand their knowledge and measure their uncertainty. Kadavath et al. (2022) also instruct LLMs to generate their confidence score for their responses. As studies find that LLMs tend to be overconfident [29 32 49), many researchers explore representation-basel perception. Researchers identify unknown questions [14] or evaluate correct and incorrect answers 110 by
implicitly learning their different representations, Chen al. 2023) train LLMs identify incorrect answers via parameter-efficient tuning. Besides, Wann et al. (2023) extract representations of answerable and unanswerable questions train classifier to predict whether a question answerable and assume questions with similar representations share the same answerability. [Si et al. (2023)
take token probability as the answer's confidence score during generation. Zhao al. (2023) detect unanswerable questions by paraphrasing questions and checking the divergence of their answer distribution. The above studies primarily perceive knowledge boundaries for LLMs by distinguishing between answerable and unanswerable questions. This type of binary division does not apply to
questions with both common easy answers and unpopular hard answers. Our study is the first to investigate the perception of knowledge boundaries on cami-open-ended questions.
2.2 Questions Answering for LLMs
Existing studies on Question Answering (QA) can be categorized into open-ended QA and close- ended QA based on the type of questions. Close-ended questions correspond to a limited number of correct answers, usually the form of yes or no, true or false, or multiple-choice options, constraining the answers to predetermined answer set [31 15 40 33 In addition, Researchers also study open- ended questions that allow the respondent to provide a more detailed and subjective response such as personal opinions and explanations [21] 40 4.
Researchers study the performance of LLMs on QA tasks mainly through various prompting strategies. Wei et al. (2023) explore "Chain-of- Thouuhhh prompting (CoT), a simple and broadly applicable method for enhancing question answering ability of LLMs. Yao et al. (202)) and [Besta et al.
(2024) introduce similar frameworks for more complex QA tasks, namely, "Tree of TTouuhh" and "Graph of Thoughts" prompting. As studies show that relying solely on an LLM's internal knowledge may lead to hallucinations (28), many researchers have also improved model performance in QA by incorporating external information (RAG systems [47 and knowledge graph [I] [)) More recently, researchers have studied adaptive retrieval to avoid misinformation in the retrieved documents [1 Ni et al. (2024) estimate the answerability of the given question and determines whether to retrieve 10 29 Xu et al. (2024) learn to identify the knowledge boundaries of LLMs
and refuse to answer certain questions to avoid risks [14 42].
Our code and data are available wttp:///gttb....mmmmmm//LL/aaoo
3

---

3 Perception of Knowledge Boundary for LLM via Semi-open-ended QA
3.1 Overview
Our framework consists of three parts (see Fig. D. We first exploit the instruction-folloving ability of strong L.M create dataset consisting of semi-open-ended questions various domains with LLM's answers. To discover more pieces of unfamiliar knowledge for the target LLM, we apply an open-sourced auxiliary model to incur more ambiguous answers by encouraging more distinctive generations. Finally, we evaluate whether the ambiguous answers to each question are beyond the knowledge boundary of the target LLM by comparing the self-evaluation results against RAG-based evaluation.
3.2
Semi-open-ended QA Dataset
3.2.1 Dataset Construction
To study the performance of semi-open-ended questions across various domains, employ an LLM-bassd 2-step approach to obtain ceni-open-endes questions and collect answers.
Domain selection. We first prompt the LLM to generate list of domains, encompassing world knowledge, which includes areas such as biology, geology, music, etc.
Question generation. We prompt the LLM multiple times under each domain to generate a set of semi-open-ended questions Q. To ensure the quality of the generation quality, we provide human-written sample questions as demonstrations and specify the following requirements for the generation of candidate questions: 1. The question should correspond multiple correct answers, making challenging to answer. The question should also be relatively easy for non-expert users to understand. 2. The judgment of question answers should be based on objective standards in the real world instead of the subjective standards of the evaluator. 3. The truthfulness of an answer to a question should not change constantly over time. 4. The questions share the same template: Tell me a list of ....... We use the same vanilla prompt to eliminate the influence of different question styles.
Answer collection. For each question q in Q. we query the LLM times and collect all responses A = {ap, 01 .... d1-1). ). In the i-th interaction, previously generated responses Al: i] and obtain a;-1 by querying the LLM with question q' which repeats the same criteria specified in q and highlights the need for more answers] Finally, we extract all answer entities in A and construct an answer list A.
inform the LLM of all
3.2.2 Dataset Descriptions
We create dataset to investigate the performance of mainstream LLM, GPT-4, on semi-open-ended questions. In dataset construction, we set / to 3 to exploit the knowledge of GPT-4 through multi- round conversations. Like humans, when faced with such questions (c.g.. What are the animals unique to Australia?), LLMS tend first to give answers which they hold high confidence (like the red angaroo). The latter answers are less certain and may have more mistakes (like echidna). We define the initial 75% of answer entities from GPT-4 generations as common-sense answers, while the remaining 25% as ambiguous answers. Our dataset comprises 953 questions covering 32 domains, with well-distributed data within each domain. On average, GPT-4 yields 52 answers for each question, including an average of 13 ambiguous answers. See the data samples in App. A See the full prompts and demonstrations in App. C
3.3 Ambiguous Knowledge Discovery
We apply open-sourced auxiliary LLM to effectively discover more ambiguous answers that may be beyond the knowledge boundaries for black-box LLMs. Our intuition is that low-probability ambiguous answers reflect LLM's unfamiliarity with certain pieces of knowledge. However, it is challenging to collect low probability ambiguous answers as 1) the generation probability of black- box LLMs (e.g. GPT-4) are inaccessible and their dyper-parameee
(e.g. temperature) cannot target
For example, if q is: Tell me list of animals unique to Australia, then g' is Tell me more animals unique to Australia.

---

Semi-open-ended QA Dataset construction
Ambiguous Knowledge Verifieation
Amhiguous Knowledge Categorization
Unnqualiited ①D
Domain generation
R WHOLE Evalaation
introduee diflcrett deraiss knowledge As many prssible.
Unvenfibble
Food and Diet in 2, Music in Biolugy 4. Religion.....
FRGA h-hae-dddd Evalustion search online
YLEM Self cvaluscion
Does lale belnng segeablo Jat an in grgudeiied
Unforttier Knowiolge of $S
Unespeoted ( Weong Evalistio
Corn
(q. Okra
Fidiens Correct 2 Ansoerr
Questiin generation
with Food Diet...
Auxiliary LLM
Ambiguous Knowledge Discovery
Intializedd Probabilitiss
Embhedding of 0
Fall fizt of vegetahias that are high in proselts....
Okra
Original Probabilities
Moore Penruse recudo-imenne of
Tell regetablls ihat high pronvine
Ansow er collection
pea lennl
V
Y1
E
Estimatedd Probabilites of High probability answers
Samantic Space of CLami Squac Seraane
OO
2 Soimat la Lentll
y/2 = V1 18y
by
E
Reproestaiin
166
E
4y
The overview of our framework. For the question in the constructed dataset, The open- sourced auxiliary model prevent the high-probabiility answers based on existing answer entities A from the black-box -bx LLM (i.e. GPT-4) and generate 4 categories of ambiguous knowledge that are unfamiliar knowledge for the target model.
Figure 1:
ambiguous answer-related tokens (see Sec. 4.2 2) violently prompting the LLM with question q many times to approximate the generation probability is inefficient as it prioritizes high-pme sogh-probabilitty answers that may already be present in existing answers. Answers that semantically similar to high-probability answers also tend to have a high probability during generation. Hence, we propose to prevent the generation of high-probabilite open-sourced auxiliary LLM to incur more low-probability ambiguous answers for the perception of the knowledge boundary for the black-box L.M.
answers with their semantic-related counterparts on an
We denote that the open-sourced auxiliary model's final layer is E R Vixd where d and V is the dimension size and V the output vocabulary. In each generation step, LLM encodes the semantic representation of the input context as x E Rdx1 and uses it to calculate the next token generation probability as y/1 = Ex. Specifically, we take the following steps to estimate and decrease the probability of high-probability answers.
1. Probability initialization for existing answers. We choose the first token in an answer entity a E A "anchor token", which is suppose to represent the primary information about since the first token is indispensable to its generation[9] [9] For a given question g, we define a vector Ay E V indicating the existence of unique anchor tokens in all answer entities to the question. In Ag, we assign the value 1 to the anchor tokens' position and assign 0 to other positions, where n is the number of all anchor tokens. As ΣS Ay; 1, we deem Ay as the initialized probability distribution of all existing answers for the given question.
2. Semantic estimation of high-probability answers. We estimate the semantic representation 5x for high- probbbbbbay answer entities from initialized existing answer probability Ay. We calculate óx E Ay, where E E = I. Here, E+ is the left Moore- Penoose pseudo- inverse, and I is the identity matrix. Because the pseudo-inverse of a non-square matrix is often used to find the least squares solution of equations, In this way, we obtain da as the least squares semantic representation that is the nearest to approximate the real semantic representation (which is unknown) of anchor tokens.
3. Probability Reduction. We calculate the probability of high-probabiln answers as gy- Edz and ultimately obtain an adjusted generation probability y2:
1/ 31
dōy y1
XEōr 91 XEET Ay,
5

---

where À is scaler that controls the extension of the reduction. Since anchor tokens and their semantic tokens almost share the same semantics] interpreting the estimated semantic representation of anchor tokens propagates the semantic information to their semantic- related ones. Thereby, we obtaín an estimated probability distribution of high-probability tokens dy and use it to prevent the generation of high-probability answers.
During generation, the auxiliary LLM samples words the adjusted probability distribution y2 instead of the original distribution y1. In this way, we only reduce the probability of answers that are semantically related to existing high-probability answers. while preserving the remaining probability distribution almost intact. The black-box model has already exhausted almost all high-probability common answers in A, forcing the auxiliary model to generate new ambiguous answers with low probabilities. These ambiguous answers are either low-probabilityy correct answers neglected by the black-box model or delusive incorrect answers.
3.4 Ambiguous Knowledge Verification
We compare the results of LLM self-evaluation against the ground truth from RAG-based evaluation to verify the truthfulness of ambiguous answers, thereby identifying answers beyond the knowledge boundary for the target L.M.M We conduct self-evaluation on the target LLM with well-designed instructions. We craft a prompt template including 1) an incentive statement that encourages better performance: I'll pay you $100 for a factually correct answer 77: 2) an instruction think step by step which prompts the LLM to analyze before reaching to conclusion [37 24]. 3) multiple human-crafted examples as in-context learning demonstrations.
We believe answer in each test case Aq to be factually correct to if verified on public, trustworthy sources. As questions in our dataset correspond numerous low- wwwbpabbbbiity hard answers, it is cost-prohibitive to annotate the truthfulness of each answer with expert knowledge. Inspired by Web-GPT (28] we adopt cost-efficient approach to mimic human behavior when faced with unfamiliar knowledge. We instruct an Internet-connected LLM with the same prompt for self-evaluation and require search online for related information before making judgments.
For both evaluations, we evaluate each answer a to its corresponding question as 1) incorrect, which contradicts reliable sources; 2) correct, which is supported by reliable sources, and 3) unverifiable, which is for cases that cannot be verified based on available information. Finally, for each answer a, we compare the differences between LLM self-evaluation and RAG-based evaluation to categorize different types of ambiguous knowledge.
3.5 Ambiguous Knowledge Categorization
We classify the ambiguous answers into four categories based the above two evaluation results and posit that they are beyond the knowledge boundary of the target LLM. For a question, we categorize the following types of answers to be the LLM's ambiguous answers:
Unqualified answers: answers from the target LLM's response that identified as incorrect or unverifiable according to the ground truth.
Inaccurate evaluations: answers from the target LLM's response whose self-evaluation results contradict their ground truth.
Hidden correct answers: answers that are neglected by the target LLM, yet supplemented by the auxiliary model, which are correct according to ground truth.
Unexpected wrong evaluations: answers that are neglected by the target LLM but generated by the auxiliary model whose sclf-evaluation results misalign with their ground truth.
The above categorization helps us understand different types of misunderstanding of the target LLM regarding specific pieces of unfamiliar knowledge.
Near- -upricup tokens roughly share the same semantic meaning but are different under tokenization due to typos, capitalization, whitespace marking. For example, Pea and peas near-duplicate answers.

---

Experiments
4.1 Settings
In our experiment, we investigate the knowledge boundary of GPT-4 on our constructed dataset.
We use two LLaMA-2-13b 13b [35] models, as our auxiliary models Sec. 3.3 Our method sets À in Sec. 3.3 to 80. See more implementation details in App.
We compare our method with several baselines. Following prompt-based approaches [36], ..r.pp instructs the auxiliary model to generate more answers via prompt-engineering. Inspired by Zhang al. (202)), MASK belongs the representation-basee perception that uses an average initialization for the probability of tokens from existing answers to represent the likelihood of high nigh-probability answers and reduce their generation probabilities in the auxiliary model.
For the evaluation metrics, we use widely adopted Exact Match [2] (EM) and F1 scores 32 36 to measure the performance of in discovering ambiguous answers. Different from previous research, our semi-open-ended questions correspond to a large number of correct answers, making it hard to build a comprehensive answer set for evaluatiors Instead, we select ambiguous answers identified by the RAG-evaluation and confirmed by our human annotators as their ground truth and compare them with the full response to calculate the EM and FI. In this way, EM is entity-level metric that measures the percentage of ambiguous answers within the response. Fl is a word-level metric that quantifies the word overlap between the ambiguous answers and the ground truth. We adopt Bleu [30] to measure word-level overlap between responses from the GPT-4 response and the auxiliary model. We also use answer overlap rate (AOR) to evaluate the efficiency of generating distinctive ambiguous answers. AOR is an entity-level metric that calculates the proportion of words in a list of answer entities that duplicate the reference response. See more evaluation details in App. B
4.2 Overall Performance
Table 1: The performance of different auxiliary models with various strategies in discovering more ambiguous answers.
Method
Auxiliary Model
EM f
F1
AOR
Bleul
Bleu2
Bleu3
Bleu4
Prompt
LLaMA-2-13B
0.300
0.461
0.490
0,252
0.118
0.052
0.023
LLaMA-2-7B
0.458
0.570
0.342
0.185
0,075
0.021
0.004
MASK
L.LMMA2-133B
0.470
0.587
0.344
0.189
0.075
0.021
0.004
Ours
LLaMA-2-13B
0.481
0.587
0.326
0.181
0.071
0.018
0.004
We analyze the effectiveness our auxiliary model in exploring the knowledge boundary of GPT-4 by comparing it with multiple baselines. We randomly sample 200 questions from our dataset and discover ambiguous answers with an auxiliary model. Then, we verify the truthfulness of these answers using RAG-based evaluation with human annotation following Sec 3.4 and measure their performance with our evaluation metrics. Tab shows the results of ambiguous answers on different evaluation metries when using different auxiliary models and strategies. Prompt directly prompts the auxiliary model to generate answers, achieving the worst performance on all metrics. This suggests that directly prompting the auxiliary model may generate many repetitive answers (results in high AOR and Bleu scores) and, therefore inefficient in discovering new ambiguous answers (results in low EM and F1). MASK reduces the generation probabilities of anchor tokens during generation. When employing MASK on the same auxiliary model, its EM and F1 increase to 0.47 and 0.587 respectively while achieving lower AOR (0.344). It indicates that reducing the probability of the generation of anchor words effectively achieves a more diverse generation. Replacing the auxiliary model with LLaMA-7B results in a slightly lower EM of 0.458. This marginal decrease implies that while a larger model can offer a broader knowledge base, reducing anchor word probabilities is more influential in generating distinctive answers. Our strategy estimates and reduces the generation probability of near- -upricup answers. This approach achieves the best performance on all metrics. It
There are many low- frequency answers for these questions. For example, there are maybe hundreds of accurate answers to the question: What animals native to Australia?
7

---

underscores the effectiveness of our strategy in generating diverse set of ambiguous answers that less likely to duplicate existing ones, thus exploring the knowledge boundary for the target LLM. See our case study in App
4.3 Ablation Study
Table 2:
Ablation study on the key components in our method. We use the same metrics as in Sec. 4.2 apart from those that require manual annotation. AOR and Bleu measure the entity- and word-level overlap respectively between answers from GPT-4 and different model variants. Auxiliary Model prompt GPT-4 with existing answers as examples for more ambiguous answers. Inverse Matrix keeps the near-duplicate tokens during generation. Besides, adjust the probability influence scaler (A) to verify its impact on the generation results.
Variants
AOR,
Bleu
Bleu2,
Bleu3.
Bleu4
Auxiliary Model
0.535
0.267
0.106
0.037
0.010
Inverse Matrix
0.344
0.189
0.075
0.021
0.004
Ours(X=60)
0,419
0.224
0.098
0.038
0,013
Ours(X=70)
0.352
0.192
0.076
0.021
0,005
Ours
0.326
0.181
0.071
0.018
0.004
We conduct an ablation study on our proposed method to verify the importance of each component in eliciting more distinetive ambiguous answers (as shown in Tab Auxiliary Model abandons the auxiliary model and use existing answers as in-context learning examples to prompt GPT-4 for more answers. It achieves the highest on all metrics, indicating that prompting the black-box model violently for more answers is inefficient as it results in many repetitive answers. We also try to encourage more diverse generation by increasing the generation temperature of GPT-4. However, we find that GPT-4 starts generating scrambled texts after just few words and the perplexity of these texts exceeds 1000. while normally GPT-4 generates low perplexity of around 10. This indicates that increasing the sampling temperature results in the generation of scrambled texts. Although adjusting the generation temperature can change the generation probabilities, it does not alter the original probability relationships, nor can it specifically target tokens related to ambiguous answers. Inverse Matrix only reduces the probability of existing answers without considering their near-duplicate answers. It performs better than Auxiliary Model while underperforms Ours on all metrics. It shows that estimating and reducing the probability of near-duplicate tokens augment the auxiliary model for higher generation diversity. We increase the intervention on the generation probability by lowering X, the probability influence scaler. From row 3 to row Tab. 2 À increases from to 80. resulting in decrease all metries. This suggests that the extent to which intervene in the generation probability is positively correlated with the diversity of ambiguous answers produced by the auxiliary model.
4.4 Results of Perceiving the Knowledge Boundary for GPT-4
Table 3: Percentages of different categories of answers comparing the LLM self-evaluation and ground truth labels. Following the categorization in Sec. 3.5 we calculate that the percentage of unqualified answers is 40.1%% by adding up the underlined results that incorrect or unverifiable according to the ground truth. We also obtain the percentage of inaccurate evaluations as 28.47%, by adding up the results highlighted in red where self-evaluation is inconsistent with the ground truth.
Ground Truth)
Incorrect
Correct
Unverifiable
Self-evaluation
Incorrect
20.98
8.37
2.25
Correct
9.30
47.77
2.78
Unverifiable 3.30
3.73
1.52
By analyzing the ambiguous answers in our dataset, we arrive the following findings: (1) GPT-4 performs poorly the semi-open-ended questions and generates many unqualified answers.
8

---

We calculate the percentage of questions where at least one of the GPT-4's answers is unverifiable or incorrect according to the ground truth. We find that GPT-4 generates incorrect or unverifiable answers in 82.90% of questions. By adding up the proportion of incorrect answers (row in Tab33) and unverifiable answers (row in TabB3), we identify that 40.15% of ambiguous answers belong to unqualified answers. (2) GPT-4 makes many inaccurate evaluations regarding the truthfulness of ambiguous answers, indicating that the LLM lacks understanding of the relevant knowledge. We identify answers whose ground truth misaligns with self-evaluations in Tab 3 and find that 28.47% ( by adding up the results in red color from 3 of ambiguous answers belong to inaccurate evaluations for GPT-4. It indicates that GPT is unfamiliar with these pieces of knowledge, and (3) GPT-4 has limited ability to recognize its knowledge boundary, while in most cases it continues to produce unqualified answers. We search for keywords in all responses that reflect GPT-4's admission of its knowledge boundary (e.g. apologize and I' m afraid) and calculate the proportion of corresponding questions in the dataset. We find that in about 7% of questions, GPT-4 admits that it has listed all the answers and refuses to provide more answers (it generates response like apologize for any confusion, but to the best of my knowledge, the list I provided includes all the correct answers.). However, it fails to recognize its knowledge boundary in the rest questions and continues generate unqualified answers. Our findings indicate that advanced LLM (i.e. GPT-4) is easy to hallucinate on semi-open-ended questions, indicating the importance of detecting the LLM knowledge boundary via these questions.
retrieval is helpful for LLM to draw the correct conclusion.
4.5 Results of the Auxiliary Model on Perceiving the Knowledge Boundary for GPT4
Table 4: Percentages of different categories of ambiguous answers comparing the GPT-4 self- evaluation results and their ground truth. Following the categorization in S.5 we calculate that the percentage of hidden correct answers is 75.12% by adding up the starred (*) results that are correct according to the ground truth. We also obtain the percentage of unexpected wrong evaluations as 62.43% by adding up the results highlighted in orange where GPT---evaluation is inconsistent with the ground truth.
Ground Truth)
Incorrect
Correct
Unverifiable
GPT---evaluntion
Incorrect
0.04
9.53
11.31
Correct
23.97*
37.54*
13.61*
Unverifiable
3.18
0.83
0.00
shows the fine-grained results of GPT-4 self-evaluation and the ground truth of ambiguous answers discovered by auxiliary model. By analyzing the results, we conclude some interesting findings: (1) LLaMA-2-133 effectively supplements GPT-4 by identifying hidden correct an- swers. We add up the starred results in Tab obtain the proportion of L.M neglected hidden correct answers (75.12%) Notably, 23.97% and 13.61 % of correct ambiguous answers are both neglected by GPT-4 and deemed to be incorrect or unverifiable under GPT-4 self-evaluation, show- casing the GPT-4's unfamiliarity with the corresponding knowledge. (2) LLaMA-2-13B is easy to incur unexpected wrong evaluations during GPT-4 self-evaluation. We add up the results highlighted in orange from Tan find that 62.43% of the GPT-4 self-evaluations are inconsistent with the ground truth. It implies that GPT-4's self-evaluation mechanism may not be fully aligned with the actual correctness of the ambiguous answers, especially when they are supplemented by the auxiliary model. (3) LLaMA-2-13B also able to discover situations where GPT-4 admits for its knowledge boundary. We add up the results in column 3 where GPT-4 admits that it cannot make judgment (unanswerable) during self-evaluation and obtain 24.92% of aligned ambiguous answers. It means that GPT-4 aligns well with these ambiguous answers because it neglected these answers during generation and deems them as unknown knowledge during self-evaluation.
4.6 Practical Implications
Perceiving LLMss knowledge boundaries is important to understand and alleviate hallucination 18 49]. Ambiguous answers for semi-open-ended questions are highly likely beyond the knowledge boundaries of LLMs (see Sec 4.4). Discovering ambiguous answers benefits many applications,

---

including: 1) It helps detect the knowledge scope of LLMs more faithfully. While many close-ended
hallucination evaluation benchmarks face the danger of data contamination 25 13 semi-open-endee
questions are easy to design and correspond to large number of undocumented answers; 2) Flagging
ambiguous answers with higher uncertainty enhances the L.M outputs 22 19]: 3) Identifying
ambiguous answers helps achieve selective retrieval that augments LLM with indispensable external
information while reducing the distraction of irrelevant data [36 26 39]; 4) helps align LLMs for
more honest generation by teaching the LLM to admit its knowledge limit on the knowledge it is
unfamiliar with (ambiguous answers)
5
Conclusion
We investigate the perception of knowledge boundary for LLMs with remi-open-endedd questions, an important yet underexplored type of question corresponding to a large number of accurate answers. We introduce answers from the target LLM. Then, we discover more pieces of unfamiliar knowledge for the target LLM by eliciting ambiguous answers from auxiliary model that the LLM neglects. To achieve a more effective generation, we estimate and reduce the generation probability of existing answers with their near-duplicate counterparts. With our methods, construct dataset evaluate the performance of GPT-4 and discover many ambiguous answers with our auxiliary model, LLaMA-2- 13B. Our findings reveal that GPT-4 produces many unqualified answers and suffers from inaccurate evaluations. Besides, we verify that LLaMA-2- 1AMMMA-2-13- 13B is effective in discovering more unpopular correct answers and delusive wrong answers neglected by GPT-4. Our findings underscore the importance of semi-ended questions and the effectiveness of our method in assisting in perceiving knowledge boundaries for L.LSs.
-LMebased approach to construct semi-open-ende ceni-open-ended questions and collect LLM
6 Acknowledgement
This work is supported by the following findings: Young Elite Scientist Sponsorship Program by CAST (2023QNRC001) under Grant No. YESS20230367, the National Natural Science Foundation of China under Grant No. 62306330, No. 62106275, No. 62025208, No. 62421002, and the Grant of No. WDDZC2023520103..
References
[1] Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi, and Huan Liu. Can knowledge graphs reduce hallucinations in llms? A survey, 2024.
[2] Alfonso Amayuelas, Liangming Pan, Wenhu Chen, and William Wang, Knowledge of knowl- edge: Exploring known-unknowns uncertainty with large language models. arXiv preprint arXiv:2305. 13711, 2023.
[3] Yejin Bang, Nayeon Lee, Tiezheng Yu, Leila Khalatbari, Yan Xu, Samuel Cahyawijaya, Dan Su, Bryan Wilie, Romain Barraud, Elham J. Barezi, Andrea Madotto, Hayden Kee, and Pascale Fung. Towards answering open-ended ethical quandary questions, 2023.
[4] Man Luo;Shailaja Keyur Sampat,Riley Aha:Akarshan Sajja;Chitta Baral. 'just because you are right, doesn't mean am wrong": Overcoming bottleneck in the development and evaluation of open-ended visual question answering (vqa) tasks. In Conference of the European Chapter of the Association for Computational Linguistics, 2021.
Hallman;Yankal Zeng:Manuha Van-
[5] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski. Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph Thoughts: Solving Elaborate Problems with Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 2024. doi: 609//aaii..... 138aaaai..88118 v38i16.297999
Mar URL httpt:://///tt aaai org/ index, www.:/hpppppa AAAT/
10

---

[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, lya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https: /ttpps::/ppooceednggggssss neurips.. cttpcccpppp.. cc/paper_aa. f files/ httpppppaaeeeeppppr///////////// wwwpp/ppppp///////p 1457c0d6bf htw.aabbbbbbbb771466666666611260
Paper. pdf
[7] Sondos Mahmoud Bsharat, Aidar Myrzakhan, and Zhiqiang Shen. Principled instructions are all you need for questioning llama-1/2, gpt-3.5/4, 2024.
[8] Lang Cao. Learn refuse: Making large language models more controllable and reliable through knowledge scope limitation and refusal mechanism. arXiv preprint arXiv2211..1041, 2023.
[9] Haw-Shiuan Chang. Ruei- Yao Sun, Kathryn Ricci, and Andrew McCallum. Multi-CLS BERT: An efficient alternative to traditional ensembling. In Anna Rogers, Jordan Goyd-Graber and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 821-854 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v 1215533/1/222.... wat.///2........ng....a.. acI- URL https://aclarthollooe org/ 2023 acl- acl-allg.......... .48
[10] Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Arik, Tomas Pfister, and Somesh Jha. Adaptation with self-evaluation to improve selective prediction in LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguisties: EMNLP 2023, pages 5190-520 5203,,,,,555 Singapore, December 2023, Association for Computational org/
Linguisties. doi: 10. 10611833//112023 http://///222...nnnnneemmemmmm/ findings-emnlp.345.. URL/ittps::/cccaatttoooo 2023. findffiiiinnggmmmm. 345
[1]] Zhuo ChenaJiaoyan Chen; Yuxia Geng:Jeff Z. Pan:Zonggang Yuan Huajun Chen. Zero-shot visual question answering using knowledge graph. In The Semantic ISWC 2021, 2021.
[12] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barbam, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem,, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
(13) Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Inves- tigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, ACI. Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1/2024.aacllll1oggg org/2024 naacl-long, 118.--1118..222
URL
[14] Yang Deng, Yong Zhao, Moxin Li, See- Kee-Kiong Ng, and Tat-Seng Chua. Gotcha! don't trick me with unanswerable questions! self-aligning large language models for responding unknown questions. arXiv preprint A40i 15062, 2024.
[15] Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the task or the annotator? an investigation of annotator bias natural language understanding datasets. Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference
11

---

Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing TEMNLL-IJCNLLL pages 1161-1166, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10118633/1/D11-1100. URL
[16] Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo. Lijie Wen, Philip S. Yu, and Zhijiang Guo. Towards understanding factual knowledge of large language models. In The Twelfth International Conference on Learning Representutions, 2024. URL (tLLhhtts::/// htttps://openreiier
net/forumiid- 2ddooruund?d--00ewuddddd
[17] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qian- glong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallu- cination in large language models: Principles, taxonomy, challenges, and open questions. ArXiv, 0as/2311.05222 2023. URL https://api semanti wwwp:orpos//Coolccco 265067168
[18] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong. Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023.
[19] Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. TACL, 9, 2021. doi: 10.1 162/tacc___sa a a_00077 URL httpp:://acaaaaa///
202g/2021112 tac1-1.57
[20] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Jran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan, Language models (mostly) know what they know, 2022.
[21] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769- 6781, Online, November 2020. Association for Computational Linguistics. doi: 18.18653/v1/ 2020.emnlp-main.551 URL httph:tt:///eet http://aaaaantooogyooo org/ 2220 emnlp-main 550
[22] Kimiya Keyvan and Jimmy Xiangji Huang. How to approach ambiguous queries in con- versational search: A survey of techniques, approaches, tools, and challenges. ACM Comput. Surv, 55(6), December 2022. ISSN 0360-0300. doi: 10.1145/3534965. URL
https://doi. org 100 10..1144/333498
[23] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generaliza- tion through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL httpss://openreviiee wwwpeerreveeeeeee/oooreeeeeeeeee
wwt. wwt.:/ettnnnne...o BatrrummmidHiiBBCCKKCCC
[24] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 22199-22213 Curran Associates, Inc., 2022. URL
nwwrgps..ee.....ee.. cttpeecccpppae wwtppppppppppppeeetppe/eeeee. faaper_filess pttle///ppprr/22///1eeee abbpessaccccc00011100006663633-3 Compper-Coofeeence pdf
[25] Yucheng Li, Frank Guerin, and Chenghua Lin. An open source data contamination report for large language models, 2024. URL htt.httts:://rxivv org/ abs/ 2310. 17589
[26] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of paramet- ric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguisties (Volume 1: Long Papers), pages 9802-9822, Toronto, Canada, July
12

---

2023. Association for Computational Linguistics. doi: 10. 10.:8653//1/023..clloogggg URL https://aclanttollogy org 2ttg//223..a0 acl long..45
[27] Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, Tolga Aktas, and Todd Hendry. Injecting new knowledge into large language models via supervised fine-tuning. 2024.
[28] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browscr-assisted question-answering with human feedback, 2022.
[29] Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng. When do Ilms need retrieval augmentation? mitigating lims' overconfidence helps retrieval augmentation. arXiv preprint arXiv:2402. 1 2024.
[30] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.31 15111///0033300733553
URL /ttptttt////aaaatolollo halpa/aaaantooooooyoooooo
org/PO2-1040 00/0:///aattollooorrrr////000000
[31] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In Anna Korhonen, David Traum, and Lluís Märquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages Computational Linguisties. doi: 10.18663/v1/11--4453
Florence, Italy, July 2019, Association for /tLLtttss///caaaate
org/
P19- 1487
[32] Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. Investigating the factual knowledge boundary of large language models with retrieval augmentation. arXiv preprint arXiv:2307.1 1019, 2023.
[33] Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering, 2023.
[34] Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd- Graber, and Lijuan Wang. Prompting GPT-3 to be reliable. In The Eleventh International Conference on Learning Representations, 2023. URL hnttps://openrevier wwwprrvvvieeeee/ffruue net/forum? id 98p5x5115af
[35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marein Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[36] Yile Wang, Peng Li, Maosong Sun, and Yang Liu. Self-knowledge guided retrieval augmen- tation for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10303-10315, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1/253//3/1// findings-emmlp.69) URL
wwwploggy.rrg///20/ findingsiiniiiiiniie eerediinnnneenlllppmmm 691
13

---

[37] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le. and Denny Zhou. Chain-of- of-thought prompting elicits reasoning large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems, volume 35, pages 24824-24837, Curran Associates, Inc., 2022. URL itt/ neur seeurips... cc/ wc/paper_fftteeeeeccee ffl.ffflllpppppleee Paa.aaaaaeepppper--oeeeeeeeeee Conference. pdf
/aper//
[38] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thoughe prompting elicits reasoning in large language models, 2023.
[39] Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. Repo- former:
Selective retrieval for repository-level code completion. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver. Jonathan Scarlett, and Felix Berkenkamp. editors. ICML, volume 235 of Proceedings of Machine Learning Research, pages 53270-53290. PMLR, 21-27 Jul 2024. URL http:::/ mml. www.eppesssseee vDdps//2223wwuu24e html
[40] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777-9786, June 2021.
[41] Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models knowledge conflicts, 2024.
[42] Hongshen Xu, Zichen Zhu, Situo Zhang, Da Ma, Shuai Fan, Lu Chen, and Kai Yu. Rejection improves reliability: Training Ilms refuse unknown questions using rl from knowledge feedback, 2024.
[43] Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. Alignment for honesty, arXiv preprint arXiv:2312.07000 2023.
[44] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I.. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models, 2023.
[45] Xunjian Yin, Xu Zhang, Jie Ruan, and Xiaojun Wan. Benchmarking knowledge bound- ary for large language model: different perspective on model evaluation. arXiv preprint a4Xiv:2402.1 1493, 2024.
[46] Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguisties: ACL 2023, pages 8653-8665, Toronto, Canada, July 2023. Association for Computational org/
Linguistics. doi: 00.18653/v1/2022 windiigns-ncl..... ac1.551. URL 2023 www.fiinnnnsaaa aalaanns-all.550 551
[47] Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4615-4635, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10111866511222233 htt.:aaawwwooloooooo/2//2../3/.
findings-emmlp.3077 URL stttpe://aclanntollog. www.oog///223/ findingiindnnggg emnlp ammnp.307. 307
[48] Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:23/2..57110 2023.
[49] Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. Knowing what 1lms do know: simple yet effective self-detection method. arXiv preprint arXiv:2310.179.1 2023.
14

---

A Dataset Information
Table 5: Example of data samples in our dataset. Each question corresponds to many ambiguous answers. Different colors of the answers represent different ground truth truthfulness labels: yel- lowgreen represents that the answer is verified as being factually correct, red is verified as being incorrect, and salmon is for unverifiable answers.
Domain
Percentage
Question
Ambiguous Answer
Malabar Chestaut, Miraclo Fruit Tree,
Guapinol Tree, Peanut Tree,
Sapodilla Tree, Desort Date.
Biology
4.93%
Tell me list of trees that produce fruit with hard shells are native to tropical regions.
Castor Oil Plant, Chiclé Tree,
Oriental Persimmon, Dika Tree, Langsal Tree,
Pequi Tree, Korite Tree,
Imbu Tree, Caraipa Tree
The Hurt Locker, Zalla Letters Trom Iwo Jama,
me list of films whose plot centered on
Doctor Zhivago, Black Hawk Down, Miscrables,
Music
4.20%
historically significant event, but the narrative from the perspective of an imagined protagonist.
Lawrence of Arabia, The Wind that Shakes the Barley,
The Longest Day, Anthony Adverse,
The Navarone, Lion
Quidam Busin, Dishi-c Kavir, Dustite Ltl.
Geology
4.09%
Tell me a list of current deserts across the world that proviously covered by an ancient
The Registan Desert, The Franklin Basin,
Makran
The Soneran Desert,
The Jornada Del Muerto, Great Karoo, Tanami Desert
Sacha Inchi Nuts, Hemp Seeds, Chokeherries,
Tell me list of fruits that are sources of healthy fats, not including avocados and cocomuts.
Food and Dict
5.67%
Eiderberries, Nuts,
Fruit Seeds,
Tibetan Goji Berries
Helen Taylor, Catherine Helen Spence, Octavia Hill,
Literature
1.6%%
list of female writers from the Victorian era whose work focuses on social reform.
Emmeline Pankhurst, Alice Meynell, Elizabeth Robins, Mary Augusta Ward, Constance Gamett
Rhoda Broughton, Elizabeth Missing Sewell,
Our dataset covers the following topics: Environment and Climate, Technology and Industry, Politi- cal Science, History and Archaeology. Sociology, Economy and Finance, Philosophy, Languages, Art, Architecture, Music, Physics, Astronomy, Chemistry, Biology, Geology, Computer Science, Anthropology and Cultures, Education, Psychology and Mental Health, Fitness and Physical Health, Literature, Religion, Law and Criminology, Military and War Agriculture, Tourism, Film and Tele- vision, Sports and Athletics, Food and Diet, Energy and renewable resources, Mathematics and Statistics, Medicine and Health, Games, Clothing and Fashion.
B Implimentation Details
In our experiments, we query GPT-4-Turbo through API calls. For our RAG-based evaluation in Sec. 3.4 we first employ Microsoft Copilot to search the Internet to find evidence and draw conclusion. For our evaluation to calculate EM and Fl, we hire human annotators with Master's degrees to manually review the responses of Copilot, ensure the credibility of the references, and determine the truthfulness of each answer.
During the generation of our auxiliary model, We apply nucleus sampling (with p=0.9) during generation, setting the generation temperature 0.7, and the repetition penalty to 1.15.
For our metrics, EM and F1 are common in QA tasks and knowledge boundary detection tasks. The Bleu metric is often used measure the n-gram overlap between the model response and the ground truth. A lower Bleu score means a more dissimilar response generated by the auxiliary model. Before calculating the above metrics, we normalize answer entities with the NLTK library by lowercasing the answers and turning them into the singular form.
C Important Instructions
Here provide important instructions in building our dataset, guiding LLM for self-evaluation. and show guidelines for human annotators in the supplementary materials.
Domain Generation. I hope to test my students' knowledge different domains. Which domains can I use create questions?
15

---

Question Generation. I am professor of [CATEGORY] and need to test students' under- standing of [CATEGORY] by asking a series of challenging questions. These questions require respondents to list entities that they know meet a series of certain conditions. You need to create more different and diverse challenging questions according to the require- ments. Read the following requirements carefully. I'm going to tip $100 for a perfect list of questions! In The questions should meet the following criteria: \n 1. Each question should start with "Tell me list of"; In 2. To make the question challenging enough, each question should contain multiple limiting conditions. in 3. The requirement of the question should not involve specific numbers (which makes the question too hard to answer) or vague descriptions (which makes it hard to evaluate the truthfulness of the answer). like "long lifespan", "quick speed", "popular", and "important", In 4. The boundaries of the question should be very clear, making it easy to evaluate its truthfulness; In 5. The answers to the questions should be consistent through a relatively long time and not change frequently, for example, yearly. In Refer to the style in the following two examples from an exemplary subject, biology. In Question 1: Tell me list of land animals unique to Australia. In Question 2: Tell me a list of fruits that grow on trees in tropical regions.
Self-evaluation. Does [AMBIGUOUS ANSWER] belong to [QUESTION REQUIREMT: NTSJ? I'll tip $100 for the factually correct answer. Think step by step and then give your answer.
RAG- evaluation. Search online for highly credible information related to the following question, and answer the question based on the search results. In Does [AMBIGUOUS ANSWERJ belong to [QUESTION REQUREEMTNTSP I'll tip $100 for the factually correct answer. Think step by step and then give your answer.
D Future Work
Our approach of modifying the LLM representations to guide answer generation may provide insight for different kinds of normal QA tasks: It may help alleviate the hallucinations in knowledge-extensive QA tasks via representation engineering. Editing LLM representations considering existing answers can reduce the probability of semantically related words, helping generate more diverse answers for open-ended QA tasks. In our future work, we plan to investigate representation engineering for more diverse and honest responses for different QA tasks. Additionally. plan to evaluate the performance of more cutting-edge L.MSs on semi-open-ended questions and integrate user feedback mechanisms guide L.MMs in recognizing their limitations.
E
Limitations
We only investigate the performance of one black-box LLM, GPT-4 our dataset due expensive human annotation. As a result, we have not engaged in comparative analysis with other black-box models, such galcon-180B, which could have provided a broader perspective on the performance metrics. Our dataset also does not contain all unpopular answers to because there may be hundreds and thousands of potential answers that require expert knowledge. Given the complexities and the extensive scope involved, it was deemed unfeasible to incorporate this aspect within the purview of our current study.
questions
F
Social Impact
We discuss the social impact of our research on the knowledge boundaries of LLMs as follows. On one hand, our work could enhance the reliability of Al systems by identifying their knowledge limits, thereby improving user trust and the accuracy of -eeerrttee information. also contributes to reliable AI development by emphasizing the importance of factual information in AI outputs, which can particularly beneficial educational and professional settings. On the other hand, there is a risk that the ambiguous answers generated by LLMS could lead to misinformation if not properly resolved. Additionally, an overreliance on LLM for knowledge-seeking could undermine human critical thinking.
16

---

G
Case Study
Table 6: Examples of two ambiguous answers with their related questions. The Raspberry is an unqualified answer generated by GPT-4, which yields a different answer in another question. Plantain an answer that is neglected by GPT-4, yet supplemented by the auxiliary model, which is correct according to ground truth. However, GPT-4 believes it to be wrong and generates wrong information for another question. Texts in yellowgreen are truthful information, while texts in red are non-factual.
Semi-open-endee Question
Tell list foods
rich in Vitamin
GPT-4 Response
1. Carrots to 2. Spinach 46. Raspberries in 47. Red Leaf Lettuce
For semni--pee-mdle Question
Auxiliary Model
1. Bell poppers ln 2. Liver i. 14, Meal such boof liver In 15, Plantains
Response
Ambiguous Answer
Raspberrie
Plantains
Answer Type
U Unqualittod Answer
Hidden Correc Answer
Related Question
vitamins are rich in raspbernes?
Is a good choice Plantains many Vitamin AY
GPT-4 Response
Raspberries are rich in vitamins C, K, E, and B B-comples.
They also contain small amounts vitamin
but not in very high amounts...
Plantains do contain vitamin A.
We showcase the importance of ambiguous answers in perceiving the knowledge boundary for LLMs. Tab. [0] shows example with two ambiguous answers for the same question. First, we sample a semi-open-ender question with its GPT-4 responses and ambiguous answers augmented by the auxiliary model (row 2 and in table Tob Then, we manually construct two related questions, each involving different types of ambiguous answers, and request new responses from GPT-4, For the case displayed on the left side of Tab T0 GPT-4 falsely deems the raspberry as food that is rich in vitamin A but low in fat, yet it answers correctly in another related question. It indicates that GPT-4 inconsistent in answering different questions involving the same ambiguous answer. For the case on the right side of TTb given the same semi-open-ended question, the auxiliary model discovers another ambiguous answer, Plantain, which is correct the question. Interestingly, GPT-4 generates misinformation regarding this answer entity. It shows that GPT-4 falsely believes plantain is incorrect. It also indicates that the auxiliary model helps discover ambiguous answers that elicit misinformation in the target LLM. It strengthens the importance of perceiving knowledge boundaries for LLMs by discovering ambiguous answers memi-open-ended questions. See App. H for more examples.
H Cases of Different Types of Ambiguous Answers with Misinformation on the Related Questions
We show different types of ambiguous answers and GPT-4's performance on their related questions from Tab to Tab g See full cases in Tab 9 Tab 10 and Tab I1
Table 7: Examples of two ambiguous answers with their related questions. The Malabo, Equatorial
Guinea is an answer that is neglected by GPT-4, yet supplemented by the auxiliary model, which is correct according to ground truth. However, GPT-4 believes it to be wrong and generates wrong information for another question. Rome, Italy is an answer that is neglected by GPT-4 but generated by the auxiliary model, whose self-evaluation results misalign with the ground truth. However, GPT-4 believes to be wrong and generates wrong information for another question. Texts yellowgreen are truthful information, while texts in red are non-factual.
Question
Tell me a list of world capitals where more than two languages considered official.
GPT-4 Response
1. London, United Kingdom In 2. Montreal, Canada
for Semi-open-ended Question
$2, Zugreb, Croutia In 53, Prague, Czech Republic Brussels, Belgium In 2. Ottawa, Canada ...
Auxiliary Model
1.
Respouse
Rome, Italy 'n 10. Malabo, Equatorial Guinea
Ambiguous Answer
Mulabo, Equatorial Guinea
Rome, Italy
Answer Type
Hidden Correct Answer
Unexpected Wrong Evaluations Is Latin the official language of Rome?
Related Question
How many official languages are there in Malaho?
There are two official languages
GPT-4 Response
No, the official language Rome is Italian.
in Malabo: Spanish and Pidgin English.
17

---

Table 8: Examples of two ambiguous answers with their related questions. The Sand Island Light an unqualified answer generated by GPT-4, which yields a different answer to another question. The Portsmouth Harbor Light is an answer from GPT-4, whose self-evaluation results contradict the ground truth. However, GPT-4 believes it to be true and generates wrong information for another question. Texts in yellowgreen are truthful information, while texts in red are non-factual.
Semn-opee--nned Question
Tell me list of famous lighthouses located on islands in the Allantic Ocean.
GPT44 Response
Pemsquid Point Light 'n 2. Poriland Head Light o. 46. Sand Island Light la 47. Pottsmouth Harbor Light la
for Semi-open-ended Question
Auxiliary Model
Maing Cousi Giard Station and Pier Light 'n 2. Minut's 's Leading Light
Respome
Woodenbout School Light lo Chappaquidlicc Light
Ambiguon Answer
Pottsmouh Harber Light
Sand Ishad Light
Answer Type
Iraccurtt Evalutions
Unqualified Answer
Related Question
Is Portsmooth Harbor Light locased on which island in the Atlantic Gcean?
Where is Sand Island Light located?
Sund Island Light located Alibama, USA,
GPT-4 Response
Puttsmouth Hurbor Light located New Casile Island the Atlanii Occan, in the state of New Hampshäre, USA.
specifically Sand Island off esast of Dauphin Island
Table 9: The full version of the cases in Tab
Semi-open-roded Question
TSll me list of foods that are rich in Vitamin A but low in fat.
1. Carrots 2. Spinach In 3. Sweet Potato 'n 4. Kale 'n S. Butternut Squash 'n
6. Red Bell Poppers 7. Apricots in 8. Broccoli Va 9. Romaine Lettace 'n Tangerines
Papaya 15. Peaches 'n 16, Mangoes in 17. Watermelon In 18, Green Peas in 19, Kiwi In
Contaloupe lo 11. Yollow Corn
Tomatocs
20. Honeydow Melon ta 21. Asparagus la 22, Black-Eyed Pess In 23. Plums to
24. Pincapple in 25. Blueberries in 26. Grapefruit 27. Oranges 28. Green beans In
29. Brussel sprouts. la 30. Guava in 31. Dandelion Greeos In 32. Swiss Chard 'n
GPT.4 Response
Rose Hips In 38. Parsley la 39. Watereress In 40, Radish In 41, Red Cabbage In 42. Bost Greens 'n
Mustard Greens 34, Arctic Char 35, Prunes
Nectarines
for Seml-open-ended Question
43. Endives la 44, Arugula ln 45, Pomogranates ln 46. Raspberries to 47. Red Leaf Lettuce in
48. Pumpkin 49. Bok Choy 'n 50. Tumip Greens in 51. Red Grapes In 52. Passionfruit In
53. Collard Groens in 54, Zucchini lo 55. Strawberries In 56, Clementines In 57. Lima Beans la
58. Cucumbers 59, Yellow Bel Peppers
Pawpaw 61. Winter squash 62, Cantaloupes In
63. Tuna tn 64. Persimmon 65, Piak Grapefruit la 66. Honeydew melon In 67. Guavas In
68. Papayas in 69. Passion fruit In 70. Kiwi fruit In 71. Blackherries \n 72. Cherries In
73. Goji bemres in 74, Raspberry 75, Fenugreck leaves 76. Okra in 77. Chayote in Eggplant
I. Bell peppers in 2. I Liver In 3. Collards In 4. Camots In 5. Spinach 'n
Yogurt 7. Salmon in 8. Whole grains In 9. Fortified cercals
Clams
Auxiliary Model Response
II. Milk in 12. Shrimp in 13. Whole eggs In 14, Meal such beef liver in 15. Plantains
16. Fresh herbs like basil, parsley and dill in 17, Cheese (low-fat varieties) 'n
18. Vegetable oils such as soybean corn
Table 10: The full version of the cases Tab 7
Semi-open- ended Question
Tell me a list of world capitals where more ihan two languages are considered official.
1. London, United Kingdom to 2. Montrcal, Canada in 3. Brassels, Belgium 4. Ottawa, Canada Suva, Fiji la 8. Biel/Beenne, Switzerland
Bern, Switzerland 'n 6. Johannesburg, South Africa
9. Singapore. Singapore In 10, Luxembourg City. Luxembourg In 11. New Delhi, India
12. Addis Ahaba, Fihiooia In 13, Nairobi, Kenya In 14, Helknnki, Finland 15. Vienna, Austria In
16. Maputo, Mozambique ln 17. Oslo, Norway's la 18. Reykjavik, Iceland In 19, Valletta, Malia la
Nicosia, Cypmis 21. Dublin, Ireland in 22, Vaduz, Liechtenstein In 23. Tirana, Albania In
24. Andorm la Vella, Andorra \n 25. Baku, Azerbaijan in 26. Manama, Bahrain In 27. Dhakn, Bangladesh In
28. Saraevo, Bosnia and Herzegovina 29. Sonia, Bulgaria la 30. Copenbagen. Denmark la In 34. Jerusalem, Israel 'n
GP'T-4 Response
31. Tallinn, Estonia 32. Tbilisi, Georgia ln 33. Baghdad,
for Semi-open-ended Question
35. Astana, Kazakhstan in 36. Beirut, Lebanon In 37. Tripoli. Libys la 38. Kualn Lumpur, Malaysia In Islamabud, Pakistan
39. Abuja, Nigoria
Bucharest, Romania ko 43. Riyadh, Saudi Arabia in 44, Madrid, Spain in 45. Lome, Togo In 46. Ankara, Turkey In
Dota. Qatar
47. Kiev, Ukminne in Dubai, United Arab Fmirates In 49. Canherra, Australia
Prain, Cape Verde In 51. N'Djumena, Chud in 52. Zugreb, Crostia in 53, Prague. Czech Republic in 54, Beijing. China
55. San Jose, Costa Rica In 56. Santo Domingo. Dominican Republic In 57. Quito, Ecuador In
Cairo, Egypt 59, Port Vila, Vanuatu in 60. Windhoek, Namibia la 61, Kathmandu, Nepal In
62. Wellington, New Zealand to 63. Asuncion, Paraguay in 64. Lisben Portugal In
Moscow, Russia in 66. Dakar, Senegal 'n 67. Ljuuijana, Slovenia
Brussels, Belgium in 2. Ottawa, Canada in 3. Panama City, Panama in 4. Lima, Peru 'nI
5. Taronto, Canada in 6. Paris, France la Hong Kong, China 'n 8. Moskva, Russia In Athens, Greeee
Auxiliary Model
9. Rome, Italy ln 10, Malabo, Equatorial Guinca
Stockholm, Sweden
Respouse
13. Tokyo, Japan to 14, Jakasta, Indonesia la 15. Mexico City, Mexico in 16. Bangkok. Thailand In
São Paulo, Brazil Buenos Aires, Argentinu Caracas, Venezulla in 20. Bogota, Colombia
18

---

Table 11: The full version of the cases in Tab 8 famous ligihooses located
Semi-open-ended Question
Ocean.
Pemaquid Point Light in 2. Foriland Head Light In Cupe Hatterus Lighthouse in 4. Tyhee Island Light
Light 7. Lindan Lighthouse in 8. Quessant Lighthouse 9. St. Mary's ss Lighthouse in 10, Belle Tout Lighthouse In 11. Lighthouse of Genoa in 12. Tower of Hercules to
Boston Light
George
13. Lighthouse of Alexandria 14, Capelinhos Lighthouse in 15. Start Point Lighthouse la
16. Cupe Spear's Lightbouse, New Newfoundland In 17. Sumbro Island Lighthouse In
18. Cape Borda Lighthouse in 19, Amédée Lighthouse 'n 20. Roches Douvres Lighthousee In
21. Bishop Rock Lighthouse in 22 Skenyvore Lighthouse 23. Fastnct Rock Lighthouse
24. Thacher Island Twin Lights 25. Cape Lookout Lighthouse In 26, Point Judith Lighthouxe
Bass Harbor Head Lighthouse in 28. Sandy Hook Lighthouse 29, Montauk Point Lighthouse
30. Cape May Lighthouse vn 31. Lundy Island Lighthouse 'n 32. Lighthouse at Il Vierge In
Tory Island Lighthouse 36. Cape Wrath Lighthouse In 37. Eilean Glas Lighthouse In 38. Andnamrrchan Lighthouse 'n
Maughold Head Lighthouse
Ballycntion Island Lighthouse
Longstone Lighthouse
Happishurgh Lighthouse 41. Kinnaird Head Lighthouse
42. The Ncedles Lighthouse la 43. Corbicre Lighthouse ta 44, Casquets Lighthouse 'n
45. The Round Island Lighthouse In 46. Sand Island Light in Portsmouth Harbor Light
GPT-4 Respore
48. Neist Point Lighthouse in 49. Ilannan Isles Lighthouse in 50. Cromarty Lighthouse 'n
Semi-open-endel Question
51. Fair Isle South Lighthouse in 52. Copinsay Lighthouse in 53, North Ronaldsay Lighthouse 'n
Noup Head Lightbouse 55, Sule Skerry Lighthouse 56. Skokholm Lighthouse
57. Muckle Plugga Lighthouse 58. Brier Island Lighthouse In 59, Articled Rock Lighthouse 'n
60. Nubbic Lighthouse 61. Cape Recife Lighthouse
Temriblc Tilly
63. Butt of Lewis Lighthouse in 64, Scaham Lighthouse In 65. Girdle Ness Lighthouse In
Lighthouse 69. Hirtsalk Lighthousse In 70. Cordouan Lighthouse in 71. Sonkaty Head I Light in 72, Gay Head Light 'n
66. Walney Lighthouse 67. Europa
Lighthouse
Whales
Cape Romain Lighthouses to 74, Ram Island Lodge Light la Boca Grande Lighthouse
76. Petit Manan Light 77. Cape Elizabeth Light to 78. Cape Ann Light Station
79. Rose Island Lighthouse in 80. Saint Augustine Light in 81. Egmont Key Lighthouse 'n
82. Dry Tortugus Light 83. Nobska Point Light in 84, Sullivan's Island Lighthouse In
85. Hatteras Island Lighthouse in 86, Hunting Island Lighthouse in 87, LAnge Erik in 88. LAnge Jan
89. Ushunt Island Lighthouse in 90. Blackhead I Lighthouse in Hook Lighthouse la
92, Cape Clear Island Lighthouse in 93, Rathlin West Lighthouse in 94, Copcland Lighthouse In
95. Fisgard Lighthouse
Quoddy Head Light
Cape Jourimain Lighthouse
Maine Coast Guard Station and Pier Light in 2. Minot's s Leading Light Vn 3. Monhegan Island Light
Eastern Breakwater Light 5. Little River Light to 6. Baldhead Island Light
Auxiliary Model
7. Oak Bluft's Harber Light
Brenton Reef Lightn 9. Block Island Southwest Light In
Response
10. Woodenbost School Light 11. Chappaquidickk Light 12. Race Rock Light la
13. Plum Gut Light 14. Great Round Shoal Light 15. New London Island Lighthouse
16. Smith's 's Island Light in 17. Highfield Hall Beach Light in 18. Scituate Light House
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The last paragraph of our Introduction accurately reflects the paper's contribu- tions and scope. See Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in the appendix. See App. E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

We provide the full set of assumptions and complete (and correct) proof in Perception of Knowledge Boundary for LLM. See Sec.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose the implementation details, Dataset Information, Important Instructions and the results of experiment in Experiments and appendix. See Sec. and
App.
B A
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We release our data and code with instructions other researchers to reproduce our results. We share these resources in https://gthubb.com/aaraaakkklk/
knowledgeBBunddary
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
wwwp://wweeeeeeeeeeeeeeeee Saatlees/oooSSmmiiiiia SowpoddeumiissiooPoo.. wwriiisssiiiicyoyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We fully disclose the implementation details, Important Instructions in Experi-
ments and appendix. See Sec. 4.I and App. B A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: I would like to clarify that the validation of experimental results currently requires manual assistance, and the cost of repetition relatively high. Therefore, we have not conducted significance analysis this stage. We plan to address this in future work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information the computer resources (type of compute workers, memory, time of execution) in the appendix. See App. B
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httts://neurips.c httpeipppp.cpppplcceeeeee ctw.hiieeeeecccccceeeeeeeeeee
Answer: [Yes]
Justification: When generated the dataset, we added number of requirements to make the dataset comply with the NeurIPS Code of Ethics. See App
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the social impact of our work from both positive and negative perspectives in App
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, we describe guidelines for the safe use of our data in the readme file in our GitHub repository.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, we have cited the creator and releaser of our used LLMs the reference.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode.hco... com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we provide a detailed description of our dataset in App A Besides, we also introduce our dataset in the repository where we release the data.
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We introduce the settings for our human evaluation in the paper and include the full instructions in the supplementary materials.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Yes, the paper thoroughly addresses potential risks to study participants, ensuring that all ethical considerations are met. We confirms that necessary Institutional Review Board (IRB) approvals, or equivalent reviews required by the country or institution, were obtained prior to conducting the study. This demonstrates strong commitment to ethical research practices and the protection of participants' rights and well-being.
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "5zhgGRu6ZK", "reviewer": "Reviewer_e3tW", "summary": "The paper presents a study of generating answers to questions from the tail of an LLM's distribution (GPT-4). They begin by constructing a dataset by generating questions with multiple answers from GPT-4. For each question, they continue decoding multiple answers and define the first 75% of generated answers as \"common-sense answers\" and the last 25% as \"ambiguous answers\". Their objective is to then to develop methods for generating such \"ambiguous answers\" from GPT-4 more efficiently, not relying on prompting many times (as they did to construct their dataset) or decoding parameters (which are not fully accessible for many API services).\n\nTheir approach relies on using an auxiliary language model (LLama) to generate low-probability answers by reducing the probability of existing answers.They evaluate their method for generating low-probability answers using their constructed dataset, evaluating its ability to recover \"ambiguous answers\" as defined above. \n\nThey conclude their work by performing analysis of these tail answers (\"ambiguous\" answers in their dataset and answers generated using their method), using retrieval+GPT-4 to verify each of these tail answers for correctness. They find that roughly half of these answers could were correct using thier system.", "questions": "See the last point in weaknesses above.\n\nAlso, its somewhat unclear to me how LLAMA model is used as an auxiliary model for GPT-4. Is it auxiliary because GPT-4 can be used to validate generated answers?", "limitations": "See final note under weaknesses", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work develops a method for generating a large, diverse answer candidate set, which may be useful for a number our other tasks.\n\nThe authors perform several ablation experiments, demonstrating the efficacy of different components of their method and sensitivity to hyperparameters.", "weaknesses": "Methods for generating a diverse set of answer candidates are evaluated against their ability to recover the set of tail-answers from GPT-4. Analysis, however, demonstrated that roughly half of these tail-answers are incorrect. Evaluations are therefore designed to generate answers that match GPT-4's tail distribution and generating the same set of incorrect answers. \n\nThe evaluation dataset is quite small and results only demonstrate minor improvement without significance testing.\n\nMany components of the work lack explanation. The authors reference \"human annotation\" for verifying answers; however, it is not clear how this is done, or what the exact instructions were. While the checklist notes that full annotation instructions and compensation details were provided in supplementary materials, but I did not find these. The retrieval-based system used for validating answers also lacks description, only noting that they used Microsoft Copilot to perform retrieval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable questions! We have incorporated the suggested experiment and provided important clarifications. We hope that we have addressed your concerns and resolved possible misunderstandings. Based on our clarifications, we would greatly appreciate it if you would reconsider the final evaluation of our work.\n\n## Q1: Clarification on the possible misunderstanding of our task\n\nWe kindly point out \"Methods ... to recover the set of tail-answers from GPT-4 … roughly half of these tail-answers are incorrect.\" in your review is a misunderstanding. It should be clarified that \"Methods ... are evaluated based on their ability to identify shortcomings (new ambiguous answers) in the target LLM.\"\n\nOur task is not to “recover the set of tail-answers from GPT-4”.  Instead, **our motivation is to identify LLMs’ shortcomings by discovering more different ambiguous answers for LLMs where they tend to make mistakes** (beyond the knowledge boundary of GPT-4), refer to Lines 11, 39 and 61 of our paper for details. In next version, we will be clearer about our task to avoid such misunderstandings.\n\n## Q2: Unclear about how the auxiliary model is used\n\nIntuitively, **the auxiliary model aims to generate hard (ambiguous) answers that reveal GPT-4's shortcomings**—answers that GPT-4 is easy to make mistakes (as discussed in Lines 71 and 181). That is, we employ the auxiliary model (LLaMA) to uncover many new ambiguous answers that GPT-4 struggles to produce. Our findings show that **50% new ambiguous answers found by the auxiliary model reflect the shortcomings of GPT-4** (see Sec 4.5). Specifically, we use the Pseudo-inverse model embedding to estimate the nearest semantic representation of the existing answers (from GPT-4), then reduce their generation probability to produce new answers with lower probabilities (see Sec 3.3). \n\n## Q3: Improvement is minor\n\nFollowing your suggestion, we conducted a bootstrap significance test to compare our method with the baselines. We found that the improvement in the Average Overlap Ratio (AOR) is **statistically significant under the t-test with p<0.05**. This indicates that our method effectively reduces answer repetition without compromising overall performance.\n\nThe core empirical contribution of our work extends well beyond the results presented in Table 1 of our paper. **A more significant contribution is identifying GPT-4's shortcomings** (please see Line 85): Our findings reveal that GPT-4 yields unsatisfactory results in 82.9% of the questions. Furthermore, about 50% of the new ambiguous answers identified by our method fall outside the knowledge boundary of GPT-4 (see Sec 4.5).\n\nOur performance has been acknowledged by all the other reviewers. Reviewer frNN praises our work, stating, \"Experiments show significant improvements in understanding LLM knowledge boundaries\", Reviewer hUqe comments, \"The experiments are solid enough\", and Reviewer bcni observes, \"The performance of the proposed methods seems effective in detecting GPT-4’s knowledge boundary.\"\n\n## Q4: The dataset is quite small\n\n**Our dataset is comparable in size to many other hallucination evaluation datasets within the research community**, as shown in the following table. It covers multiple domains and is highly effective at identifying the knowledge boundary of GPT-4. It successfully identifies shortcomings in GPT-4's responses for 82.9% of the questions, using only vanilla prompts.\n\n| Dataset | TruthfulQA(Lin et al., 2022) | HaluQA(Cheng et al., 2023) | FreshQA(Vu et al., 2023) | FELM(Chen et al., 2023d) | ChineseFactEval(Wang et al., 2023a) |\n| ------- | ---------------------------- | -------------------------- | ------------------------ | ------------------------ | ----------------------------------- |\n| Size    | 817                          | 450                        | 600                      | 817                      | 125                                 |\n\n**Constructing and expanding the dataset is expensive and time-consuming.** Specifically, It takes a human annotator about 2 minutes to assess and double-check the truthfulness of each answer. Our dataset contains approximately 1k questions, each with an average of 13 tail-answers to be verified. Considering the requirement for annotators to read and assess the credibility of the retrieved information (following our annotator principle in Appendix. A), the total working hours are 435 hours. As we paid 8 dollars per person per hour, it cost us 3483 dollars to construct the dataset. Even if we hire more than 5 annotators, it would still take more than 72 hours (**more than 7 days if they work 10 hours per day**), exceeding the time limit of the rebuttal period.\n\n## Q5: Evaluation and human annotation guidance need a clearer explanation\n\nWe kindly point out that we have introduced the human annotation procedure in Appendix A and provided full guidelines in the file named **\"human guide.docx\" in the supplementary material**, as well as a reminder in Appendix G. Specifically, in human evaluation, we ask the annotators to read the judgments generated by the retrieval-based evaluation system (Microsoft Copilot), assess the authority of the retrieved information, and evaluate the degree of certainty of the tone of the Copilot judgments. Then, annotators categorize answers into correct, incorrect, and unverifiable (see Line 563 of our paper and **the fourth note in \"human guide.docx\" in the supplementary material**).\n\nIn retrieval-based evaluation, we instruct a RAG system with well-designed instructions (please see details in Sec 3.4) to verify the truthfulness of each tail answer. Specifically, we concatenate each candidate's answer with the question and prompt Microsoft Copilot to search online for related information, generate a summary, and make judgments (see Appendix G for full instructions). In our revised paper, we will explain these components in the experimental setting to avoid such confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nnwPzQsAtp", "reviewer": "Reviewer_bcni", "summary": "In this submission, the authors aim to explore the detection of large language models’ (LLMs)  knowledge boundary, which is a borderline to tell us what can LLMs really know. The detection of knowledge boundary wound play a crucial role to help the researchers to deal with hallucination. Different from the widely-used QA setting, the paper focuses on another QA setting (semi-open-ended questions), which may be ignored by other researches but seems common in our real life. To detect the knowledge boundary, the authors hire a small-scale open-source model (Llama) to help the GPT-4 to detect GPT-4’s boundary. This paper have some interesting findings. For example, GPT-4 underperform the auxiliary model, LLaMA-2-13B in 50% cases.", "questions": "1. As mentioned in Weaknesses, could you please give a fully version of cases in Table 6 and 7?", "limitations": "Yes, the limitation section addressed by the authors seems fine with me.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The authors explore in a new question answering setting (i.e. semi-open-ended questions). This setting is unexplored in QA tasks around LLM, and it seems quite interesting and useful for LLMs.\n\n2. Detecting the knowledge bounding is a promising and crucial direction to achieve a better usage of LLM and deal with the notorious hallucination problem.\n\n3. The proposed idea makes sense and novel enough, which use a open-source LLM to assist the black-box LLM (GPT-4) to detect GPT-4’s knowledge boundary by accessing the prob dist and find the low-prob words. The minor changing of the representation of output hiddens helps to obtain the most similar answers for LLMs. \n\n4. The paper writing is quite clear to follow with detailed graphical descriptions (the overview figure).\n\n5. The performance of the proposed methods seems effective in detecting GPT-4’s knowledge boundary. It found that in some (50%) cases, GPT-4 underperform the auxiliary model, LLaMA-2-13B.", "weaknesses": "1. The setting of “semi-open-ended questions” is quite interesting and seems useful and common in our real life. However, is the term of “semi-open-ended questions” widely used in the QA research area? Although I am not so familiar with with QA and its terminology, I suggest the authors conduct a fully survey of the related terms and explain more about the meaning of “semi-open-ended questions” in the next version.\n\n2. The proposed method works well on the “semi-open-ended questions”. It would be better to give a detailed statistic (e.g. questionnaire) or empirical study or related analyses about “the proportion of semi-open-ended questions out of the whole QA scenario”. It shows the practical use of the proposed method in our real life.\n\n3. Could you please illustrate the full version of the cases in Table 6 and Table 7 (Some cases are omitted due to the space limit)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are immensely grateful for your insightful and positive comments. we have addressed each point with careful consideration to ensure that our findings are presented with greater precision and rigor.\n\n## Q1: Explanation regarding the meaning of \"semi-open-ended questions\"\n\nThank you for your insightful comment and suggestion. Based on our research, we have not found evidence of the term \"semi-open-ended questions\" being widely used in the QA research area. These questions, though having multiple potential correct answers, are challenging to fully enumerate. For example, when asked to \"Tell me about some exercise habits that are easy to overlook but are good for your health,\" there are many correct answers, yet the question remains constrained by the context of 'exercise habits', 'easy to overlook' and 'good for your health'.\n\nIn the next version, we will provide a more detailed explanation of the term \"semi-open-ended questions\" including diverse examples to facilitate a comprehensive understanding of our study.\n\n## Q2: The proportion of semi-open-ended questions out of the whole QA scenario\n\nFollowing your suggestion, we find that in real life, semi-open-ended questions are quite common. We extract all question statements from an open-source general-domain corpus OpenWebText [1] and randomly sample 1k questions. We conduct statistics to find that approximately 33.6% of the questions are semi-open-ended, indicating that the research problem we are addressing has strong practical significance. We distinguish semi-open-ended questions by querying ChatGPT.\n\n## Q3: The full version of the cases in Table 6 and Table 7\n\nFollowing your suggestion, we demonstrate the full cases in Table 6 and Table 7 of our paper as follows. We will provide the full version in our next version.\n\n| Semi-open-ended Question                    | Tell me a list of foods that are rich in Vitamin A but low in fat. |\n| ------------------------------------------- | ------------------------------------------------------------ |\n| GPT-4 Response for Semi-open-ended Question | 1.Carrots 2. Spinach 3. Sweet Potato 4. Kale 5. Butternut Squash 6. Red Bell Peppers 7. Apricots 8. Broccoli 9. Romaine Lettuce 10. Cantaloupe 11. Yellow Corn 12. Tangerines 13. Tomatoes 14. Papaya 15. Peaches 16. Mangoes 17. Watermelon 18. Green Peas 19. Kiwi 20. Honeydew Melon 21. Asparagus 22. Black-Eyed Peas 23. Plums 24. Pineapple 25. Blueberries 26. Grapefruit 27. Oranges 28. Green beans 29. Brussel sprouts. 30. Guava 31. Dandelion Greens 32. Swiss Chard 33. Mustard Greens 34. Arctic Char 35. Prunes 36. Nectarines 37. Rose Hips 38. Parsley 39. Watercress 40. Radish 41. Red Cabbage 42. Beet Greens 43. Endives 44. Arugula 45. Pomegranates 46. Raspberries 47. Red Leaf Lettuce 48. Pumpkin 49. Bok Choy 50. Turnip Greens 51. Red Grapes 52. Passionfruit 53. Collard Greens 54. Zucchini 55. Strawberries 56. Clementines 57. Lima Beans 58. Cucumbers 59. Yellow Bell Peppers 60. Pawpaw. 61. Winter squash 62. Cantaloupes 63. Tuna 64. Persimmon 65. Pink Grapefruit 66. Honeydew melon 67. Guavas 68. Papayas 69. Passion fruit 70. Kiwi fruit 71. Blackberries 72. Cherries 73. Goji berries 74. Raspberry 75. Fenugreek leaves 76. Okra 77. Chayote 78. Eggplant. |\n| Auxiliary Model Response                    | 1.Bell peppers 2. Liver 3. Collards 4. Carrots 5. Spinach 6. Yogurt 7. Salmon 8. Whole grains 9. Fortified cereals 10. Clams 11. Milk 12. Shrimp 13. Whole eggs 14. Meat such as beef liver 15. Plantains 16. Fresh herbs like basil, parsley and dill 17. Cheese (low-fat varieties) 18. Vegetable oils such as soybean oil or corn oil |\n\n\n\n\n\n| Semi-open-ended Question                    | Tell me a list of world capitals where more than two languages are considered official. |\n| ------------------------------------------- | ------------------------------------------------------------ |\n| GPT-4 Response for Semi-open-ended Question | 1. London, United Kingdom 2. Montreal, Canada 3. Brussels, Belgium 4. Ottawa, Canada 5. Bern, Switzerland 6. Johannesburg, South Africa 7. Suva, Fiji 8. Biel/Bienne, Switzerland 9. Singapore, Singapore 10. Luxembourg City, Luxembourg 11. New Delhi, India 12. Addis Ababa, Ethiopia 13. Nairobi, Kenya 14. Helsinki, Finland 15. Vienna, Austria 16. Maputo, Mozambique 17. Oslo, Norway 18. Reykjavik, Iceland 19. Valletta, Malta 20. Nicosia, Cyprus 21. Dublin, Ireland 22. Vaduz, Liechtenstein 23. Tirana, Albania 24. Andorra la Vella, Andorra 25. Baku, Azerbaijan 26. Manama, Bahrain 27. Dhaka, Bangladesh 28. Saraevo, Bosnia and Herzegovina 29. Sofia, Bulgaria 30. Copenhagen, Denmark 31. Tallinn, Estonia 32. Tbilisi, Georgia 33. Baghdad, Iraq 34. Jerusalem, Israel 35. Astana, Kazakhstan 36. Beirut, Lebanon 37. Tripoli, Libya 38. Kuala Lumpur, Malaysia 39. Abuja, Nigeria 40. Islamabad, Pakistan 41. Doha, Qatar 42. Bucharest, Romania 43. Riyadh, Saudi Arabia 44. Madrid, Spain 45. Lome, Togo 46. Ankara, Turkey 47. Kiev, Ukraine 48. Dubai, United Arab Emirates 49. Canberra, Australia 50. Praia, Cape Verde 51. N'Djamena, Chad 52. Zagreb, Croatia 53. Prague, Czech Republic 54. Beijing, China 55. San Jose, Costa Rica 56. Santo Domingo, Dominican Republic 57. Quito, Ecuador 58. Cairo, Egypt 59. Port Vila, Vanuatu 60. Windhoek, Namibia 61. Kathmandu, Nepal 62. Wellington, New Zealand 63. Asuncion, Paraguay 64. Lisbon, Portugal 65. Moscow, Russia 66. Dakar, Senegal 67. Ljubljana, Slovenia |\n| Auxiliary Model Response                    | 1. Brussels, Belgium 2. Ottawa, Canada 3. Panama City, Panama 4. Lima, Peru 5. Toronto, Canada 6. Paris, France 7. Hong Kong, China 8. Moskva, Russia 9. Rome, Italy 10. Malabo, Equatorial Guinea 11. Stockholm, Sweden 12. Athens, Greece 13. Tokyo, Japan 14. Jakarta, Indonesia 15. Mexico City, Mexico 16. Bangkok, Thailand 17. São Paulo, Brazil 18. Buenos Aires, Argentina 19. Caracas, Venezuela 20. Bogota, Colombia |\n\n## Reference\n\n[1] Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText Corpus.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mvjk8tjJTs", "reviewer": "Reviewer_hUqe", "summary": "This paper focuses on detecting the “knowledge bounding” of the current large language models (LLMs), which would be helpful in handling the well-known hallucination problem in LLMs. In this paper, the authors explore in a new question answering setting (i.e. semi-open-ended questions). The authors employ LLM-based approach to construct semi-open-ended questions and obtain answers from a target LLM. Employing a open-source LLM, which is flexible to edit the parameters/variables, to detect black-box LLMs’ drawback. The proposed method that calculating the nearest semantic representation to select the related answers seems interesting. Finally, the paper not only constructs a dataset to find the knowledge boundary for GPT-4, but also indeed discovers 82.90% of GPT-4’s answers are not so satisfied. It also founds that 40.15% of its hard (ambiguous) answers generated are unqualified.", "questions": "1. Though the proposed model shows its effectiveness on semi-open-ended question setting, I am wondering the potential future work on normal QA tasks (e.g. multiple-choice test). Could you please provide some insights on the proposed models’ potential strengthen on the normal QA tasks?", "limitations": "The authors' limitation section indeed addressed some concerns about this paper, which is fine with me.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method sounds novel for handling the knowledge boundary discover tasks. The method utilizes an open-source Large Language Model (LLM) to aid a black-box LLM, GPT-4, in identifying its knowledge limitations. They pinpoint words with low probabilities according to the model variable of the open-source LLM. Subtle modifications to the representation of the output layers facilitate the extraction of answers that are most analogous for LLMs.\n\n2. The proposed semi-open-ended question task sounds important to explore. The task is more challenging than the current QA tasks, since the number of candidate answers and answer space (and may be the correct answers) is not fixed and deterministic.\n\n3. The paper writing is quite clear and the paper organization is easy to follow.\n\n4. This paper also fall into a popular and important direction. It tries to solve the effect of hallucination in LLMs in another way: different from case-by-case detecting the hallucination  cases, this paper aims to discover the knowledge boundary of a given LLM (GPT-4). \n\n5. The experiments are solid enough. It includes multiple base models (Llama with different size) and multiple evaluation metrics. The overall performance also includes study cases.\n\n6. As GPT-4 is quite powerful, detecting the knowledge boundary of GPT-4 and find the shortcomings of GPT-4 is not an easy duty. The outcome of this paper is quite attractive, since it discovers 82.90% of GPT-4’s answers are not so satisfied with a simple LLama2 model.", "weaknesses": "1. The proposed model works well on detecting the knowledge boundary in the setting of semi-open-ended question. Even if that setting makes sense, (1) I am not sure about how many questions in the real applications belong to “semi-open-ended question”. (2) how about the effectiveness of the proposed method on normal QA tasks (e.g. multiple-choice test). Could you please provide some insights on the proposed models’ potential strengthen on the normal QA tasks? \n\n2. The model name used in the paper should by consistent. For example, the authors use both “LLaMA-2-13b” (in line 252) and “LLaMA-2-13B” (table 1) in the paper.\n\n3. Typos: In line 252, it should be “we use two LLaMA-2-13b models”. The authors missed to add “model”. In table 1, the title of the second column should be “Auxiliary Model” instead of “Auxiliary Model Size”, since LLaMA-2-13B is a name instead of a kind of size.\n\n4. What is the meaning of the underline results in Table 3? The authors should make it clear of each mark or notation in the paper is well-described.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful and encouraging feedback! We have carefully incorporated your suggestions and provided explanations as follows, aiming to enhance the quality and robustness of our research.\n\n## Q1: How many questions in the real applications belong to \"semi-open-ended questions\"?\n\nFollowing your suggestion, we find that in real life, semi-open-ended questions are quite common. We extract all question statements from an open-source general-domain corpus OpenWebText [1] and randomly sample 1k questions. We conduct statistics to find that approximately 33.6% of the questions are semi-open-ended, indicating that the research problem we are addressing has strong practical significance. We distinguish semi-open-ended questions according by querying ChatGPT.\n\n\n\n## Q2: Insights on the proposed models' potential strengthen the normal QA tasks.\n\nOn normal QA tasks, our method can help discover highly delusive wrong answers by reducing the probability of the ground truth answer. By comparing them with the ground truth, we may explain the factual hallucination problems at a more granular level. Besides, they can be used to construct a more challenging benchmark for normal QA tasks by including more delusive wrong answers.\n\nOur approach of modifying the LLM representations to guide answer generation may provide insight for different kinds of normal QA tasks:\n\n1. It may help alleviate the hallucinations in knowledge-extensive QA tasks via representation engineering.\n2. Editing LLM representations considering existing answers can reduce the probability of semantically related words, helping to generate more diverse answers for open-ended QA tasks.\n\nBesides, ambiguous answers found by our approach benefit QA systems in many ways, including:\n\n1. Flagging ambiguous answers with higher uncertainty enhances the LLM-based QA systems [2, 3];\n2. Identifying ambiguous answers helps achieve selective retrieval that augments LLM-based QA systems with indispensable external information while reducing the distraction of irrelevant data [4, 5, 6].\n\nWe will discuss these potential research directions in our revised paper and explore them in our future works.\n\n## Q3: Inconsistent model name & grammatical errors.\n\nThank you for your careful reading! We will unify the naming of model names and resolve all grammatical errors in the revised version.\n\n## Q4: Meaning of the underlined results in Table 3.\n\nThe underlined results are either incorrect or unverifiable according to the ground truth, belonging to \"Unqualified answers\" in our categorization. We will explain the underlined results in more detail in both the caption of Table 3 and the main text to make it easier to understand.\n\n## Reference\n\n[1] Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText Corpus.\n\n[2] How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering\n\n[3] How to Approach Ambiguous Queries in Conversational Search: A Survey of Techniques, Approaches, Tools, and Challenges\n\n[4] Self-Knowledge Guided Retrieval Augmentation for Large Language Models\n\n[5] When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories\n\n[6] REPOFORMER: Selective retrieval for repository-level code completion", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0QetG9DuXJ", "reviewer": "Reviewer_frNN", "summary": "This paper investigates the knowledge boundaries of Large Language Models (LLMs) using semi-open-ended questions, revealing limitations in LLMs' understanding. The authors introduce a novel method involving an auxiliary model to discover low-probability ambiguous answers, constructing a dataset of 953 questions across 32 domains. Their findings show that GPT-4 frequently produces unqualified answers and struggles with accurate self-evaluation, highlighting the need for improved detection of LLM knowledge boundaries.", "questions": "1. Has the author analyzed the performance of other large language models except GPT-4 on semi-open datasets?\n2. Can you provide a more detailed analysis of the types of errors made by GPT-4 on semi-open-ended questions?\n3. How do you ensure the reliability and consistency of human annotations in verifying ambiguous answers?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work introduces a novel approach to evaluate LLMs using semi-open-ended questions, revealing limitations in existing methods focused on close-ended questions. The proposed method of identifying low-probability ambiguous answers is innovative and well-motivated.\n2. Experiments show significant improvements in understanding LLM knowledge boundaries, particularly where models like GPT-4 may hallucinate, enhancing evaluation reliability.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The evaluation scope of this paper is limited, as it only assesses the performance of GPT-4 Turbo, which may be insufficient to represent the overall performance of LLMs. Have the authors analyzed other mainstream large language models such as Claude or LLaMA? It would be better to provide the performance of other large language models on this dataset.\n\n2. The practical implications of the findings are not fully demonstrated. It is recommended to provide specific examples of applications to illustrate how these findings can improve the reliability of LLMs in practical use. Further explanation is needed on how improving training data or algorithms can mitigate these issues and enhance the practical utility of the models.\n\n3. The paper mentions that GPT-4 performs poorly on semi-open-ended questions, but it may not have thoroughly analyzed the different types of errors (such as factual errors, logical errors, etc.) and their causes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable questions! We have incorporated all the suggested experiments and the results meet the expectation. We are confident that we have addressed all of your concerns as outlined below. Based on our new experimental findings and explanations, we appreciate it if you would reconsider the final evaluation of our work.\n\n## Q1: It is better to evaluate more LLMs.\n\nFollowing your advice, we analyze the performance of the Claude model (claude-3-sonnet-20240229) on semi-open-ended questions and find it also performs poorly and generates many unsatisfying answers.\n\n|        | Unqualifie Answer%(↓) | Inaccurate Evaluation %(↓) | Hallucinated Question %(↓) |\n| ------ | --------------------- | -------------------------- | -------------------------- |\n| GPT-4  | 40.15                 | 28.47                      | 82.90                      |\n| Claude | 37.15                | 33.73                      | 88.00                    |\n\nThe above table shows the evaluation results of two powerful LLMs. Results show that similar to GPT-4, Claude also performs poorly on the semi-open-ended questions.\n\nDue to the time limit, we randomly sample 100 questions from our dataset and evaluate the performance of Claude. As the most advanced LLMs perform poorly on this task, we need human annotation to fact-check each generated answer, which is time-consuming. Specifically, it takes 2 minutes for an annotator to assess and double-check the truthfulness of each answer. Our dataset contains 1k questions, each of which corresponds to 13 tail-answers on average for verification. Even if we hire more than 5 annotators, it takes us 72 hours (more than 7 days if they work 10 hours per day).\n\n## Q2: Provide more application examples and practical implications of the findings.\n\nPerceiving LLMs' knowledge boundaries is important to understand and alleviate hallucination [1,2]. Ambiguous answers for semi-open-ended questions are highly likely beyond the knowledge boundaries of LLMs (see Sec 4.4). Discovering ambiguous answers benefits many applications, including: \n\n1. It helps detect the knowledge scope of LLMs more faithfully. Many close-ended hallucination evaluation benchmarks face the danger of data contamination [3, 4]. Semi-open-ended questions are easy to design and correspond to a large number of undocumented answers;\n2. Flagging ambiguous answers with higher uncertainty enhances the LLM outputs [5, 6];\n3. Identifying ambiguous answers helps achieve selective retrieval that augments LLM with indispensable external information while reducing the distraction of irrelevant data [7, 8, 9].\n4. It helps align LLMs for a more honest generation by teaching the LLM to admit its knowledge limit on the knowledge it is unfamiliar with (ambiguous answers) [10, 11, 12].\n\nWe also find that in real life, semi-open-ended questions are quite common, indicating that the potential impact of our work is quite large. To estimate the proportion of semi-open-ended questions, we randomly sample 1k questions from an open-source corpus and conduct statistics to find that 33.6% of questions are semi-open-ended.\n\n## Q3: Detailed analysis of types of errors\n\nFollowing your suggestion, we categorize different types of errors made by our target LLM and analyze their causes according to evaluation results.\n\n| Error Types | Factual Inconsistency | Factual Fabrication |\n| ----------- | --------------------- | ------------------- |\n| Ratio       | 91.45%                | 8.55%               |\n\nFollowing [1], we distinguish different types of hallucinations: \n\n1. Factual inconsistency takes up 91.45% of errors for semi-open-ended questions. It happens when the answer can be grounded in real-world information, but mismatches certain requirements in the question.\n2. Factual fabrication leads to 8.55% errors, which occurs when the answer is unverifiable from public sources.\n\nBesides, we found that 86.15% evaluated answers met some parts of the requirement in the question while failing to satisfy the rest requirements. This may be because some conditions in the question overshadow others, leading to unqualified answers [13].\n\nOur focus is the detection of LLM's knowledge limitations. Logical errors are usually observed in reasoning tasks, which is not the primary focus of this work. We will study the logical errors for QA tasks in the future.\n\n## Q4: How to ensure the reliability and consistency of human annotations?\n\nWe hired 11 annotators with Master's degrees, and provided clear evaluation guidelines (see Line 563) and feedback to human annotators during the evaluation process, thereby ensuring the reliability of the annotations across the entire dataset. Finally, we cross-check the annotation results.\n\n## Reference\n\n[1] A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions\n\n[2] Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method\n\n[3] An Open Source Data Contamination Report for Large Language Models\n\n[4] Investigating Data Contamination in Modern Benchmarks for Large Language Models\n\n[5] How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering\n\n[6] How to Approach Ambiguous Queries in Conversational Search: A Survey of Techniques, Approaches, Tools, and Challenges\n\n[7] Self-Knowledge Guided Retrieval Augmentation for Large Language Models\n\n[8] When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories\n\n[9] REPOFORMER: Selective retrieval for repository-level code completion\n\n[10] Alignment for Honesty\n\n[11] Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism\n\n[12] Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback\n\n[13] Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of identifying "knowledge boundaries" on semi open-ended question answering settings. This problem has been studied for factual queries where the model either knows or doesn't know the answer and classic calibration methods can be used. However, in the more open-ended setting, a model may know some answers and be unaware of others.  They construct a dataset of these cases by sampling from the LLM. They then augment the LLM's answers to these questions with an auxiliary model.  This process uses an open-source LLM to generate additional low-probability candidates by subtracting off a matrix computed via a pseudoinverse computed from existing answers. They check the correctness of these answers using RAG.  Finally, they verify whether these ambiguous answers are within or beyond the LLM's knowledge boundary: does the LLM generate correct answers, and does it successfully identify via self-evaluation which answers are correct or not?

Results show that the authors' approach discovers more ambiguous answers than baselines. Moreover, GPT-4 in general does so-so at the task of recognizing its knowledge boundary: it frequently generates bad answers, is not able to check them, and does not often stop from generating bad answers when continuously prompted.

Strengths

- Interesting and timely problem

- Generating the additional answers by subtracting off a pseudoinverse is a nice method for this

- The overall experimental design is well-conceived and the results for GPT-4 are interesting

- Most reviewers' concerns are addressed

Weaknesses

- Applicability of the setting: I like the open-endedness, but I agree with bcni and hUqe that this set of questions is a bit narrow.  It would be nice if there were a clearer picture here of how this could broaden to other settings.

- The use of human annotation in 4.1 is confusing and not explained in depth.

- Focus on LLM self-eval: this is a bit narrow and doesn't look at other possible techniques like those based on probabilistic calibration.  It's not clear if this gives a precise sense of where a model's knowledge boundary is.
Decision: Accept (poster)