Do Large Language Models Know What They Don’t Know?
Year: 2023

Abstract:
Large language models (LLMs) have a wealth of knowledge that allows them to excel in various Natural Language Processing (NLP) tasks. Current research focuses on enhancing their performance within their existing knowledge. Despite their vast knowledge, LLMs are still limited by the amount of information they can accommodate and comprehend. Therefore, the ability to understand their own limitations on the unknows, referred to as self-knowledge, is of paramount importance. This study aims to evaluate LLMs’ self-knowledge by assessing their ability to identify unanswerable or unknowable questions. We introduce an automated methodology to detect uncertainty in the responses of these models, providing a novel measure of their self-knowledge. We further introduce a unique dataset, SelfAware, consisting of unanswerable questions from five diverse categories and their answerable counterparts. Our extensive analysis, involving 20 LLMs including GPT-3, InstructGPT, and LLaMA, discovering an intrinsic capacity for self-knowledge within these models. Moreover, we demonstrate that in-context learning and instruction tuning can further enhance this self-knowledge. Despite this promising insight, our findings also highlight a considerable gap between the capabilities of these models and human proficiency in recognizing the limits of their knowledge. “True wisdom is knowing what you don’t know.” –Confucius

heading: 1 Introduction
text: Recently, Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023), PaLM 2 (Anil et al., 2023), and LLaMA (Touvron et al., 2023) have shown exceptional performance on a wide range of NLP tasks, including common sense reasoning (Wei et al., 2022; Zhou et al., 2022) and mathe-
∗ Corresponding author.
matical problem-solving (Lewkowycz et al., 2022; Chen et al., 2022). Despite their ability to learn from huge amounts of data, LLMs still have limitations in their capacity to retain and understand information. To ensure responsible usage, it is crucial for LLMs to have the capability of recognizing their limitations and conveying uncertainty when responding to unanswerable or unknowable questions. This acknowledgment of limitations, also known as “knowing what you don’t know,” is a crucial aspect in determining their practical applicability. In this work, we refer to this ability as model self-knowledge.
The Know-Unknow quadrant in Figure 1 illustrates the relationship between the model’s knowledge and comprehension. The ratio of “Known Knows” to “Unknown Knows” demonstrates the model’s proficiency in understanding and applying existing knowledge. Techniques such as Chain-of-Thought (Wei et al., 2022), SelfConsistency (Wang et al., 2022), and Complex CoT (Fu et al., 2022) can be utilized to increase
8653
this ratio, resulting in improved performance on NLP tasks. We focus on the ratio of “Known Unknows” to “Unknown Unknows”, which indicates the model’s self-knowledge level, specifically understanding its own limitations and deficiencies in the unknows.
Existing datasets such as SQuAD2.0 (Rajpurkar et al., 2018) and NewsQA (Trischler et al., 2017), widely used in question answering (QA), have been utilized to test the self-knowledge of models with unanswerable questions. However, these questions are context-specific and could become answerable when supplemented with additional information. Srivastava et al. (2022) attempted to address this by evaluating LLMs’ competence in delineating their knowledge boundaries, employing a set of 23 pairs of answerable and unanswerable multiple-choice questions. They discovered that these models’ performance barely surpassed that of random guessing. Kadavath et al. (2022) suggested probing the selfknowledge of LLMs through the implementation of a distinct "Value Head". Yet, this approach may encounter difficulties when applied across varied domains or tasks due to task-specific training. Consequently, we redirect our focus to the inherent abilities of LLMs, and pose the pivotal question: “Do large language models know what they don’t know?”.
In this study, we investigate the self-knowledge of LLMs using a novel approach. By gathering reference sentences with uncertain meanings, we can determine whether the model’s responses reflect uncertainty using a text similarity algorithm. We quantified the model’s self-knowledge using the F1 score. To address the small and idiosyncratic limitations of existing datasets, we created a new dataset called SelfAware. This dataset comprises 1,032 unanswerable questions, which are distributed across five distinct categories, along with an additional 2,337 questions that are classified as answerable. Experimental results on GPT-3, InstructGPT, LLaMA, and other LLMs demonstrate that in-context learning and instruction tuning can effectively enhance the self-knowledge of LLMs. However, the self-knowledge exhibited by the current state-of-the-art model, GPT-4, measures at 75.47%, signifying a notable disparity when contrasted with human self-knowledge, which is rated at 84.93%.
Our key contributions to this field are summarized as follows:
• We have developed a new dataset, SelfAware, that comprises a diverse range of commonly posed unanswerable questions.
• We propose an innovative evaluation technique based on text similarity to quantify the degree of uncertainty inherent in model outputs.
• Through our detailed analysis of 20 LLMs, benchmarked against human self-knowledge, we identified a significant disparity between the most advanced LLMs and humans 1.

heading: 2 Dataset Construction
text: To conduct a more comprehensive evaluation of the model’s self-knowledge, we constructed a dataset that includes a larger number and more diverse types of unanswerable questions than KnowUnknowns dataset (Srivastava et al., 2022). To facilitate this, we collected a corpus of 2,858 unanswerable questions, sourced from online platforms like Quora and HowStuffWorks. These questions were meticulously evaluated by three seasoned annotation analysts, each operating independently. The analysts were permitted to leverage external resources, such as search engines. To ensure the validity of our dataset, we retained only the questions that all three analysts concurred were unanswerable. This rigorous process yielded a finalized collection of 1,032 unanswerable questions.
In pursuit of a comprehensive evaluation, we opted for answerable questions drawn from three datasets: SQuAD (Rajpurkar et al., 2016), HotpotQA (Yang et al., 2018), and TriviaQA (Joshi et al., 2017). Our selection was guided by SimCSE (Gao et al., 2021), which allowed us to identify and select the answerable questions semantically closest to the unanswerable ones. From these sources, we accordingly drew samples of 1,487, 182, and 668 questions respectively, amassing a total of 2,337. Given that these questions can be effectively addressed using information available on Wikipedia, the foundational corpus for the training of current LLMs, it is plausible to infer that the model possesses the requisite knowledge to generate accurate responses to these questions.
Our dataset, christened SelfAware, incorporates 1,032 unanswerable and 2,337 answerable questions. To reflect real-world distribution, our dataset
1The code pertinent to our study can be accessed https://github.com/yinzhangyue/SelfAware
contains a proportion of answerable questions that is twice as large as the volume of unanswerable ones. Nevertheless, to ensure the feasibility of testing, we have purposefully capped the number of answerable questions.

heading: 2.1 Dataset Analysis
text: To gain insight into the reasons precluding a certain answer, we undertook a manual analysis of 100 randomly selected unanswerable questions. As tabulated in Table 1, we have broadly segregated these questions into five distinctive categories. “No Scientific Consensus" encapsulates questions that ignite ongoing debates within the scientific community, such as those concerning the universe’s origin. “Imagination" includes questions involving speculative future scenarios, like envisaged events over the next 50 years. “Completely Subjective" comprises questions that are inherently personal, where answers depend heavily on individual predispositions. “Too Many Variables" pertains to mathematical problems that become unsolvable owing to the overwhelming prevalence of variables. Lastly, “Philosophical" represents questions of a profound, often metaphysical, nature that resist concrete answers. Ideally, upon encountering such questions, the model should express uncertainty instead of delivering conclusive responses.

heading: 3 Evaluation Method
text: This section elucidates the methodology employed for assessing self-knowledge in the generated text.
In order to achieve this, we define a similarity function, fsim, to compute the similarity, S, between a given sentence, t, and a collection of reference sentences, U = {u1, u2, . . . , un}, endowed with uncertain meanings.
Si = fsim(t, ui). (1) Whenever any Si surpasses a pre-determined threshold T , we perceive the text t as encompassing uncertain meanings, thereby eliminating the need for manual evaluation of the response.
Given the substantial disparity in the volume of answerable and unanswerable questions in SelfAware, we adopt the F1 score as a measure of LLMs’ self-knowledge. Our focus rests on identifying unanswerable questions, hence we designate them as positive cases and categorize answerable questions as negative cases.

heading: 4 Experiment

heading: 4.1 Model
text: We conduct a sequence of experiments to evaluate the degree of self-knowledge manifested by various LLMs, including GPT-3 (Brown et al., 2020) and InstructGPT (Ouyang et al., 2022) series, as well as the recent LLaMA (Touvron et al., 2023) and its derivative models, namely Alpaca (Taori et al., 2023) and Vicuna (Chiang et al., 2023). Our investigative approach employed three distinct input forms: Direct, Instruction, and In-Context Learning (ICL), which is encapsulated in Appendix A.4.

heading: 4.2 Setting
text: We devised the reference sentence set U through a process that combined automated generation by LLMs and manual filtering, detailed further in Appendix A.1. To quantify the similarity between target and reference sentences, we utilized SimCSE (Gao et al., 2021), setting the similarity threshold to 0.75 during our experiments. An exploration of threshold ablation is available in Appendix A.2. To counteract potential errors in similarity calculation induced by varying lengths of the target and reference sentences, we employed a sliding window of length 5 to parse the target sentence into semantic chunks. During the generation process, we set the temperature to 0.7. We selected a random sample of 100 instances for GPT-4, while the remainder of the models were scrutinized using the full SelfAware dataset.

heading: 4.3 Human Self-Knowledge
text: To establish a benchmark for human selfknowledge, we engaged two volunteers and selected 100 random samples from the SelfAware dataset. The volunteers has 30 minutes to make
judgments on the same set of questions, yielding an average F1 score of 84.93%, which we subsequently adopted as the benchmark for human self-knowledge. Detailed scores are available in Appendix A.3.

heading: 4.4 Analysis
text: We evaluate the manifestation of LLMs’ selfknowledge, centering our investigation on three fundamental dimensions: the size of the model, the impact of instruction tuning, and the influence exerted by different input forms.
Model Size. Figure 2 illustrates the correlation between model size and self-knowledge across various LLMs. It is noteworthy that across all three input forms, an augmentation in model parameter size is associated with an elevation in the F1 Score, with the most conspicuous enhancement manifesting in the ICL input form. Therefore, our analysis indicates that an LLM’s self-knowledge tends to enhance with increasing model size, a trend consistent with the scaling law.
Instruction Tuning. Figure 2 delineates that models from the InstructGPT series exhibit a superior level of self-knowledge compared to their GPT-3 counterparts. Further evidence of model enhancement is provided by Figure 4, where textdavinci models show significant improvement relative to the base davinci model. An additional comparative analysis, presented in Figure 5, evaluates LLaMA against its derivative models. The results underscore a notable increase in self-knowledge for Alpaca and Vicuna upon instruction tuning, exceeding their base model performances. Among these, Vicuna-13B outperforms the LLaMA-65B, corroborating the efficacy of instruction tuning for enhancing model self-knowledge.
Input Forms. As shown in Figure 2, the incorporation of instructions and examples serves to boost the self-knowledge of both the GPT-3 and InstructGPT series. Specifically, ICL input form, providing richer contextual information, contributes to a significant enhancement in models’ self-knowledge. This impact is particularly noticeable in the davinci model, where ICL facilitates a 27.96% improvement over the direct. Moreover, a comparison between Figure 3 and Figure 4 reveals that the inclusion of instructions and examples successfully minimizes the performance disparity between the davinci and text-davinci models, suggesting an acquisition of self-knowledge from the instructions and provided examples.
Compared with Human. Figure 3 reveals that, without supplementary samples, GPT-4 currently performs best among the tested models, achieving an impressive F1 score of 75.47%. However, a noticeable gap becomes evident when comparing this
performance to the human benchmark of 84.93%. This underscores the considerable potential that remains for enhancing the self-knowledge level of LLMs.
Answerable Questions. Figure 6 traces the performance evolution of the InstructGPT series in addressing answerable questions, adhering to the closed-book question answering paradigm (Touvron et al., 2023), where output accuracy is contingent on the presence of the correct answer. Our observations underscore a steady enhancement in QA task accuracy corresponding to an increase in model parameter size and continuous learning. Particularly, the accuracy of text-davinci-001 experiences a significant ascent, scaling from a meager 2.48% in text-ada-001 to 10.61%, whereas GPT-4 marks an even more striking jump to 42.64%.

heading: 5 Conclusion
text: This study investigates the self-knowledge of LLMs by evaluating their ability to identify unanswerable questions. Through the introduction of a novel dataset and an automated method for detecting uncertainty in the models’ responses, we are able to accurately measure the self-knowledge of LLMs such as GPT-3, InstructGPT and LLaMA. Our results reveal that while these models possess a certain degree of self-knowledge, there is still an apparent disparity in comparison to human selfknowledge. This highlights the need for further research in this area to enhance the ability of LLMs to understand their own limitations on the unknows. Such efforts will lead to more accurate and reliable responses from LLMs, which will have a positive impact on their applications in diverse fields.
Limitations
• Generalization of reference sentences. At present, we have selected sentences with uncertain meanings exclusively from the GPT-3 and InstructGPT series, potentially overlooking uncertainty present in responses generated by other LLMs. However, it is not feasible to catalog all sentences with uncertain meanings exhaustively. As a direction for future research, we propose to concentrate on the automated acquisition of more accurate reference sentences to address this concern.
• Limitations of input forms: Our examination was confined to three unique input forms: direct, instruction, and ICL. There is burgeoning research aimed at bridging the gap between models and human-like methods of reasoning and problem-solving, including but not limited to approaches like Reflexion (Shinn et al., 2023), ToT (Yao et al., 2023), MoT (Li and Qiu, 2023). Future endeavors will integrate additional cognitive and decision-making methods to delve deeper into the self-knowledge exhibited by these LLMs.
Ethics Statement
The SelfAware dataset, meticulously curated to evaluate LLMs’ ability to discern unanswerable questions, is composed of unanswerable questions extracted from sources such as Quora and HowStuffWorks, alongside answerable questions procured from three distinct open datasets. Every question was thoroughly examined for relevance and harmlessness. To ensure content validity, three annotation analysts, compensated at local wage standards, dedicated regular working hours to content review.
Throughout our research process, we underscored the significance of privacy, data security, and strict compliance with dataset licenses. In order to protect data integrity, we implemented anonymization and content filtration mechanisms. Our adherence to OpenAI’s stipulations remained unyielding for the usage of GPT-3 and InstructGPT models, and likewise for Meta’s terms pertaining to LLaMA models. We rigorously vetted the licenses of the three publicly available datasets for compliance, ensuring that all our research methodologies were in alignment with ethical standards at the institutional, national, and global levels.
Adhering to the CC-BY-SA-4.0 protocol, the dataset, once publicly released, will be reserved exclusively for research purposes. We pledge to promptly and effectively address any concerns relating to the dataset, while concurrently anticipating researchers to maintain high ethical standards in their utilization of this data.

heading: Acknowledgement
text: We wish to express our gratitude to our colleagues in the FudanNLP group whose insightful suggestions, perspectives, and thought-provoking discussions significantly contributed to this work. Our sincere appreciation also extends to the anonymous reviewers and area chairs, whose constructive feedback was instrumental in refining the quality of our study. This work was supported by the National Natural Science Foundation of China (No. 62236004 and No. 62022027) and CAAI-Huawei MindSpore Open Fund.

heading: A Appendix
text: A.1 Uncertainty Text To assemble a set of reference sentences, we randomly chose 100 entries from the SelfAware dataset. For each model in the GPT-3 and InstructGPT series, we conducted a preliminary test using the direct input form and manually curated sentences that displayed uncertainty. From this pre-test, we procured 16 sentences manifesting uncertain connotations to serve as our reference sentences. After normalizing these sentences by eliminating punctuation and converting to lowercase, we utilized them to compute similarity with target sentences throughout our experimental procedure.
1. The answer is unknown.
2. The answer is uncertain.
3. The answer is unclear.
4. There is no scientific evidence.
5. There is no definitive answer.
6. There is no right answer.
7. There is much debate.
8. There is no known case.
9. There is no concrete answer to this question.
10. There is no public information available.
11. It is impossible to know.
12. It is impossible to answer.
13. It is difficult to predict.
14. It is not known.
15. We do not know.
16. I’m not sure.
A.2 Threshold ablation We generated 100 new responses using the textdavinci-002 with direct input form and manually filtered out sentences that contained uncertainty. We then used SimCSE (Gao et al., 2021) to calculate the similarity between these sentences and the reference sentences in Appendix A.1. We tested various thresholds for filtering sentences with uncertain meanings and compared them to manually
annotated sentences. We considered unanswerable questions as positive examples and calculated precision, recall, and F1 score. The results in Table 2 indicate that a threshold of 0.75 produced the highest F1 score, balancing precision and the inclusion of other uncertain sentences. As a result, we selected 0.75 as the similarity threshold for subsequent experiments.
A.3 Human Self-Knowledge Test The evaluation results for the responses from our invited volunteers are presented in Table 3. The F1 scores for the responses were high, indicating that both volunteers exhibited a strong level of selfknowledge.
A.4 Template The input templates used in our experiments, Direct, Instruction, and ICL, are illustrated in Figures 7, 8, and 9, respectively. In the ICL template, we composed 3 answerable and 3 unanswerable questions and provided the corresponding answers manually.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We explore the possible flaws of our article in detail in the Limitation section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: We examine potential risks to data collection and privacy in our Ethics Statement.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: We succinctly outline the key points of the article in the abstract and the initial section introduction.
7 A4. Have you used AI writing assistants when working on this paper? We utilize Grammarly solely for spell checking and grammar correction during the composition process.
B 3 Did you use or create scientific artifacts? We strictly adhere to the terms of use set by OpenAI and Meta when using the GPT-3, InstructGPT, and LLaMA models. We also strictly adhere to the dataset licenses when using the SQuAD, HotpotQA, and TriviaQA datasets, as outlined in the Ethics Statement section.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We meticulously cite the appropriate literature regarding our use of the model and dataset in the Experiment section and the Dataset Construction section.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: We examine the terms and conditions in the Ethics Statement section and justify the appropriateness of our usage.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We outline the models employed and the alignment of their usage intentions in the Ethics Statement section. For the datasets we created, we clearly define the scope of use and conduct the study in strict accordance with the specified scope.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? In the Ethics Statement section, we state that we rigorously eliminate any data containing personally identifiable information, offensive content, and is completely anonymous during the data screening process.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? In the Ethics Statement section, we delve into the SelfAware dataset, which is primarily employed to assess a model’s self-knowledge. The dataset is in English and comprises both answerable and unanswerable questions.

heading: 3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader
text: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Section 2, Dataset Construction, we analyze the data distribution of the SelfAware dataset, including the number of answerable and non-answerable questions. All data is utilized solely for testing purposes.
C 3 Did you run computational experiments? In Section 4, Experiment, we conducted computational experiments.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? In Section 4.4, Result, we present the number of parameters of the model used.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In Section 4.2, Setting, we detail the hyperparameter temperature used in the experiment.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Considering the cost of experimentation, we did not conduct multiple experiments. However, replication of select experiments confirmed that there were no substantial variations in the outcomes, thus ensuring the reliability of our results.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We developed our own pre-processing and evaluation metrics, instead of utilizing existing packages.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? In Section 2, Dataset Construction, and Section 4, Experiment, we employ human annotators to assist us in sorting through the data and identify sentences with uncertain meanings.
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We verbally communicated our expectations to the annotators, clearly outlining their roles and responsibilities.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? In the Ethics Statement section, we recruited three annotators at rates that comply with local wage standards.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? In the Ethics Statement section, we clearly specify in the dataset usage regulations that it can only be used for scientific research.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: In the Ethics Statement section, we demonstrate our unwavering adherence to ethical and moral guidelines for data use.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We do not possess access to the personal information of the question creators during the data collection process.

heading: Future Work
text: As a direction for future research, we propose to concentrate on the automated acquisition of more accurate reference sentences to address this concern. • Limitations of input forms: Our examination was confined to three unique input forms: direct, instruction, and ICL. There is burgeoning research aimed at bridging the gap between models and human-like methods of reasoning and problem-solving, including but not limited to approaches like Reflexion (Shinn et al. , 2023), ToT (Yao et al. , 2023), MoT (Li and Qiu, 2023). Future endeavors will integrate additional cognitive and decision-making methods to delve deeper into the self-knowledge exhibited by these LLMs