Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models (2024)

Abstract: This paper investigates the capabilities of Large Language Models (LLMs) in the context of understanding their knowledge and uncertainty over questions. Specifically, we focus on addressing known-unknown questions, characterized by high uncertainty due to the absence of definitive answers. To facilitate our study, we collect a new dataset with KnownUnknown Questions (KUQ) and establish a categorization framework to clarify the origins of uncertainty in such queries. Subsequently, we examine the performance of open-source LLMs, fine-tuned using this dataset, in distinguishing between known and unknown queries within open-ended question-answering scenarios. The fine-tuned models demonstrated a significant improvement, achieving a considerable increase in F1-score relative to their prefine-tuning state. Through a comprehensive analysis, we reveal insights into the models’ improved uncertainty articulation and their consequent efficacy in multi-agent debates. These findings help us understand how LLMs can be trained to identify and express uncertainty, improving our knowledge of how they understand and express complex or unclear information.

heading: 1 Introduction
text: “To know what you know and to know what you do not know, that is true knowledge
— The Analects of Confucius”
Large Language Models (LLMs) have grown in size and capabilities (Wei et al., 2022) (Chen et al., 2021). Consequently, different works raise the question of what the models learn and know (Jiang et al., 2020) and how they can express uncertainty (Lin et al., 2022) (Zhou et al., 2023).
Source code and data available on: https://github.com/amayuelas/knowledge-of-knowledge
We look at cognitive psychology, where metacognition (Garner and Alexander, 1989) is defined as the awareness and thoughts of one’s own thought process. Do LLMs know what they know? And more importantly, are they aware of what they do not know? This is an important question to calibrate the certainty of their statements or prevent such language models from confidently generating false answers, commonly known as hallucinations.
Given the division of knowledge in Table 1, we pay special attention to Known-Unknowns. These are questions that do not have a definitive answer. The answers to such questions are often subjective and may even be unanswerable due to a lack of information or inherent complexity. As a result, the answers are considered to have high uncertainty levels. For example, If the Universe started at the Big Bang, what existed before then?. Our goal is to understand how language models deal with these uncertain questions.
We identify several reasons why questions may be unknown. In some cases, questions that ask about future events inherently lack definitive answers due to the uncertainty of future outcomes. In other cases, some questions address unresolved issues in science or history, for which conclusive answers are currently unavailable. Additionally,
6416
questions may contain incorrect assumptions, making them unanswerable as they are based on false premises. A comprehensive categorization of such questions is presented in Table 3.
The topic of known-unknown questions has barely been studied in the area of Large Language Models. SelfAware (Yin et al., 2023) introduces the topic, but our work carries a detailed analysis of open-source models, how they can be finetuned, and a categorization of the questions with explanations. Below, we present the main research questions and contributions from our work: • Can open-source models differentiate between known and unknown questions?. We introduce a dataset of Known-Unknown Questions (KUQ) and evaluate it on the open-source LLama family of models. We show how the LLama models fall behind in this task when compared to GPT-3.5 and GPT-4, and introduce a fine-tuning strategy that brings them on par with these models. • How does fine-tuning improve the ability of open-source models to differentiate between known and unknowns? Fine-tuning proves to be a good strategy for adding new abilities to opensource models. Identifying the question of uncertainty is one of them. However, we see a trade-off between this ability and correctly answering known questions. We also show the generalization ability of these models on the self-aware dataset. • Can a fine-tuned model on our KUQ dataset improve the results of a downstream task? Understanding and expressing uncertainty has many potential applications. We show our fine-tuned model on KUQ can enhance the results of multiagent debate on some reasoning tasks.

heading: 2 Related Work
text: Language Models Knowledge. Since the beginning of the first pre-trained language models, some researchers have studied what information is stored in their weights and how we can extract that knowledge (Jiang et al., 2020), along with how confident the models are about their knowledge (Jiang et al., 2021). More recently, another work has explored whether LLMs can evaluate the validity of their claims (Kadavath et al., 2022). The question of what a model should know has also been explored in computer vision systems (Sharifi Noorian et al., 2022), with a human-in-the-loop process to investigate what the models really know and what they should know. These works look at the model
knowledge, but we want to take a step further by providing questions that are uncertain by themselves. This question has initially been explored in SelfAware (Yin et al., 2023).
Language Models Uncertainty. (Hu et al., 2023) Modeling uncertainty has been a persistent challenge for the linguistics community. Uncertainty can be divided into epistemic uncertainty, which refers to the model uncertainty, and aleatoric uncertainty, which belongs to the data’s inherent randomness. Cole et al. (2023) introduces the notion of denotional uncertainty for the uncertainty contained in the meaning of the question. Denotional uncertainty is the area addressed in this work. Expressing uncertainty requires knowledge about one’s own knowledge and the ability to define the level of confidence in one’s response. Several studies have examined various approaches to expressing uncertainty (Kuhn et al., 2023; Szarvas et al., 2012; Farkas et al., 2010). In the area of Large Language Models, some works have quantified uncertainty (Xiao et al., 2022), and explored how they behave when expressing uncertainty (Zhou et al., 2023) and how they can learn to express uncertainty in words (Lin et al., 2022).

heading: 3 Data
text: In the evaluation domain, the significance of datasets containing known-unknown questions is paramount. Known-unknown questions are those that do not have definitive answers, such as "Are there other forms of intelligent life in the universe?" or "Which year will the next financial crisis occur?". Upon surveying available resources, we identified a mere 46 known-unknown questions in the Big-Bench benchmark (Srivastava et al., 2022) and 1 evaluation dataset in SelfAware (Yin et al., 2023). This limited quantity is insufficient for robust training + evaluation, prompting us to generate additional samples for a full-scale dataset. Our dataset is the first of its kind to include annotations on the questions pointing to the reason for uncertainty. The dataset is publicly available.
We have collected questions from crowd-source workers and generated a new dataset: KnownUnknown Questions, KUQ. We have depicted the data collection process in Figure 1 and explained question generation in §3.1. Table 2 shows the number of questions generated per source. Hereafter, we refer to known-unknowns as unknown questions and known-knowns as known questions.
In addition, we have identified several classes of unknown questions, shown in Table 3. These classes serve as a guideline on the source of uncertainty for each question. This dataset is more comprehensive than previous ones as it includes a larger set of categories and questions. For example, in SQuAD2 (Rajpurkar et al., 2018), they adversarially generate unanswerable questions. This differs from the general KUQ unknown questions, which are hard to answer because we cannot provide a correct answer, instead of the question being wrong by itself. We can argue similar cases for ControversialQA (Wang et al., 2023), which focuses on questions where there are controversial answers to the questions. Or AmbigQA (Min et al., 2020), which focuses on questions that need clarification.

heading: 3.1 Data Generation
text: Unknown Questions. The unknown questions were carefully generated by crowd-sourced workers, an inherently difficult process. To ensure quality, we explained the concept of known-unknown questions to the workers and provided them with a category from Table 3 along with examples. They were asked to generate a question and detail why its answer remains unknown. The tasks given to the workers are documented in Appendix B.
Known Questions. The set of known questions has been selected to match the unknown questions. We have selected a set of well-known datasets with Question-Answer pairs: SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), and HotPotQA (Yang et al., 2018). From the pool of these 3 datasets together, known questions have been selected to match each of the unknown questions with SimCSE (Gao et al., 2021), a contrastive-learning framework used to find similar sentences.
Categories. Initially, we collected a diverse range of uncategorized questions from the web and via crowd-sourcing, which our team then analyzed and categorized based on identified common features. Some of these categories matched those in existing research, while others were new. In the final stage, we shared these categories with our crowd-sourced contributors, ensuring a clear understanding of the task and helping us collect a well-balanced set of questions across all categories.

heading: 4 Methodology

heading: 4.1 Tasks
text: We study 3 different tasks in this paper which are analyzed in the Experiments Section §5.
1. Known vs Unknown. The underlying idea of our work is the ability of open-source models to differentiate known and unknown questions in an open-ended question-answering scenario. Given the question, can the language model answer the question or express the question uncertainty otherwise? We also examine the ability of the models to differentiate the question categories introduced in Table 3.
2. Effects of fine-tuning on KUQ. We perform an analysis of the trade-offs of using fine-tuning to gain the skill to differentiate between known and unknown questions.
3. Downstream Application: Multiagent Debate. The fine-tuned models on KUQ can be useful to improve downstream applications. In particular, we look into Multiagent Debate (Du et al., 2023), where different versions of the language model discuss and compare their answers and thought processes over several rounds. Through this back-andforth conversation, they work together to agree on a final answer on different knowledge and reasoning datasets.

heading: 4.2 Fine-tuning
text: In this work, we employed the KUQ dataset to finetune several open-source Large Language Models from the Llama-2 family. Our objective was to enhance their capabilities in expressing uncertainty when confronted with questions of an unknown or uncertain nature. The process is described in Figure 2
The dataset was specifically tailored for the finetuning process, incorporating either direct answers
for known questions or expressions of uncertainty (see §5.1) and category explanations (appendix G) as provided by crowd-sourced workers. The format used was: ###Question:... ###Answer: {Answer} / The question may be unknown because....
We adopted LoRA (Hu et al., 2021) within the Peft framework from Huggingface, as this is a less resource-intensive fine-tuning approach. We conducted fine-tuning on the Llama-2 7B and 13B models, as well as their respective RLHF Chat versions (Touvron et al., 2023), utilizing the Nvidia Titan RTX (24GB) and the RTX A6000 (48GB) graphics cards.

heading: 4.3 Evaluation
text: The experiments conducted are centered around the open-ended Question-Answering scenario. In this setup, models are presented with questions and are expected to generate their answers. Our default approach is direct question-answering, unless specified otherwise. A key aim of our evaluation is to discern whether the text generated by the language models expresses uncertainty when responding to unknown questions. Additionally, we assess if these models can accurately provide the correct category after undergoing fine-tuning. This approach is based on methodologies established in previous work from SelfAware (Yin et al., 2023).
We define a similarity function, fsim as a binary metric between the generated text (ti) and some reference text (refi) to be 1 if they express the same content, or 0 if they do not. If the reference text is contained in the generated text or the similarity measured with SimCSE (Gao et al., 2021) is higher than a threshold, τ , the function returns 1.
Simi = fsim(texti, refi) = {0, 1} (1)
For differentiating between known and unknown questions, the reference texts are a predefined set of phrases that encompass general uncertainty. The full list can be found in Appendix C. For example, in the following generations, we would expect: It is difficult to predict the future, the reason why . . . -> Contains Uncertainty Expression 19th-century architecture in the United States is characterized by -> Does not Contain Uncertainty Expression
As this is an automated metric, we also conduct a human evaluation to validate it. The results can be found in Appendix D.
In our evaluations, we utilize two key metrics: the F1 score and the Equal Error Rate (EER). The F1-score, derived from the similarity metric, is calculated with the positive class being either unknown questions or the chosen category. Concurrently, we measure the EER to assess the balance between false acceptance and false rejection rates, offering a holistic view of the system’s performance. Additionally, we evaluate answer accuracy on Known Questions, where a response is deemed correct if it includes the ground truth answer. This metric helps us understand potential regressions in known questions that may arise due to fine-tuning.

heading: 5 Experiments
text: In this section, we dive into a series of experiments centered around "known-unknown" questions. Our main aim is to see how well current Large Language Models (LLMs) handle these highly uncertain queries. We discuss the experiments and their results in more detail in the following sections.

heading: 5.1 Known vs Unknown
text: We want to determine the ability of Large Language Models to distinguish between Known and Unknown Questions. Furthermore, we aim to reproduce the closest to a real-world scenario as possible, where the model may be prompted with questions and needs to provide the answer, with a varying level of uncertainty.
We present the results on the KUQ evaluation set before fine-tuning for GPT-3.5, GPT-4, and the Llama Models. We also present the results after fine-tuning for Llama-7B, Llama-13B, and its derived chat versions.
Figure 3 presents the ROC curve (receiver operating characteristic curve), showing the performance of the classification at different classification thresholds. From the plot, we extract the EER and the corresponding F1-score. These results are presented in Table 4. From these results, we can observe there are 2 trends: (1) Bigger models tend to obtain better out-of-the-box results. (2) Chat versions have a better performance in differentiating between known and unknown questions. This may
be due to some existing similarity between their RLHF data and this experiment.
In order to test the fine-tuned models in an out-ofdomain distribution, we also show the results after fine-tuning on the KUQ dataset and evaluating the SelfAware (Yin et al., 2023) dataset. The results are shown in Figure 4 and Table 5. We observe fine-tuning on KUQ improves the general ability to differentiate between known and unknown questions. In general, we can see a similar behavior of the models to evaluation on the KUQ dataset.

heading: 5.2 Effects on Fine-tuning
text: In this section, we analyze the impact of fine-tuning on model performance, with a specific emphasis on determining the minimal dataset size required for effectively learning to distinguish between known and unknown questions.
We specifically investigate the number of training samples necessary for models to discern between known and unknown questions. To this end, we have conducted experiments using the Llama7b and Llama-13b models, training and evaluating them on datasets ranging from 32 to 1024 questions.
Figure 5 presents the F1-scores for the fine-tuned models in distinguishing between known and unknown questions. Our findings indicate that basic models show improved performance at approximately 256 samples, whereas chat-oriented models require around 512 samples. We hypothesize that this difference may stem from the chat models’ pre-existing training in question-answering tasks, which could lead to inherently better initial performance but also necessitate more data for significant behavioral adjustments during question-answering.
Additionally, our analysis reveals a trade-off inherent in fine-tuning with the KUQ dataset. While it enhances the model’s capability to express uncertainty when confronted with unknown questions, it also results in a slight decrease in overall accuracy in answering the known questions correctly. This effect can be seen in Figure 6, where the accuracy to known questions drops at the same time as the models gain the ability to differentiate knownunknown questions in their responses. Overall, the accuracy of known questions drops slightly in the fine-tuned models, as shown in Appendix F.

heading: 5.3 Downstream Application: MultiAgent Debate
text: In (Du et al., 2023), they have demonstrated an approach where multiple language model instances propose and debate their responses and reasoning processes over multiple rounds to arrive at a common final answer. Their findings show how this debate improves the results over a single model on several reasoning, factuality, and question-answering tasks.
A highlighted insight from multiagent debate is the fact that models may converge to a final answer, even when the answer is not correct. Despite arriving at the same answer, models can confidently affirm that their answers are correct. This phenomenon could potentially be attributed to a limitation inherent in the models’ design: the inability to accurately represent uncertainty within their response generation process.
With the hypothesis that expressing uncertainty can help downstream applications, we want to show that the fine-tuned models on KUQ can potentially better understand question uncertainty and express it accordingly, which leads to a performance increase in a multiagent debate.
Experiment We follow the same approach as the one presented in the original Multiagent Debate paper (Du et al., 2023), where a model is first asked to generate the answer to the proposed questions. In the next turn, the model is presented with the response from the previous turn and asked to improve it. The experiments have been carried out on the default original settings. This means we select the number of agents = 3 and the number of rounds = 2, which provides a good trade-off between results and speed to generate the results. The prompts used in this experiment can be found in Appendix H.
Data For this experiment, we have evaluated the following datasets to test the LLM abilities to reason and provide complex knowledge: • MMLU (Hendrycks et al., 2020), a dataset to measure the factuality of language models in answering questions typically found in different exams. • CommonSenseQA (CSQA) (Talmor et al., 2018), a question-answering dataset for commonsense questions based on the knowledge from ConceptNet (Speer et al., 2017)
• AI2 Reasoning Challenge (ARC) (Clark et al., 2018), a question-answering dataset containing natural, grade-school science questions that require knowledge and reasoning abilities.
• BIG-Bench Chess State Tracking (Srivastava et al., 2022), a dataset that contains an unfinished sequence of moves from a chess game paired with a set of valid moves to complete the sequence.
Evaluation We evaluate our approach to four distinct reasoning tasks, which involve reasoning and extracting factual knowledge from the models. These tasks are presented through three datasets (MMLU, CSQA, ARC), each comprising multiplechoice Question-Answer pairs. In these datasets, only one answer option is correct, and we assess the model’s accuracy in choosing this option. Additionally, we analyze the Chess-State Tracking benchmark, where a set of given chess moves is deemed correct, and we measure the model’s accuracy in producing any appropriate move from that given list. This research follows the methodology of the original paper, focusing on a subset of the entire dataset. We conducted several experimental runs using different seeds to ensure a representative sample. Our comparison involves the fine-tuned model LLama2-7B-Chat and its predecessor, the original Llama2-7B-Chat.
Results Figure 7 illustrates the performance outcomes of our experiment. It is evident that the finetuned model utilizing the KUQ dataset attains a performance enhancement in the multiagent debates when compared to the baseline model (Llama2-7BChat). Despite this advancement, a human analysis of the responses denotes a deficiency in the models’ ability to consistently represent uncertainty. Further research is necessary to accurately represent the models’ inherent uncertainty, in addition to the explicit uncertainty derived from the questions.
Analysis The results from the experiments show an improved accuracy of the fine-tuned models in the benchmark datasets. In order to validate the hypothesis that uncertainty expression helped increase the accuracy, we measure the uncertainty expressions generated in the responses. To do so, we compare the generated texts to the list of uncertainty expressions from Appendix C and report the average similarity according to SimCSE (Gao et al., 2021). We also report the average number of responses with a high similarity – higher than 0.75 –. The analysis for all 4 datasets is presented in Table 6. In all cases, the fine-tuned models express more uncertainty over the questions. This may be due to the fact that some questions can be interpreted ambiguously, as seen in an example debate response in Appendix I.

heading: 6 Conclusion
text: This work explores how open-source LLMs handle Known-Unknown questions, which are characterized by high uncertainty and the expectation of non-confident answers. We introduce a new dataset with Known and Unknown Questions, KUQ. In addition, a categorization of unknown questions is introduced, offering different reasons for them being unknown.
Along this work, we evaluate the current opensource models in open-ended question-answering on our KUQ dataset. We evaluate (1) the ability to tell the difference between known and unknown questions and (2) the ability to distinguish between the different categories of questions.
Finally, we show how the expression of uncertainty may help in specific applications of Large Language Models. In particular, we show how the fine-tuned model on KUQ improves the results of multiagent debate when compared to the baseline original model.
Future research directions should focus on enhancing evaluation techniques and tackling the challenge of gauging model epistemic uncertainty, potentially leading to broader applications. Investigating the capacity of LLMs to convey their uncertainty probabilities is also a key area of interest.
Limitations
This paper acknowledges several limitations encountered during its research process.
Initially, the task of generating known-unknown questions presents inherent complexity. While these questions have been validated as knownunknowns through human assessment, there remains a possibility of contention regarding their categorization. The compilation of known questions datasets was curated to include a broad set of questions and topics.
Moreover, the methodology for evaluating openended question-answering tasks continues to be a subject of ongoing discourse and investigation within the academic community. In this study, we employed a similarity metric to measure uncertainty expressions, acknowledging that the chosen similarity threshold is a variable factor influencing the results. This approach is consistent with methodologies employed in prior research.
Lastly, the decision to fine-tune and evaluate the Llama 2 models was influenced by their significance and popularity in the open-source community at the time of this study. The choice of smaller models (7B, 13B) over larger variants (70B) was dictated by the computational resources available during the research period.
Ethics Statement
Human evaluation was conducted through crowdsourcing using the Amazon Mechanical Turk platform. To ensure the quality of our experiments, we only considered workers with a HIT approval rating of at least 95% from the Mechanical Turk Masters pool. We compensated the workers at a rate of $0.25 per task. We estimate each task can be completed in 1 minute or less and therefore it translates to a rate of $15.0 per hour, which exceeds the federal minimum wage in the USA during the time of our research. The data annotation is classified as an approved exempt protocol from the IRB. Details about the tasks, including screenshots and task descriptions of each Mechanical Turk study are included in the appendices sections.

heading: Acknowledgements
text: This work was supported by the National Science Foundation award #2048122. The views expressed are those of the authors and do not reflect the official policy or position of the US government.

heading: A Dataset Statistics
text: This appendix presents, in Figure 8, the First-Word Distribution for each of the sources employed in our Known-Unknown Questions (KUQ) dataset.

heading: B Crowd-Source Question Generation and Classification
text: In this appendix, we present the tasks from the crow-source workers to generate the KUQ dataset. First, in Figure 9, we show the task to generate the known-unknown questions. In Figure 10, we show how the workers were explained the different categories and asked to provide 1 category. Their results were confirmed with a majority vote from 3 different workers.

heading: C List of Uncertainty Expressions
text: We present the list of uncertainty expressions used in our evaluations with SimCSE in Table 7. Since there are multiple valid positions in the chess move validity experiment, we have additional phrases to see if the model can recognize the existence of multiple correct answers for that experiment only.

heading: D Human Evaluation on Similarity-based Metric
text: The evaluation metric to detect uncertainty expressions, described in §4.3, uses an automated similaritymetric. It is based on the similarity of the generated text to a list of uncertainty expressions. We conduct a human evaluation to validate this use of this method.
We provided a sample of questions – 200 samples/model – and their generated answers to human evaluators. And we asked them to identify expressions of uncertainty contained in the generated answer. Then, we computed the agreement percentage between the crowd-source workers and our similarity-based metric. The agreement is computed as the percentage where both evaluations agree on the outcome: uncertain/not uncertain.
We found that the agreement rate between the crowd-sourced evaluations and our metric was 0.90 (± 0.03) for the Llama 7B-Chat model and 0.96 (± 0.02) for its fine-tuned counterpart, at a similarity threshold of 0.75 – Table 12 –. Interestingly, the agreement between the two models becomes closer together as the threshold increases, as shown in Figure 11.
As we expected, the fine-tuned model showed a slightly higher agreement rate, since it was specifically trained to identify certain expressions of uncertainty included in the list. Nonetheless, the agreement levels between the human evaluators and our metric were remarkably high for both models, indicating the effectiveness of our approach.
E Instruction Prompting
In the previous experiments, all answers have been generated through direct prompting, which is closer to a real-world scenario. In this section, we observe what happens when the models are instructed with a specific request to provide the answer or generate an ’unknown’ phrase. The prompt is provided in Appendix H.
Table 8 shows the result of this analysis. Here, we observe that (1) models without fine-tuning achieve better results with instruct-prompt than on the zero-shot setting. And (2) models specifically fine-tuned on this prompt modality are on par with the models trained for direct question-answering.

heading: F Effects on Answer Accuracy
text: In this section, we look at the answer accuracy of known questions. We want to investigate how the models can provide the correct answer to the known questions in KUQ.
We observe a trade-off in the fine-tuning process on the KUQ dataset. This is represented in Table 9, where the fine-tuned models have a small accuracy drop for the known questions. This may be due to an over-expression of uncertainty for known questions.

heading: G Effects between categories
text: We analyze the differences between fine-tuning on all categories at the same time versus training an independent model for each of the given categories. The results of this analysis are shown in Figure 13. In this plot, we compare the performance of the model trained on the whole KUQ dataset (All Categories) with the performance of a model trained on each of the categories individually.
We would expect the model trained on each of the categories to perform better on the specific category it has been trained on because it should not get confused with other categories. However, this is not the case for counterfactual questions. Counterfactual questions are the easiest to recognize at first sight as they are constructed with specific expressions such as: what if... , What would ... if .... However, in this case, we find the general model achieves better results. The causes are yet to be further analyzed.

heading: H Prompts
text: For the instruction fine-tuning experiments from Appendix E, we present the prompt used below. For the application to multiagent debate from Section 5.3, we provide the starting prompt and intermediate debate prompts in Table 10.
Instruct Prompt:
Read the following question carefully and answer it. Think before answering. If the question is unknown or highly uncertain, you may answer: ’It is unknown’. ### Question: {question} ### Answer:
Multiagent Debate Prompts:

heading: I Examples of Uncertainty Improving Debate Responses
text: In this section, we present an example of the fine-tuned model having better accuracy and demonstrating uncertainty. Table 11 shows the generated responses from the fine-tuned model, while Table 12 shows the generated responses from the base Llama model. The correct answer is (D) Transporting, but only the fine-tuned model is able to get both agents to reach this conclusion.

heading: Future Work
text: Future research directions should focus on enhancing evaluation techniques and tackling the challenge of gauging model epistemic uncertainty, potentially leading to broader applications. Investigating the capacity of LLMs to convey their uncertainty probabilities is also a key area of interest. Limitations
This paper acknowledges several limitations encountered during its research process. Initially, the task of generating known-unknown questions presents inherent complexity. While these questions have been validated as knownunknowns through human assessment, there remains a possibility of contention regarding their categorization. The compilation of known questions datasets was curated to include a broad set of questions and topics. Moreover, the methodology for evaluating openended question-answering tasks continues to be a subject of ongoing discourse and investigation within the academic community. In this study, we employed a similarity metric to measure uncertainty expressions, acknowledging that the chosen similarity threshold is a variable factor influencing the results. This approach is consistent with methodologies employed in prior research. Lastly, the decision to fine-tune and evaluate the Llama 2 models was influenced by their significance and popularity in the open-source community at the time of this study. The choice of smaller models (7B, 13B) over larger variants (70B) was dictated by the computational resources available during the research period