Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models (2024)

Abstract: As the use of Large Language Models (LLMs) becomes more widespread, understanding their self-evaluation of confidence in generated responses becomes increasingly important as it is integral to the reliability of the output of these models. We introduce the concept of Confidence-Probability Alignment, that connects an LLM’s internal confidence, quantified by token probabilities, to the confidence conveyed in the model’s response when explicitly asked about its certainty. Using various datasets and prompting techniques that encourage model introspection, we probe the alignment between models’ internal and expressed confidence. These techniques encompass using structured evaluation scales to rate confidence, including answer options when prompting, and eliciting the model’s confidence level for outputs it does not recognize as its own. Notably, among the models analyzed, OpenAI’s GPT4 showed the strongest confidence-probability alignment, with an average Spearman’s ρ̂ of 0.42, across a wide range of tasks. Our work contributes to the ongoing efforts to facilitate risk assessment in the application of LLMs and to further our understanding of model trustworthiness.1

heading: 1 Introduction
text: In recent years, we have witnessed the rapid development and deployment of Large Language Models (LLMs) across various disciplines. LLMs such as GPT (Brown et al., 2020; Schulman et al., 2022; OpenAI, 2023), PaLM (Chowdhery et al., 2022), Chincilla (Hoffmann et al., 2022), and LLaMa (Touvron et al., 2023), have showcased remarkable performance across a diverse range of NLP tasks, and their capabilities in empowering chatbots have ignited a surge of interest among the general public.
1The code to reproduce our experimental results as well as detailed interactions with the language models are available at https://github.com/akkeshav/confidence_probablitiy_alignment.
With the ongoing integration of these models into high-stakes areas such as healthcare, law, and education, a critical evaluation of their behavior and trustworthiness is becoming increasingly essential.
Many contemporary prompting techniques, such as Self-Consistency (Wang et al., 2023b), Tree of Thoughts (Yao et al., 2023), and Multi-Agent Debating (Du et al., 2023), rely heavily on a model’s self-evaluation of its reasoning process. Recent works like (Diao et al., 2023) use LLMs’ selfconfidence for tasks like question selection based on uncertainty. However, if a misalignment exists between the model’s expressed self-reasoning and its true internal confidence, these techniques could yield misleading results, undermining their practical utility. Trust and decision-making by users of
315
these models in real-world applications are directly impacted by understanding this alignment.
This is further complicated by LLM hallucination (Huang et al., 2021; Shuster et al., 2021; Ji et al., 2023), where LLMs produce outputs that seem plausible but are factually incorrect or fabricated. These hallucinations often come with high expressions of confidence, challenging non-expert users to distinguish them from reliable outputs. Notable examples of these are reported in (Alkaissi and McFarlane, 2023), where ChatGPT confidently cited non-existent references for specific medical claims.
Amid these challenges, our work aims to enhance trustworthiness in LLMs through a detailed understanding of model confidence. We define verbalized certainty as an LLM’s explicit expression of its confidence level in response to a question and investigate its relationship with the model’s actual internal confidence, quantified by token probabilities. Our exploration of the correlation between these two measures, termed ConfidenceProbability Alignment, posits that this alignment is crucial for the reliability of a model’s output.
Drawing upon a range of question datasets, our investigation explores the nuances of internal and verbalized confidence across an extensive array of
models, including OpenAI’s GPT-3, its variants such as InstructGPT and the RLHF version, and the more recent GPT-4, in addition to open-source models, namely Microsoft’s Phi-2-2.7B and HuggingFace’s Zephyr-7B. We probe the models with questions from a variety of tasks, gather response token probabilities, and solicit confidence in the answers (visually summarized in Figure 1). We also explore model response variability under different parameters and prompts and examine the connection between a model’s confidence and its actual performance. Figure 2 provides a visual overview of how GPT-4 responds to different questions and its associated expressions of confidence. Our contributions are three-fold:
1. Conceptualization of Confidence-Probability Alignment: We introduce a novel framework to assess LLMs’ transparency and reliability. This measure evaluates the correlation between an LLM’s internal confidence and its verbalized certainty, offering a new lens to better understand model behavior.

heading: 2. Study Across Diverse LLM Architectures:
text: Our work covers a wide array of LLM architectures. Using prompting techniques to encourage model introspection, we reveal varied alignment
dynamics, notably GPT-4’s consistent confidenceprobability alignment across multiple tasks.
3. Analysis of Confidence-Correctness Relationship: We further analyze the link between models’ expressed confidence and response accuracy, exploring the impact of model-specific parameters such as temperature. Additionally, we develop a taxonomy of misalignments for detailed error analysis, uncovering various cases of alignment discrepancies and their implications for model reliability.

heading: 2 Confidence-Probability Alignment
text: Confidence-Probability Alignment refers to the correlation between a model’s verbalized certainty and its internal confidence, each quantified in specific ways. We define verbalized certainty as a model’s explicit expression (i.e., in the generated text) of its confidence level regarding its own answer to a given question. The internal confidence, on the other hand, is quantified using answer token probabilities. The alignment of these two aspects can be determined using the following procedures.

heading: 2.1 Response Generation
text: The generation of a response from the language model commences with a structured prompt, comprising a question denoted as Q, and a set of answer options, hereafter denoted as Oset. Each individual answer option within Oset will be referred to as Oi. The answer options in Oset are potential responses that the model can select to answer Q, with this structure designed to elicit a definitive response.
For instance, consider a question Q posed as “Which of the following is a common element in the atmosphere?" The corresponding set of answer options Oset comprises five distinct responses labeled from ‘A’ to ‘E’: A. Oxygen, B. Nitrogen, C. Gold, D. Iron, and E. Helium.
The question and its options are concatenated in a structured format, each answer option prefixed with a corresponding label, and separated by newline characters to maintain a clear distinction. The prompt culminates with the term ‘Answer:’, designed to solicit a model’s response.
The prompt is constructed as follows:
Prompt = Q + "\n" + Option A + "\n" + ... + Option E + "\nAnswer: "
Upon presenting this structured prompt to a language model, the model generates a response text. We subsequently extract the chosen answer ai from
the output response text. This chosen answer, ai ∈ Oset, symbolizes the model’s selected response from the options Oset, serving as the basis for the ensuing evaluation of the model’s internal confidence and verbalized certainty.

heading: 2.2 Internal Confidence
text: In the context of LLMs, internal confidence for a chosen output is quantified as the probability assigned to the selected output token, Tchoice. This probability can be derived by applying the exponential function to the log probabilities or logits of each output token, depending on the model’s output format. A higher score for Tchoice signifies greater model confidence in that choice. However, exact confidence assessment can be challenging due to potential token ambiguities, such as differing case sensitivity (’B’ vs. ’b’) for answer tokens.2
To address this, we introduce the concept of adjusted answer token probability. This involves calculating token probabilities and adjusting them to account for potential ambiguities and variations in token format, which we detail in Algorithm 1.
Our algorithm starts by converting log probabilities (or, alternatively, logits) to standard probabilities for all tokens. Then, for each answer option, it identifies all corresponding tokens, calculating the highest probability among them to represent the option’s most confident token representation. Finally, the algorithm normalizes the highest probability of the selected answer option against the sum of probabilities for all options, yielding an adjusted internal confidence measure, PIC .
GPT-Family Models: For GPT-family models, which return log probabilities (logP (Ti)), token probabilities are derived using:
P (Ti) = exp(logP (Ti)). (1)
Open-Source Models: Many open-source models provide logits (L(Ti)) for output tokens. To convert these logits into probabilities, we employ the softmax function:
P (Ti) = eL(Ti)
∑ j e L(Tj) . (2)

heading: 2.3 Verbalized Certainty
text: We define verbalized certainty as the language model’s explicit expression of its confidence level
2Although accounted for, we observed this phenomenon to occur in less than three percent of instances across datasets.
Algorithm 1 Procedure for computing adjusted token probabilities, PA and internal confidence, PIC Require: Model output tokens T and their corresponding log
probabilities logP (T ). 1: Compute token probabilities, P (Ti), for all tokens Ti in
T using P (Ti) = exp(logP (Ti)). 2: for each Option O in the list of possible options do 3: Determine the set of tokens, TO , corresponding to option O. 4: Compute the maximum probability among tokens in
TO to obtain PO = max i∈TO P (Ti).
5: Append PO to the set of adjusted probabilities PA. 6: end for 7: Select maximum probability, PM from PA 8: PS ← 0. 9: for each probability PI in PA do
10: PS ← PS + PI 11: end for 12: Compute internal confidence PIC , by dividing PM by
PS . 13: return PIC
in its own answer to a given question. To extract this measure, we construct a Confidence Querying Prompt (CQP) that asks the model to analyze its certainty in the context of the question, the available options, and its chosen answer. The CQP is presented as follows:

heading: A language model was asked: {question} Options were: {choices_text}.
text: The model's answer was: {response_text}.
Analyse its answer given other options. How certain are you of the model's answer?
a. Very Certain b. Fairly Certain c. Moderately Certain d. Somewhat Certain e. Not Certain f. Very Uncertain
This query effectively asks the model to take on an observational role and to analyze the certainty of its answer. The model’s response gives us its verbalized certainty. Below, we outline the key components of the CQP.
Simulation of Third-Person Perspective (TPP): The CQP initiates with a third-person perspective (TPP) to mitigate potential self-preferential biases (Wang et al., 2023a; Zheng et al., 2023). It presents the question and answer as if they may not have necessarily been generated by itself. In this scenario, the model’s ‘confidence’ reflects its probabilistic estimation of the response’s correctness,
based on its training. It’s important to clarify that this does not equate to personal confidence or selfawareness, but rather an approximation of answer accuracy. Ultimately, the TPP helps us elicit a measure of confidence that is less susceptible to potential biases, which have been noted in the mentioned studies, arising from the language model’s own generation process.
Option Contextualization (OC): The Option Contextualization (OC) aspect of the CQP equips the model with a framework for gauging its chosen response by stating, ‘Options were: {choices_text}’. This accomplishes two goals:
Comparative Evaluation: By displaying all options, the model can contextualize its chosen response, allowing for more informed confidence judgments compared to isolated evaluations.
Answer Verification: Providing all potential answers facilitates comprehensive evaluations and enables the model to adjust its confidence if the selected answer is sub-optimal in comparison to the other options.
Likert Scale Utilization (LSU): The Likert Scale Utilization (LSU) phase of the CQP employs a qualitative scale ranging from ‘very certain’ to ‘very uncertain’. The choice of a qualitative scale instead of a numerical one aims to maintain a consistent understanding across different model instances. In the context of LLMs, the interpretation of a numerical certainty scale (e.g., ‘rate your certainty from 1 to 10’) can vary significantly between different model instances due to the lack of concrete experiential or emotional grounding that humans use to interpret these numbers. A qualitative scale leverages the model’s training on a vast corpus of human language, which makes the gradations (e.g., ‘fairly certain’ is more confident than ‘moderately certain’) generally more universally understood and consistent.
Responses to the CQP are then mapped to a numerical score using a predefined system: ‘very certain’ equals 1.0, ‘fairly certain’ equals 0.8, ‘moderately certain’ equals 0.6, ‘somewhat certain’ equals 0.4, and ‘not certain’ or ‘very uncertain’ correspond to 0.2 and 0, respectively. This scoring method provides a quantifiable measure of the model’s verbalized certainty. Importantly, this measure reflects the model’s self-evaluated certainty and is distinct from its internal confidence as quantified by the adjusted token probabilities.
The finalized CQP design emerged after explor-
ing various alternative designs during preliminary experiments, as detailed in Appendix Table A.3.

heading: 2.4 Alignment Evaluation
text: We define alignment as the correlation between internal and verbalized confidence metrics, evaluated using Spearman’s rank correlation coefficient.
Spearman’s Rank Correlation Coefficient: The Spearman’s rank correlation coefficient is a non-parametric test that measures the degree of association between two variables. Unlike the Pearson correlation coefficient, which requires the assumption of normally distributed variables, Spearman’s correlation does not require this assumption and can handle ordinal, interval, and ratio data. This makes it ideal for comparing the non-normally distributed token probabilities and verbalized confidence values in our study.
Given two variables X and Y , the Spearman correlation ρ is computed as follows:
ρ = 1− 6 ∑
d2i n(n2 − 1) , (3)
where di is the difference between the ranks of corresponding values of X and Y , and n is the number of observations.

heading: 3 Experiments

heading: 3.1 Tasks & Datasets
text: We select a diverse set of tasks that allow us to assess an LLM’s confidence-probability alignment across various question types and complexities.
• CommonsenseQA (CSQA) (Talmor et al., 2019): CSQA poses multiple-choice questions that require a grasp of common-sense knowledge to answer effectively.
• Question Answering via Sentence Composition (QASC) (Khot et al., 2019): QASC necessitates models to infer answers by composing information sourced from multiple sentences, thereby testing their ability to form connections.
• RiddleSense (Lin et al., 2021): The unique dataset of RiddleSense tests models on riddles, challenging them to use a blend of world knowledge and lateral thinking.
• OpenBookQA (Mihaylov et al., 2018): OpenBookQA tests models on science-based multiple choice questions, gauging their understanding of factual scientific information.
• AI2 Reasoning Challenge (ARC) (Clark et al., 2018): The ARC challenges models with complex, knowledge-intensive questions that necessitate deep reasoning and the integration of multiple information sources, far beyond simple question answering.

heading: 3.2 Models
text: We used the following LLMs: GPT-3 (text-davinci001) (Brown et al., 2020); InstructGPT-3 (textdavinci-002) (Ouyang et al., 2022); InstructGPT3 + RLHF (text-davinci-003) (Ouyang et al., 2022); GPT-4 (gpt-4-0613; (OpenAI, 2023)); Microsoft’s Phi-2-2.7B3 and HuggingFace’s Zephyr7B (Tunstall et al., 2023).4

heading: 3.3 Experimental Design
text: We prompted models with the complete set of all dataset questions, obtained the responses, and extracted answer token probabilities for internal confidence estimation (following Algorithm 1). We then prompt the models (using CQP) to verbally express their confidence to compute alignment with their internal confidence. For a complete visual walkthrough of the procedure through an example, please refer to Appendix Figure A.1.5

heading: 4 Results
text: Alignment Evaluation Table 1 presents the dataset-wise alignment evaluation using Spearman’s rank correlation coefficient. Here, GPT-4 consistently outperforms its counterparts across every dataset. Specifically, it registered the highest coefficient on the QASC datasets of nearly 0.5, indicative of a moderate correlation. In contrast, OpenbookQA and ARC datasets marked the lowest correlations, although they still remained higher than other model versions with values of 0.41 and 0.35, respectively.
The standout performance of GPT-4 highlights potential advancements in model architecture and training methodologies, including the scale and possibly more refined human feedback integration.
3https://ai.azure.com/explore/models/microsoft-phi2/version/4/registry/azureml-msr
4Selected for their balance between computational efficiency and answer quality, these open-source models offer an ideal compromise—being sufficiently compact for algorithmic tractability while robust enough to deliver meaningful responses for question answering and verbalized certainty evaluations.
5To conduct all experiments, approximately 132.5 compute hours were elapsed. For experiments with GPT models, their public OpenAI API was used; https://openai.com/api/.
Temperature Stability In Figure 3, we analyze the impact of the temperature parameter on the variability of verbalized certainty across different datasets for GPT-4. Our findings indicate a direct trend between increased temperature and heightened variability in certainty scores. In particular, OpenBookQA and Riddle Sense, both of which emphasize multi-step reasoning and deep topic understanding, exhibit the sharpest sensitivity to temperature variations. This could be attributed to the multi-faceted nature of their questions, making the model’s certainty more volatile under increased randomness. Conversely, QASC, focusing on sentence composition within grade school science, demonstrates stability in certainty across diverse temperature settings. Such observations highlight the imperative of dataset-specific temperature calibration, given the inherent complexities and requirements of individual datasets.
Consistency of RLHF variants RLHF vari-
ants of the GPT family, including GPT-3’s RLHF version and GPT-4, have demonstrated consistent confidence-probability alignment, with GPT-4 showing a significant improvement. Our current working hypothesis suggests that this consistency may be related to, and possibly correlated with, other desired attributes of RLHF, such as the helpfulness, sensitivity, and creativity of responses.
Dynamics of Internal Confidence and Verbalized Certainty Table 2 details the alignments between internal confidence and verbalized certainty. Here are the alignment types, supplemented by observations:
• Consistent Alignment: The metrics are in harmony, indicating clear model comprehension. For instance, a query about the habitat of white mice yields high confidence in both metrics, signifying a straightforward understanding.
• Internal Overconfidence: The model exhibits high internal confidence despite low verbalized certainty. An example involves a question about historical events, where the model’s internal assurance doesn’t match its cautious verbal certainty assessment.
• External Overconfidence: Contrary to internal overconfidence, here, the model’s verbalized certainty surpasses its internal confidence. A question about common social practices may elicit an overly confident verbal assurance, possibly reflecting overlearned societal norms rather than a deep understanding.
• Consistent Discordance: Both confidence metrics are low, indicating model uncertainty. Questions that explore abstract concepts or complex reasoning, such as the philosophical implications of a theoretical scenario, can
lead to this alignment, reflecting the model’s awareness of its limitations.
Further examples in addition to the models’ explanations for their responses can be found in Appendix Tables A.1 and A.2. Examples comparing the alignment between all models on example instances are also provided in Appendix Table A.5.
Dismal performance of open-source models: Table 3 demonstrates that both Microsoft’s Phi and Zephyr (Tunstall et al., 2023) fail to generate appropriate verbalized certainty responses. This issue may arise from the difficulty smaller models face in needing to be proficient at both generating and evaluating simultaneously. If either of these processes is compromised (as often observed with Zephyr/Phi), it results in low confidence alignment.

heading: 5 Analysis and Discussion

heading: 5.1 Correctness and Confidence
text: In Figures 5 and 6, five alignment matrices display an analysis of GPT-4’s verbalized certainty and internal confidence, respectively, in relation to correctness across five datasets. Each matrix categorizes responses based on the degree of certainty and correctness, with the green zones representing accurate alignment: ‘very certain’ for correct responses, ‘fairly certain’ for incorrect ones. Due to
the continuous nature of internal confidence, we differentiate ‘very certain’ from other levels by using the median internal confidence value. Here, ‘very certain’ represents values above the median, while ‘fairly certain’ represents values below the median. Assessment of verbalized certainty and accuracy for InstructGPT-3 + RLHF is provided in Appendix Figure A.4.
The data from the matrices demonstrate a clear correlation between verbalized certainty and accuracy for GPT-4 across all datasets, including CSQA, QASC, ARC, OpenbookQA, and RiddleSense. A similar trend is also observed between internal confidence and accuracy for GPT-4 across all datasets. This indicates that higher confidence or certainty levels consistently correspond with correct responses. This finding is particularly noteworthy as it offers a counterpoint to recent work by (Meister et al., 2022), which demonstrated that high probability does not always coincide with high quality in LLMs.
This analysis, focusing primarily on the ‘very certain’ and ‘fairly certain’ categories, is informed by the predominant high levels of certainty (verbalized and internal) in GPT-4 responses across datasets. Figures 7 and 8 show the distribution of verbalized and internal confidence levels, highlighting distinct patterns and variability across contexts.
Further details are provided in the Appendix, where Figures A.2 and A.3 illustrate the variation in verbalized certainty and internal confidence distributions across model types. To see the distribution of verbalized certainty and internal confidence for InstructGPT-3 + RLHF, refer to Appendix Figures A.5 and A.6 respectively.

heading: 5.2 Component Ablation Analysis
text: In Table 4 we analyzed the individual and combined impacts of three proposed prompting techniques: Simulation of Third-Person Perspective (TTP), Option Contextualization (OC), and Likert Scale Utilization (LSU). The highest alignment was observed when combining all three techniques. While LSU consistently improved performance across datasets, TTP was less influential alone, and OC’s efficacy varied by dataset. The interaction of TTP, LSU and OC was found to be most effective.

heading: 6 Related Work
text: Estimation of Confidence in LLMs Numerous studies have explored estimating confidence in LLMs. Some emphasize model sensitivity to input changes (Vasudevan et al., 2019) or employ hints in Neural Machine Translation for confidence (Lu et al., 2022). Others use prompt engineering for verbalized probabilities (Lin et al., 2022), fine-tune models on question-answer accuracy probabilities (Mielke et al., 2022), or modify prompts for uncertainty expressions (Zhou et al., 2023). Additional research quantifies uncertainty using metrics such as semantic entropy (Kuhn et al., 2023; Meister et al., 2022) or assesses overconfidence with atypical inputs (Yuksekgonul et al., 2023). With many LLMs being proprietary, some have probed confidence using both transparent and opaque methods (Lin et al., 2023). Our work explores a different facet of confidence in both open-source and proprietary LLMs, focusing on the alignment between
verbalized certainty and token probabilities. Self-Evaluation in Prompt-Based Techniques Prompt-based techniques have been a major focus in the research community, with a particular emphasis on the accuracy of LLMs’ self-assessments for effective functionality. Newer prompting techniques, such as Tree of Thoughts (ToT) (Yao et al., 2023) and Self-Consistency (SC) (Wang et al., 2023b), have embraced this emphasis, utilizing LLMs’ self-evaluation to enhance their effectiveness. This represents a departure from previous approaches like Chain-of-Thought prompting (Wei et al., 2023), which did not involve the use of language models as evaluators. In our work, we too leverage the self-evaluation paradigm, but extend this by posing multiple-choice questions and providing the original answer options in the prompt, followed by a Likert scale as new answer options.
Language Model Communication and Collaboration Recent research has also seen the development of techniques that involve multiple LLMs communicating or collaborating to perform tasks such as planning and information extraction (Zhuge et al., 2023), or generating text-to-image prompts (Xu et al., 2023). In some cases, outputs from one LLM have been used to fine-tune another in a ‘selfimprovement scenario (Huang et al., 2022). Our work differs from these as it primarily focuses on a single LLM’s self-assessment of its confidence, highlighting the importance of understanding the internal coherence of individual models. Our work leverages the strengths of these existing techniques, but expands upon them to explore the alignment between LLMs’ expressed confidence and their internal token probabilities.

heading: 7 Conclusion
text: We introduced the concept of ConfidenceProbability Alignment to critically assess the transparency and reliability of LLMs. Our findings show GPT-4 exhibiting moderate alignment, high-
lighting both advancements and the pressing need for further improvements. The next steps involve formulating strategies to enhance this alignment, establishing a concrete metric for gauging model trustworthiness. Innovating in this direction is crucial for advancing the development of LLMs that are both dependable and open.
Limitations
Accessibility to Token-Level Probabilities: Our work is bound by the confines of models for which we have access to token-level log probs/logits. This access is limited to specific models like GPT-3, and its Reinforcement Learning from Human Feedback (RLHF) variant, and GPT-4 while excluding others such as PaLM 2 and Chinchilla. Consequently, our findings are constrained to this subset of models, potentially impacting the broad applicability of our results. There is a pressing need for future research to explore the Confidence-Probability Alignment across a wider spectrum of models as their internals become available for scrutiny.
Language-Specific Limitations: Our study predominantly focuses on the English language, a language with relatively limited morphology. While we’ve incorporated diverse datasets to analyze Confidence-Probability Alignment, the intricacies and subtleties of languages with richer syntactic complexity could lead to different outcomes. As such, our results may not seamlessly extend to LLMs designed for languages with more complex morphological structures. This underlines the necessity for further research to understand Confidence-Probability Alignment in LLMs developed for a wide range of languages.
Meta-Level Reasoning: Our study design inherently requires the ability to query the model about its own confidence. This introduces a meta-level of reasoning, which may not always be in line with the model’s ‘base’ level reasoning, engaged during its
primary task. The model uses its own underlying architecture to introspect and express its confidence, which could potentially introduce complex biases in the verbalized confidence.
Reliance on Prompting Techniques: Our investigation, while offering promising findings concerning the correlation between a model’s verbalized and internal confidence, relies heavily on carefully constructed prompting techniques. While our findings provide valuable insights, it’s important to note that they are largely dependent on precise prompting. As such, variations in the prompt formulation can affect the effectiveness of these findings in different contexts. This constraint highlights the need for developing models that can demonstrate Confidence-Probability Alignment without a significant dependency on the art of prompting.
Model Confidence and Prompt Accuracy: In the context of this study, our primary objective is not to optimize the accuracy of the model’s responses, but rather to explore the nuances in the relationship between a model’s internal and verbalized confidence. While enhancing model accuracy is undeniably important, it lies outside the primary focus of this study. For future work, the potential exists for implementing feedback loops and adjustments based on a model’s accuracy, although such explorations extend beyond the scope of our current work.
Ethical Implications: While enhancing LLM transparency, our exploration of confidenceprobability alignment also reveals ethical challenges. Misaligned confidence can spread misinformation, and understanding model confidence could be exploited maliciously. Despite our focus on responsible LLM use, users must critically evaluate model outputs, emphasizing the need for stringent ethical guidelines and safeguards against potential misuse.

heading: Acknowledgements
text: This work was supported by the Natural Sciences and Engineering Research Council of Canada and by the New Frontiers in Research Fund.

heading: A Appendix

heading: Future Work
text: There is a pressing need for future research to explore the Confidence-Probability Alignment across a wider spectrum of models as their internals become available for scrutiny. Language-Specific Limitations: Our study predominantly focuses on the English language, a language with relatively limited morphology. While we’ve incorporated diverse datasets to analyze Confidence-Probability Alignment, the intricacies and subtleties of languages with richer syntactic complexity could lead to different outcomes. As such, our results may not seamlessly extend to LLMs designed for languages with more complex morphological structures. This underlines the necessity for further research to understand Confidence-Probability Alignment in LLMs developed for a wide range of languages. Meta-Level Reasoning: Our study design inherently requires the ability to query the model about its own confidence. This introduces a meta-level of reasoning, which may not always be in line with the model’s ‘base’ level reasoning, engaged during its
primary task. The model uses its own underlying architecture to introspect and express its confidence, which could potentially introduce complex biases in the verbalized confidence. Reliance on Prompting Techniques: Our investigation, while offering promising findings concerning the correlation between a model’s verbalized and internal confidence, relies heavily on carefully constructed prompting techniques. While our findings provide valuable insights, it’s important to note that they are largely dependent on precise prompting. As such, variations in the prompt formulation can affect the effectiveness of these findings in different contexts. This constraint highlights the need for developing models that can demonstrate Confidence-Probability Alignment without a significant dependency on the art of prompting. Model Confidence and Prompt Accuracy: In the context of this study, our primary objective is not to optimize the accuracy of the model’s responses, but rather to explore the nuances in the relationship between a model’s internal and verbalized confidence. While enhancing model accuracy is undeniably important, it lies outside the primary focus of this study. For future work, the potential exists for implementing feedback loops and adjustments based on a model’s accuracy, although such explorations extend beyond the scope of our current work. Ethical Implications: While enhancing LLM transparency, our exploration of confidenceprobability alignment also reveals ethical challenges. Misaligned confidence can spread misinformation, and understanding model confidence could be exploited maliciously. Despite our focus on responsible LLM use, users must critically evaluate model outputs, emphasizing the need for stringent ethical guidelines and safeguards against potential misuse