Venue: NAACL 2025

# From Evidence to Belief:

A Bayesian Epistemology Approach to Language Models

Minsu Kim, Sangryul Kim, James Thorne

KAIST AI {minsu_kim, sangryul, thorne}@kaist.ac.kr

###### Abstract

This paper investigates the knowledge of language models from the perspective of Bayesian epistemology. We explore how language models adjust their confidence and responses when presented with evidence with varying levels of informativeness and reliability. To study these properties, we create a dataset with various types of evidence and analyze language models' responses and confidence using verbalized confidence, token probability, and sampling. We observed that language models do not consistently follow Bayesian epistemology: language models follow the Bayesian confirmation assumption well with true evidence but fail to adhere to other Bayesian assumptions when encountering different evidence types. Also, we demonstrated that language models can exhibit high confidence when given strong evidence, but this does not always guarantee high accuracy. Our analysis also reveals that language models are biased toward golden evidence and show varying performance depending on the degree of irrelevance, helping explain why they deviate from Bayesian assumptions.

10578

###### Contents

* 1 Introduction
* assessing whether the response of a model is correct Hendrycks et al. (2021); Srivastava et al. (2023)
- and _justification_, which encompasses explanation generation Wei et al. (2023); Camburu et al. (2018) and evidence finding Thorne et al. (2018).

In this work, we investigate whether the model believes its own responses (the _belief_ condition): specifically, the relationship between belief and the language model's justification, expressed as evidence. Since belief is a challenging concept to define, this paper focuses on belief from the perspective of Bayesian epistemology, which interprets belief as a quantitative and functional variable. According to Bayesian epistemology, the degree of belief can be interpreted and measured as probability, called _probability norm_. In particular, regarding the confirmation of belief, we should adjust the confidence of belief based on evidence. Specifically, when \(H\) represents the hypothesis (or belief), \(E\), the evidence for the belief, and \(\theta\) representing the background information or prior knowledge, we can define 3 primitive assumptions:1

Footnote 1: According to Hajek (2003); Vassend (2023), we treat conditional probability as a primitive concept representing the likelihood of an event occurring under certain conditions, rather than relying on the standard ratio formula \(P(H~{}|~{}E)=\frac{P(H(E))}{PE(E)}\). This approach allows us to apply the Bayesian assumption more intuitively, even in cases where \(P(E)=0\) or with contradictory evidence. For instance, when observing an unlikely event, such as a shark in freshwater lake, we still make judgments based on the observation despite it contradicting common knowledge. A detailed discussion can be found in Hajek (2003).

**Confirmation Assumption:**\(E\) confirms \(H\) if and only if \(P(H~{}|~{}E,\theta)>P(H~{}|~{}\theta)\)

**Disconfirmation Assumption:**\(E\) disconfilms \(H\) if and only if \(P(H~{}|~{}\theta)>P(H~{}|~{}E,\theta)\).

**Irrelevance Assumption:**\(E\) is irrelevant to \(H\) if and only if \(P(H~{}|~{}\theta)=P(H~{}|~{}E,\theta)\).



Also, if we define belief in terms of probability, the strength of the evidence should also be reflected in the confidence. That is,

* **Evidence Power Assumption:**\(E^{\prime}\) confirms \(H\) more strongly than \(E^{\prime\prime}\) if and only if \(P(H\mid E^{\prime},\theta)>P(H\mid E^{\prime\prime},\theta)\)[16, 20]. The degrees of belief should not only be a probability. The probabilities assigned to these beliefs must align with the _calibration norm_, meaning they should correspond to the actual likelihood of the event occurring, that is, the actual frequency [35].

The goal of this paper is to explore whether different types of evidence are reflected in language models' confidence and responses. The evidence here is not merely perturbations altering the correctness of information, i.e., informativeness, but our dataset also includes variations and modifications for reliability factors such as coincidence, timeliness, source of credibility, etc.

Our paper shows that language models can exhibit high confidence and accuracy when encountering true evidence but respond inconsistently with conflicting evidence and reduce confidence and accuracy with irrelevant evidence, contrary to Bayesian assumptions. We also found that LLMs are biased toward golden evidence (typical of annotated information that forms part of datasets) and perform differently across the gradient of irrelevance, supporting an understanding of why LLMs deviate from Bayesian epistemology.

## 2 Related Works

Calibration of LLMsCalibration of language models is a key metric for ensuring faithful responses, with log probabilities often representing model confidence [7, 48, 9]. As models have scaled, research has expanded to verbalized confidence, where models generate their own confidence [10, 11, 12]. While confidence can improve model performance [11, 12], some studies focus on interpreting this confidence as measuring uncertainty in semantic space [35] and exploring confidence through prompt and sampling methods [13].

Zhou et al. zhou23 explored how epistemic markers affect calibration. However, unlike their focus on linguistic markers, our work examines how changes in epistemic evidence, containing information on both content and reliability, influence confidence and calibration. Yu and Ji yu24 report that logical probability and language model probability can differ. This supports our findings that language models may not incorporate evidence into their responses and confidence.

Adversarial ContextWith in-context learning, studies have examined how few-shot demonstrations and explanations affect responses [15, 16]. Wang et al. wang23 showed even inaccurate demonstrations could be used in Chain-of-Thought (COT) prompting, while Chia et al. chia23 improved question accuracy with

Figure 1: The overall experimental method and simple examples of the evidence dataset for Confirmation Task. As golden evidence that aligns with the question is given to language models, it shows high confidence and accuracy. However, if language models encounter irrelevant evidence, it responds with low confidence. Further results and analysis are reported in Section 4.1.

 

contrastive demonstrations. Chen et al. (2023) studied how the number of demonstrations impacts accuracy. Feng et al. (2023) measures language models' responses and probability shifts in temporal relations based on subtle contextual changes. While these works focus on accuracy, we explore how direct question evidence influences not only accuracy but also confidence and calibration.

Turpin et al. (2023); Lanham et al. (2023) tested perturbations in COT inputs and their impact on answers, similar to our approach. While they focused on modifying explanations based on informativeness (e.g., incorrectness or relevance), our paper investigates whether LLMs reflect diverse evidence in their confidence and calibration, specifically exploring the effects of coincidental evidence and varying source credibility on model confidence.

## 3 Methods

We first generate various types of evidence by few-shot prompting large language models (LLMs) with questions and annotated support from SciQ Welbl et al. (2017), TriviaQA Joshi et al. (2017), GSM8K Cobbe et al. (2021). Refer to Appendix I.2 for details. We then used this evidence to evaluate how the models' confidence and responses change based on the type of evidence provided as shown in Figure 1. Influenced by Bayesian epistemology, we defined a confirmation task to measure whether language models can reflect the confirmation, disconfirmation, or irrerelevance assumption introduced in Section 1. Also, we created a strength-of-evidence task to assess LLM's ability to represent the various power of evidence. To measure the probability norm for adjusting confidence according to the evidence, we used an average confidence across all samples. In order to measure the response, such as correctness or calibration norm, we used accuracy (ACC) and Expected Calibration Error (ECE). In both the confirmation task and the strength-of-evidence task, we used zero-shot prompting for inference.

### Experimental Design

We estimated the confidence of language models using verbalized confidence Verb. 1S top-1 Tian et al. (2023), token probability, and sampling Lee et al. (2023); Xiong et al. (2024). Refer to Appendix H.2 and I.1 for details. Smaller-scale open-source LLMs did not tend to generate responses in the correct format matching the prompt of verbalized confidence. Also, following observations from Tian et al. (2023) that closed-source models are better at generating verbal confidence than open-source models, we used GPT-3.5-turbo-0125 and GPT-4o-2024-05-13 for inference. We used SciQ Welbl et al. (2017), TriviaQA Joshi et al. (2017) and GSM8K Cobbe et al. (2021) as the source datasets for our Confirmation task, and used only SciQ dataset for Strength of Evidence task, as a scientific question is suitable for making various degree of reliable evidence (see Appendix H for experimental details and dataset statistics).

### Confirmation Task

The objective of the confirmation task to observe and analyze the changes in the language model's confidence and responses when presented with various types of evidence, compared to scenarios where the language models receive the original evidence, \(E\), or in the absence of evidence, and assess how these changes align with three assumptions: Confirmation, Disconfirmation, and Irrelevance introduced section 1. Let the entire dataset be

\[D=\{ (Q_{i},A_{i},E_{i})~{}|~{}Q_{i}\text{ is a question, } \tag{1}\] \[A_{i}\text{ is an answer for $Q_{i}$},\] \[\text{and }E_{i}\text{ is evidence for $Q_{i}$ and $A_{i}$}\}.\]

and

\[E_{i}=(s_{i1},s_{i2},\ldots,s_{in}) \tag{2}\]

where \(s_{ij}\) is an evidence sentence in the collection of sentences \(E_{i}\) indexed by \(j=\{1,\dots,n\}\). For the experiment, we need to create modified \((Q_{i},A_{i},E^{\prime}_{i})\) where \(E^{\prime}_{i}\) is a perturbation of \(E_{i}\). The following are the types of \(E^{\prime}_{i}\):

1. **Conflicting Evidence** Conflicting evidence refers to abnormal information that hinders reaching the correct answer, introducing misinformation or conflicting beliefs with the golden evidence \(E_{i}\). Specifically, evidence where \(s_{ij}\) in \(E_{i}\) are replaced with their conflicting counterpart sentences \(\tilde{s}_{ij}\). Thus, \(E^{\prime}_{i}\) is conflicting evidence if and only if \[E^{\prime}_{i}=(\tilde{s}_{i1},\tilde{s}_{i2},\ldots,\tilde{s}_{in}),\forall s _{ij}\in E_{i}.\]
2. **Incomplete Evidence** Evidence that includes only a subset of sentences from the original evidence collection \(E_{i}\). Thus, \(E^{\prime}_{i}\) is a proper subset of \(E_{i}\). In our experiments, we discard approximately half the sentences from \(E_{i}\) (i.e. \(|E^{\prime}_{i}|\approx 0.5\times|E_{i}|\)) 

3. **Contradictory Evidence** The original evidence \(E_{i}\) is concatenated with additional negated sentences from \(E_{i}\). Thus, \(E^{\prime}_{i}\) is contradictory evidence if and only if \[E^{\prime}_{i}=E_{i}\cup N\quad\text{where}\quad N\subset\{\tilde{s}_{ij}\mid s_{ ij}\in E_{i}\}\] such that \(|N|=0.5\times|E_{i}|\). That is, adding 50% of the conflicting evidence to the original evidence.
4. **Irrelevant Evidence** Irrelevant evidence is \(E^{\prime}_{i}=E_{j}\) where \(j\neq i\). That is, \(E_{i}\) is randomly shuffled within the dataset \(D\) so that the evidence \(E_{i}\) of tuple \((Q_{i},A_{i},E_{i})\) is replaced with evidence \(E_{j}\) from a different tuple \((Q_{j},A_{j},E_{j})\).
5. **Coincidental Evidence** For the SciQ and TriviaQA dataset, unlike other types of evidence, coincidental evidence does not include incorrect answers but explanations reaching the golden answer by irrational reasoning or epistemic luck. Examples include explanations derived from guessing or vague memories. For GSM8K, coincidental evidence consists of a wrong reasoning process but a correct final answer.

We can see the concise examples of the evidence in Figure 1.

### Strength of Evidence

This task differs from the Confirmation task in that it focuses on the strength of evidence. Unlike the modified \(E^{\prime}\) used in the Confirmation task, the evidence used here includes the correct answer but perturbation of reliability. The goal is to understand how differences in the strength of evidence impact confidence and calibration and assess whether LLMs align with Evidence Power Assumption in section 1. For each \((Q_{i},A_{i})\) pair, two types of perturbation \((Q_{i},A_{i},E^{\prime}_{i})\) and \((Q_{i},A_{i},E^{\prime\prime}_{i})\) are created. \(E^{\prime}_{i}\) represents more reliable evidence, while \(E^{\prime\prime}_{i}\) represents relatively less reliable evidence. The following are the types of evidence:

1. **Source of Credibility** For each \((Q_{i},A_{i})\) pair, \(E^{\prime}_{i}\) means evidence from a highly reputable and authoritative source, while \(E^{\prime\prime}_{i}\) means evidence from an anonymous online post or an individual.
2. **Specificity and Detail** This involves varying the detail and specificity of the evidence. Similar to source of credibility, for each \((Q_{i},A_{i})\), \(E^{\prime}_{i}\) is highly detailed evidence, while \(E^{\prime\prime}_{i}\) is evidence with general mentions related to the question.
3. **Timeliness** This involves modifying the evidence based on its recency. For each \((Q_{i},A_{i})\), \(E^{\prime}_{i}\) consists of recent findings and experiments, while \(E^{\prime\prime}_{i}\) consists of relatively older findings and experiments.
4. **Experimental Evidence** For each \((Q_{i},A_{i})\), \(E^{\prime}_{i}\) includes evidence derived from precise and controlled experiments, while \(E^{\prime\prime}_{i}\) includes evidence where the answer is observed by a witness without experiments.

## 4 Results and Analysis

### LLMs on Confirmation task

The results of the Confirmation task using verbalized confidence are in Table 1, while the token probability and sampling methods are shown in Tables 2 and 3 in Appendix A. In Tables 4, 5 and 6 located in Appendix B, we calculated p-values to compare confidence, accuracy, and ECE between providing no evidence (labeled No_EVI), original (labeled EVI), and perturbed evidence sets to determine if there were significant differences in model performance across these metrics based on the type of evidence. Tables 1, 2, and 3, show the changes in various metrics based on the evidence, and Table 4, 5, and 6 indicate whether those changes are statistically significant.

LLMs follow confirmation assumptionIn Tables 1, 2 and 3, the NO_EVI and EVI column show that when \(E\) is golden evidence that helps confirm the answer, we observe \(P(H\mid E)>P(H)\) across all models, datasets and methods we used. In Tables 4, 5, and 6, when golden evidence is provided, the p-value for confidence showed at least a marginal increase, with a significant difference particularly observed in verbal, which align well with the _Confirmation Assumption_ of Bayesian epistemology. Moreover, both accuracy (ACC) and expected calibration error (ECE) showed improved results when given such confirming evidence, which leads to at least marginal difference in p-value, except for ECE in verbal. This indicates that language 

models have strong confidence and handle information well when the evidence contains purely helpful information for deriving the correct answer. This indicates language models satisfy the probability norm and calibration norm in the confirmation case.

Unlike SciQ and Trivia, the accuracy significantly improves when evidence is provided, and ECE significantly decreases in GSM8K. It shows that language models have parametric knowledge about SciQ and Trivia datasets and struggle with complex reasoning tasks without explanations and reaffirms the importance of explanation in arithmetic tasks Wei et al. (2023).

LLMs inconsistently disconfirm conflicting evidenceBased on Tables 1, 2, and 3, when conflicting evidence that does not lead to the correct answer is provided, confidence tends to decrease. However, the p-value in Tables 4, 5, and 6 reveals that conflicting evidence does not have a significant effect on confidence. With further investigation, only GPT-4o was found to significantly reduce its confidence with a p-value of 0.003.

On the other hand, accuracy decreased significantly across all confidence methods. ECE significantly increased only in the verbal and token methods. Low confidence indicates that LLMs do not follow the conflicting evidence to generate an answer, but rather that the conflicting evidence creates confusion with existing parametric knowledge, which leads to lower accuracy and higher ECE.

We made the following assumptions regarding this issue: Conflicting evidence can cause the model's confidence to become inconsistent, with overly low or high confidence appearing regardless of accuracy, which may increase ECE (although low confidence is likely to occur more frequently). Alternatively, conflicting evidence can disrupt the model's learned patterns, making its confidence less reflective of actual accuracy, thereby leading to higher ECE.

In conclusion, only GPT-4o with verbal method exhibited behavior aligned with Bayesian disconfirmation assumptions, showing a decrease in both confidence and accuracy when conflicting evidence was presented.

Llns handle contradictory evidence as golden evidenceIn most models and methods, contradictory evidence, which contains both correct and conflicting evidence in the context, shows increased confidence and accuracy compared to the no-evidence baseline, with p-values close to 0.05 for both cases. Additionally, ECE also decreases with p-value around 0.1. It means that despite the presence of conflicting information, the model appears highly confident and well-calibrated in almost all scenarios, which is similar with golden evidence case. This suggests that LLMs can effectively filter the given context and generate responses without conflicting with their parametric knowledge. Unlike the case with conflicting evidi

\begin{table}
\begin{tabular}{c|c|c c c c c c c c}  & **Dataset** & **Metric** & **No_EVI** & **EVI** & **Coincidence** & **Irrelevant** & **Conflict** & **Incomplete** & **Contradiction** \\ \hline \hline \multirow{7}{*}{**GPT-3-5-turbo**} & \multirow{3}{*}{SciQ} & Confidence & 0.851 & 0.943 & 0.835 & 0.714 & 0.827 & 0.928 & 0.945 \\  & & Accuracy \(\uparrow\) & 0.67 & 0.841 & 0.854 & 0.53 & 0.572 & 0.77 & 0.847 \\  & & EC \(\downarrow\) & 0.18 & 0.111 & 0.071 & 0.262 & 0.304 & 0.161 & 0.108 \\ \cline{2-10}  & \multirow{3}{*}{Trivia} & Confidence & 0.827 & 0.922 & 0.818 & 0.69 & 0.797 & 0.897 & 0.925 \\  & & Accuracy \(\uparrow\) & 0.846 & 0.879 & 0.971 & 0.698 & 0.702 & 0.86 & 0.869 \\  & & ECE \(\downarrow\) & 0.035 & 0.058 & 0.153 & 0.125 & 0.211 & 0.06 & 0.076 \\ \cline{2-10}  & \multirow{3}{*}{GSMSK} & Confidence & 0.74 & 0.998 & 0.988 & 0.765 & 0.931 & 0.96 & 0.949 \\  & & Accuracy \(\uparrow\) & 0.078 & 0.951 & 0.843 & 0.066 & 0.023 & 0.666 & 0.777 \\  & & ECE \(\downarrow\) & 0.662 & 0.048 & 0.148 & 0.699 & 0.911 & 0.307 & 0.197 \\ \hline \multirow{7}{*}{**GPT-4o**} & \multirow{3}{*}{SciQ} & Confidence & 0.925 & 0.986 & 0.902 & 0.861 & 0.875 & 0.948 & 0.977 \\  & & Accuracy \(\uparrow\) & 0.73 & 0.915 & 0.88 & 0.7 & 0.675 & 0.82 & 0.905 \\  & & ECE \(\downarrow\) & 0.195 & 0.073 & 0.04 & 0.171 & 0.2 & 0.128 & 0.072 \\ \cline{2-10}  & \multirow{3}{*}{Trivia} & Confidence & 0.915 & 0.933 & 0.895 & 0.878 & 0.866 & 0.909 & 0.926 \\  & & Accuracy \(\uparrow\) & 0.94 & 0.96 & 0.99 & 0.935 & 0.86 & 0.945 & 0.955 \\  & & ECE \(\downarrow\) & 0.037 & 0.027 & 0.095 & 0.063 & 0.048 & 0.036 & 0.037 \\ \cline{2-10}  & \multirow{3}{*}{GSMSK} & Confidence & 0.924 & 0.991 & 0.83 & 0.89 & 0.883 & 0.96 & 0.957 \\ \cline{1-1}  & & Accuracy \(\uparrow\) & 0.24 & 0.97 & 0.54 & 0.195 & 0.165 & 0.774 & 0.96 \\ \cline{1-1}  & & ECE \(\downarrow\) & 0.684 & 0.033 & 0.406 & 0.705 & 0.718 & 0.186 & 0.013 \\ \hline \end{tabular}
\end{table}
Table 1: The result of confirmation task with verbal confidence methods. We used 200 samples for GPT-4o due to the cost limit. NO_EFI refers the question with no context which means \(P(H~{}|~{}\theta)\), serving as baseline. Others are the case of \(P(H~{}|~{}E,\theta)\) where evidence appears in the context. EVI refers to the context in which the golden evidence from the dataset is given, while the other evidence types are those mentioned in section 3.2.

 

dence, it can be interpreted that the influence of golden evidence offsets the presence of incorrect sentences. Hence, LLMs do not consider contradictory evidence as disconfirming their beliefs.

LLMs cannot handle coincidental evidence wellWhen coincidental evidence was provided in both the token and sampling methods, confidence increased marginally (See Tables 2, 3, 5, and 6). In terms of verbal confidence in Tables 1 and 4, it was found that there was no significant difference between reliable and unreliable evidence. These indicate that the language model fails to capture the unreliability of evidence and, as a result, cannot properly reflect this in its confidence. However, based on the increased accuracy and its meaningful p-value results, it was revealed that the language model incorporated this unreliable evidence into its response to arrive at the correct answer. The ECE also decreased for both the token probability method and the sampling method. The general results for token probability and sampling methods suggest that when confidence is measured using these methods, coincidental evidence can have similar effects as golden evidence. On the other hand, there was no significant difference in ECE when using verbal confidence.

In conclusion, the language model appears to have problems with handling unreliable evidence. This is likely because, during training, the model is mostly exposed to correct data and does not frequently encounter incorrect situations.

Incomplete evidence acts as a positive hint.Incomplete evidence, though not as strong as the golden evidence case, led to a marginal increase in confidence across all confidence methods. Accuracy and ECE also showed a tendency to marginally increase and decrease, respectively. In most cases, displaying a pattern very similar to that of the golden evidence case. Incomplete evidence does not contain inaccurate information and is a partial subset of the gold evidence, acting as a hint. Similar to the contradictory evidence case, we observe that the language model is biased towards imperfect golden evidence. Therefore, while not as effective as golden evidence, the language model reflects the information from the evidence well without distraction.

LLMs are confused by irrelevant evidenceExcept for the sampling method, when irrelevant evidence was provided, confidence decreased either statistically significantly or marginally. In terms of accuracy, it was found that accuracy significantly decreased. Furthermore, in the verbal method, ECE is also notably increased.

This indicates that language models are severely distracted by irrelevant text in terms of the content of the evidence as in Shi et al. (2023). These results

Figure 2: The results of the Strength of Evidence task on the SciQ dataset with verbal confidence method. The blue bar indicates more credible, specific, recent, and experimental evidence, while the red bar represents less credible, less specific, older, and observational evidence provided to the LLMs. We found that, in all models and datasets, strong evidence leads to high confidence with verbalized confidence. However, it does not always result in improvements in ACC and ECE.

 

showed that LLMs do not align with the irrelevance assumption of Bayesian epistemology.

### LLMs on Strength of Evidence task

The results of the Strength of Evidence task using the verbalized confidence method, token probability method, sampling method are reported in Figure 2, with Figures 4 and 5 located in Appendix C. In Table 7, we calculated p-values to compare confidence, accuracy, and ECE between more reliable evidence and less reliable evidence to determine if there were significant differences in model performance across these metrics based on the strength of the evidence. Hence, as in section 4.1, we can observe the variations in different metrics depending on the power of evidence in Figures 2, 4, and 5, and check the statistical significance in Table 7.

Strong evidence can give high confidence in Verbal and Sampling methods, but cannot guarantee accurate responseIn Figures 2, 5 and Table 7, the confidence increases when more reliable evidence is provided in verbal and sampling methods. Additionally, the p-values for verbal confidence and sampling confidence between weak and strong evidence are 0.015 and 0.038, respectively. This indicates that the model's response confidence significantly increases when strong evidence is provided.

In verbal methods, as in (a), (b), (e), (f) in Figure 2, when low credible source and low detailed evidence were used, accuracy increased and ECE decreased. This suggests that in some cases, strong evidence may not be as useful as we expected for the language model to infer the correct answer. High confidence combined with low accuracy ultimately leads to overconfidence in incorrect predictions, resulting in high ECE. On the other hand, as in (c), (d),(g), (h) in Figure 2, evidence containing the latest information or experiments showed higher confidence and accuracy compared to older information or observation-based evidence. Except for GPT-3.5 with experimental evidence, the ECE of stronger evidence was also lower, indicating that using stronger evidence in the cases of timeliness and experiments results in well-calibrated models. This means that in these cases, the language model utilizes the given evidence effectively and accurately reflects the information in its predictions.

The sampling confidence showed higher accuracy when high-reliability evidence is provided in most cases except for specificity. We consider this phenomenon another positive aspect of self-consistent decoding Wang et al. (2023). A single response might not fully capture the reliability of evidence, such as credibility, timeliness, etc. However, multiple responses can increase the likelihood of accurately reflecting these aspects.

In the case of specificity, both verbalized confidence and sampling failed to adequately to reflect the concreteness of the evidence in the responses. We interpreted that more detailed information can enhance confidence, but it also suggests that such excessive information may hinder the extraction of correct answers that match the question.

Token probability cannot reflect various degrees of reliabilityAs in Figure 4 in Appendix C, with token probability, confidence did not increase even when stronger evidence was presented. For example, with token probability, when the specificity of the evidence was altered or when the source's credibility was varied in GPT-4o, it failed to reflect confidence according to the strength of the evidence accurately. However, it accurately reflected reliability changes according to the source's credibility, timeliness, and whether an experiment was conducted in the evidence to its accuracy. Additionally, it showed a decrease in ECE in cases of timeliness and experimental evidence.

Through this experiment, we found that when stronger evidence is provided to the language model, it can significantly increase its verbalized and sampling confidence. However, this does not always lead to improvements in accuracy-related performance.

## 5 Ablation

LLMs tend to focus more on correct than incorrect informationIn Section 4.1, we interpreted that the language model possesses a certain degree of knowledge about the question in its parameters and tends to be biased towards contexts aligned with this parametric knowledge rather than context hindering it, as seen in golden, contradictory, and incomplete evidence. To justify this, we conducted an experiment adjusting the ratio of golden sentences in conflicting, incomplete, and contradictory evidence. Figure 3 (a) and (d) show that as the number of original golden sentences decreases and the conflicting sentence increases, the performance of the language model gradually declines. However, it decreases significantly when there are no golden sentences left. Moreover, Figure 3 (b) and (e) demonstrate that as the original golden sentence 

decreases, performance decreases. On the other hand, Figure 3 (c) and (f) indicate that if the golden evidence sentences is sufficiently given, increasing the number of contradictory sentences does not affect the confidence and performance even if both of the numbers of contradictory sentences and golden evidence sentences are same. This shows that the language model focuses more on the given golden evidence in the context than inaccurate evidence, and this is why it maintains confidence and calibration despite incomplete and contradictory evidence.

Why do LLMs get confused by irrelevant context?Two interpretations are possible for the irrelevant case

1. The language model does not recognize irrelevant evidence as irrelevant when it is in the same field but differs in content.
2. The language model considers irrelevant evidence as a kind of noise, which distracts the model and causes confusion.

To verify (1), instead of extracting irrelevant evidence from the same dataset, we used contexts from different datasets, for SciQ and TriviaQA dataset, we used evidence of GSM8K, and for GSM8K, using TriviaQA. As we can see in Figure 6 in Appendix G, even when using a new irrelevant sentence, it did not completely match the completely irrelevant assumption. However, surprisingly, when using evidence from a completely different field, we found that the confidence, accuracy, and ECE metrics approached closer to the baseline no evidence case (P(H)) than when we used evidence where the content was different but the field was the same. This implies that as the irrelevance increases, the LLMs become less distracted by the context. Therefore, we interpreted that there is a possibility that the LLMs satisfy the irrelevant assumption of Bayesian epistemology.

## 6 Why Do LLMs Struggle to Follow Bayesian Assumptions?

We speculate that LLMs may not fully adhere to Bayesian assumptions due to limitations in both their training data and methods. Pretraining datasets, primarily sourced from web crawls with filtering and books Gao et al. (2020), expose models mostly to well-justified information and correct explanations. This allows them to excel in deriving correct answers but limits their ability to handle coincidental evidence or conflicting beliefs, which are underrepresented in the data.

In addition to data limitations, the training meth

Figure 3: The results for the degree of variations in evidence for the SciQ dataset with verbalized method. We modified the number of conflicting sentences in conflicting evidence, sentences in incomplete evidence, and contradictory sentences in contradictory evidence (See Appendix F for entire results).

 

ods used for LLMs diverge significantly from human language acquisition. Humans encounter unreliable and conflicting situations through interaction and experience with the real world. In contrast, LLMs are trained in a largely supervised manner, resulting in a lack of semantic understanding (Bender and Koller, 2020; Bisk et al., 2020; Soni et al., 2024). As a result, these limitations may explain why LLMs fail to fully align with Bayesian assumptions when faced with conflicting or unexpected scenarios, as they lack the experiential grounding to properly handle such evidence.

## 7 Conclusion

In this paper, we explored how changes in the informativeness and reliability of evidence affect the confidence and response of language models. Specifically, we examined how well language models stick to the probability and calibration norms outlined in Bayesian epistemology. We demonstrated that language models partially align with Bayesian epistemology, following confirmation assumptions but failing to adhere to disconfirmation and irrelevance assumptions. It can be interpreted that language models do not possess a justified belief in the view of Bayesian epistemology. Additionally, we found that LLMs show a bias toward golden evidence and modify their confidence and response relative to the degree of irrelevance, which helps clarify their deviation from Bayesian assumptions. These findings provide philosophical insight into the nature of "belief" in LLMs, highlighting their biases and limitations.

## 8 Limitations

In this paper, we did not theoretically investigate the causes behind the observed phenomena, such as the training algorithm and model architecture, leaving such investigations and their potential implications for future research. One limitation of our dataset is the varying nature of conflicting evidence, some evidence is entirely incompatible, while other types obstruct correct answers. Hence, as we varied the extent of irrelevance in the ablation study, a finer classification of conflicting evidence could benefit future research. Additionally, Bayesian episttemology is not the only theory for defining knowledge. Therefore, our results do not imply a definitive conclusion about whether LLMs possess beliefs or knowledge. We did not conduct a human evaluation as a baseline in this study as our focus was on aligning language models with ideal Bayesian episttemology. Future research could incorporate human evaluation to further assess how models' belief updating and confidence calibration compare to human cognitive processes. Lastly, further research using more complex and practical datasets, as well as developing new algorithms to address the issues identified in this study, will be valuable in advancing robust AI systems.

## 9 Ethics Statement

In the preparation of this paper, we utilized GPT-4o, for grammatical corrections and coding assistance. This technology served as an auxiliary resource to enhance the clarity and accuracy of our work, without directly influencing the research outcomes or decision-making processes involved. We acknowledge the support provided by OpenAI's GPT-4o in refining the presentation of our findings, ensuring that our use of this tool adheres to ethical guidelines and does not compromise the integrity of our research.

## 10 Acknowledgement

This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (RS-2019-HI190075) Artificial Intelligence Graduate School Program (KAIST) and Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City.

## References

* Audi (1997) Robert Audi. 1997. _Epistemology: A Contemporary Introduction to the Theory of Knowledge_. Routledge, New York.
* Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5185-5198, Online. Association for Computational Linguistics.
* Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8718-8735, Online. Association for Computational Linguistics 

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafaulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
* Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktaschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations.
* Chen et al. (2023) Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023. How many demonstrations do you need for in-context learning?
* Chia et al. (2023) Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023. Contrastive chain-of-thought prompting.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems.
* Feng et al. (2023) Yu Feng, Ben Zhou, Haoyu Wang, Helen Jin, and Dan Roth. 2023. Generic temporal reasoning with differential analysis and explanation.
* Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling.
* Team et al. (2020) Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhone Xu, Ryan Doherty, Eli Collins, Clements Meyer, Elizza Rutherford, Eric Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliakesei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, MiA Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremia Liu, Andras Orban, Fabian Gura, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJeong Choe, Agoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd Al Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piotqueras, Maxim Kirkun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Brusttemi, Nataliev Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloinarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishab Joshi, Rahma Chaabouni, Deeni Fathira, Arun Ahuya, Gauray Singh Tomar, Evn Sentert, Martin Chadwick, Ilya Kornalakov, Nithya Attaluuri, Itaki Hurrate, Ruibo Liu, Yunxuan Li, Sarah Logan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakes Hartman, Xavier Garcia, Thanumalayan Sankaramarayana Pillali, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adria Puigolennech Badia, David Reitter, Minanna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Ying Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplysmann, Geoff Brown, ElliCatt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Wetsho, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stiermgren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henrykh Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Blowich, Kehang Han, Peter Humphreys, Thisbuult Sellam, James Bradbury, Varun Godobe, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sebastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivastan Srinivasan, Heyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Panje, Jian Li, Yuji\({}_{4}\) Li, Minh Giang, Alexander Netitz, Zaheer Abbas, Sarah York, Rachel Reid, Elizabeth Cole, Akanksha Chowdhery, Dipjanjan Das, Dominika Rogozinska, Vitaliy Nikolaev, Pablo Spreedmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huji Hu, Raoul de Lieedexke, Justin Gilmer, Carl Saarotim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessite Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villella, Luyu Wang, Wenhao Jia, Matthew Rhatz, Mai Gimenez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputto, Kiran Vodraz 

halli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van Driesschee, Tao Wang, Fan Yang, Shuo yin Chang, Paul Komarek, Ross McIlroy, Mario Lucic, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakerri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shirani Agrawal, Arthur Mensch, Kadar Sopparkar, Karel Lenc, Timothy Chung, Acad Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Holubsky, Xuehn Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thias Kabaghara, Jay Pavgadshi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarial Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Inuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiaec Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphael Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjiae Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zihao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Breivins, Ted Klimenko, Chih-Kuan Yeh, Soravit Chappinyo, Jiaqi Mu, Oscar Chang, Mantus Pajarskas, Carrie Mulir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Scholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjosund, Sebastien Cevey, Zach Gleicher, Thi Avrahamt, Anudhyam Borl, Hans Srinivasan, Vittorio Selco, Rhys May, Konstantinos Aisopos, Leonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adria Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesch, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhajit Roy, Ethan Dyer, Victor Campos Campos, Alex Tomala, Yunhao Tang, Dalai Eldawady, Elspeth White, Basil Mustafa, Oran Lang, Abhishekjindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiete, Ce Zheng, Phoee Thacker, Caglar Unlu, Zhishnuai Zhang, Mohammad Saleh, James Swenson, Max Bileschi, Piiyush Pattil, Ankesh Anand, Roman Ring, Katerina Tsillas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sinath, Maribeth Reauh, Sayed Hadi Hashemi, Richard Ives, Yana Hassson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Stotitaux, Michela Paganimi, Jean-Baptiste Lespiau, Alexandre Moustfarek, Samer Hassan, Kaushik Shirakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Shaahan, Vedant Misra, Cheng Li, Nemaraja Rakicevic, Mostafa Dephhani, Fangyu Liu, Sid Mittal, Junnyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Mathew Lamm, Nicola De Cao, Charlie Chen, Siddharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristite Seymore, Sergey Lubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohannaney, Johan Jobgin, Egor Filonov, Tomasx Kepa, Yomna Eldawy, Jiawerm Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subba Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Pedapudi, Adam Kurrzoch, Shuntung Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tejo Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Liijuan Liu, Julse Walter, Hamid Moghaddam, Arun Kah Shore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddinitis Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafcezul Rahman Mohammad, Josip Matak, Yad Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Umaphyi, Anton Algmyr, Timothee Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinier, Raakeshi Shivannan, Aleksandchuk, Josie Li, Carrie Spadne, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zhihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovich, Pavan Kumar Reddy M, Aarush Selavan, Mikhail Delktarev, Golnaz Ghiasi, Erdorn Guven, Himsmuth Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akrelund, Francois-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana-Eleonora Marinescu, Martin Bolle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulse Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Rabiej, Vipul Ranjan, Krzysztof Styrc, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mutd Bansal, Alexei Robsky, Geoff Bacon, David Greene, Dani Mirylenka, Chen Zhou, Obaid Sarvana, Abhinanyu Goyal, Samuel Andermatt, Patrick Siegel, Ben Horn, Assaf Israel, Francesco Pongetiti, Chih-Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah O Donanile, Sebastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrokh, Chentskai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wojciech Fica, Eric Malmi, Qiu Tan, Dan Banica, Daniel Balle, Ryan Pham, 

Yanping Huang, Diana Avram, Hongzhi Shi, Jasiot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Franavi Fotharaju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Baeut, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chemnei Li, Sihyuan Chen, Niccolo Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarminsha Velury, Himadri Choudhury, Jamie Hall, Premel Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chilkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Abdel, Sujeevan Rajayogam, Travis Chroma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojciech Razdkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Predesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Brazilian skas, Andrei Sozanschil, Matthew Hayes, Hoctor Fernandez Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Antte Karrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushl Mittal, Akhil Uddhu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaldelin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leunert, Mahmoud Almahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Onticveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Risbluh Jain, Jonathan Usestato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Nakar, Michael Azzam, Matthews Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhjijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Louveir Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castano, Irene Giannoums, Woyoel Kim, Mikolaj Rybitski, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiiesner, Diana Gage Wright, Yawen Wei, Harsha Vashishst, Yank Kulzhskaya, Jay Hoover, Maigo Leo Lu, Lil Chimze Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Knorin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abihishe Chakalam, Ginger Perry, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnampalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Te, Rama Pasumartriushtan Linnitz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Padvaru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achindya Singhal, Dayou Du, Dan McKinnnon, Naratash Antropova, Tolga Bolukbasi, Orpad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Croker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Remi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitire, Hannal Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseen Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHosseien Bateni, Dennis Duan, Vlad Friouin, Meghana Thotsukuri, Tom Natan, Matthieu Geist, Ser an Grigni, Hui Li, Jiayu Ye, Ofir Royal, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Venukov, Roy Greig, Jennifer Beattie, Emily Cavenesi, Libin Bai, Julian Eisenschlos, Alex Korchemiy, Tomm Tsai, Mimi Jaservic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Hueye Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Liting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kaffe, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyige, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyuma Narayana, Jing Li, Saaber Fateh, John Wieting, Omar Ajmerni, Bengio Uria, Yeongil Ko, Laura Knight, Amelie Heliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santamaria-Fernandez, Sonan Goenken, Wenny Yustalin, Robin Strudel, Ali Eqlursh, Charlie Deck, Hyo Lee, Zenglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Podler, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Arnold Gulat, Jasmine Liu, Xinyu Ye, Bart Chraszcz, Lily Wang, Nikhil Sethi, Tiannu Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, TY Lu, Abe Ittycheriah, Prakash 

Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Aliurea Ghaffarkhah, Morgane Riviere, Alanna Walton, Clement Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucinska, David Bridson, Dario de Ceesare, Tom Hudson, Piermais Mendolicchio, Levi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Duan, Dong Li, Surya Bhupatriju, Dan Hurt, Haroom Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekrur, Mihir Sanjay Gale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuryuru, John Aslanides, Nikul Vyas, Martin Wicke, Xiao Ma, Evgenii Eltshyee, Ninta Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polyets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherijl Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgski, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kiearan Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Barr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jiger Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Teiyi M., Xinwn Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Ammuta Muthal, Senaka Butpitiya, Sarthak Juahari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyu Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Avietias, Paulina Lee, Vit Listik, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Muller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Thandhandania, Manish Katyal, Akshay Gupta, Athara Paurlekar, Divya Pitta, Jing Zhao, Vivana Bhatia, Yashodha Bhavanni, Olmar Alhadiao, Xiaolin Li, Peter Danchepf, Dennis Tu, Alex Pine, Verz Filippova, Abhipso Ghosh, Ben Limonchik, Bharagava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Inna Li, Kalind Thakkar, Kuanysh Omarov, Kwshal Majmundar, Michael Alverson, Michael Kubarski, Mohale Patel, Mudui Jain, Maksim Zachelin, Paolo Pelagatti, Rohin Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandran, Xiangkai Zeng, Ben Baiach, Laura Weidinger, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. 2024. Gemini: A family of highly capable multimodal models.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks.

Alan Hajek. 2003. What conditional probability could not be. _Synthesse_, 137(3):273-323.

Alan Hajek and Sterhan Hartmann. 2010. Bayesian epistemology. In DancyJ, editor, _A Companion to Epistemology_. Blackwell.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding.

Paul Horwich. 1982. _Probability and Evidence_. Cambridge University Press, Cambridge.

Golin Howson. 2000. _Home's Problem: Induction and the Justification of Belief_. Oxford University Press, New York.

Lei Huang,Weijing Yu, Weitao Ma, Weihong Zhong, Zhangying Feng, Haotian Wang, Qiangling Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. _ArXiv_, abs/2311.05232.

Mandar Joshi, Eunslo Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schaefer, Zac Hatfield-Dodds, Nova DasSamra, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, 

Jackson Kermion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know.
* Kuhn et al. [2023] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.
* Lanham et al. [2023] Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kermion, Komille Lukosite, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schaifer, Oliver Rusch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. 2023. Measuring faithfulness in chain-of-thought reasoning.
* Lee et al. [2023] Noah Lee, Na Min An, and James Thorne. 2023. Can large language models capture dissenting human voices? In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 4569-4585, Singapore. Association for Computational Linguistics.
* Lin [2004] Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words.
* Mielke et al. [2022] Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Yan Lan Boureau. 2022. Reducing conversational agents' overconfidence through linguistic calibration. _Transactions of the Association for Computational Linguistics_, 10:857-872.
* OpenAI et al. [2022] OpenAI, ; Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jakle Berdine, Gabriel Bernadeti-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fois Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dolhan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simon Posada Fishman, Juston Forte, Isabella Fullford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Goginen, Gabriel Goh, Rapha Gonitijo-Loren, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoechele, Brandon Houghton, Kenny Hsu, Shengliu Liu, Xia Hu, Joost Huizing, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhn Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tumer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christiian Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciauk, Andrew Kondrich, Arts Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephane Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Luc, Anna Makanju, Kim Malfacil, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkinkin, Vinime Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murat, Oleg Park, David Mely, Ashvin Nair, Reitichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascaldo, Joel Parish, Emy Parparita, Alex Passoso, Mikhail Pavlov, Andrew Peng, Adam Perellaman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rao, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Salatterli, Ted Sanders, Shibani Santurkar, Girish Asstry, Heather Schmidt, David Schmur, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Feijpe Petrokski Such, Natalie Summers, Thya Sutskever, Jie Tang, Nikolas Tezak, Madeeline Thompson, Phil Tillet, Amin Tootoonchin, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Ceron Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, Cy Weinmann, Akila Weihlinta, Peter Welinder, Jiayi Wen, Lilian Weng, Matt Wiethoff, Dave Willmer, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael 

Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023. Gpt-4 technical report.
* Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Scharli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context.
* Soni et al. (2024) Nikita Soni, H. Andrew Schwartz, Joao Sedoc, and Niranjan Balasubramanian. 2024. Large human language models: A need and the challenges. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8631-8646, Mexico City, Mexico. Association for Computational Linguistics.
* Srivastava et al. (2020) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abuwal Md Shoeb, Abbabakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, Agnieszka Kluska, Attor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahaune, Ananthramam S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhnuller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anhu Vuong, Animeesh Gupta, Anna Gottardi, Antonio Norelli, Annu Venkatesh, Arash Gholamidacovoodi, Afra Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avila Efrat, Aykat Erdem, Ayla Karakag, B. Ryan Roberts, Bao Shene Loe, Barret Zoph, Bartomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatina, Behman Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Douc, Catherine Stinson, Cedrick Argpueta, Cesar Ferri Ramirez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damine Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Mosegui Gonzalez, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dobah, David Draxard, David Jurgens, Debajvoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyleko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilvar Burzan, Dimitri Coello Mollo, Diyi Yang, Dong-Ho Lee, Delyn Schradder, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segai, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Einnce Engefu Manyasi, Evgenii Zhelltonozhskii, Fanyue Xia, Fatemin Sir, Fernando Martinez-Plumed, Francesca Happpe, Francois Chollet, Frieda Rong, Gauraw Mishra, Genta Indra Winata, Gerard de Melo, German Kruszewski, Giambattista Parascando, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harst Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schutzre, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernandez Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekarreira Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroon Taal, Jesse Engel, Jesusipabo Alabi, Jiacheng Xu, Jianning Song, Jillian Tang, Joan Waveru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jorg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanlercz, Karen Livescu, Karl Krauth Korabita Gopalakrishnan, Katarina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Cory Mathewson, Kristen Chaifullo, Ksenia Shkartuta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Limau Dugan, Lianhui Qiu, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luhe He, Luis Oliveros Colon, Luke Metz, Lutfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartige ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Battaran, Marco Marelli, Marco Maru, Maria Jose Ramirez Quintana, Marie Tolikohn, Mario Guialinelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Matyus Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Fu, Michael Ivantiskiy, Michael Strutt, Michael Strutvich, Michael Swedrowski, Michelie Bevilacqua, Michihiro Yasunaga, Mihit Zak, Mike Cain, Mince Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhedeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Mueninghoff, Nitish Shirishs Keskar, Viveldius S. Iyer, Noah Constant, Noah Field, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Hutt, Pinyu Hwang, Piort Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oil, Quiazhou Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphael Milliere, Rhythm Garg, Richard 

Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelev, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Sailr M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebasian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shububn Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thromeyer, Simone Melzi, Siva Reddy, Sachna Priscilla Makin, Soo-Hwan Lee, Spencer Torene, Sriharshan Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Taria Ali, Tristan Hashimoto, Te-Lin Wu, Theo Desordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberis Nkinyili, Timo Schick, Timofei Kornev, Titus Tundunya, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raumak, Vinay Ramasesh, Vinay Uday Prabhu, Visshuk Padmakumar, Vivek Srikumar, William Fedus, William Sanders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqi Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonental Belinkev, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijia Wanq, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
* Sun et al. (2022) Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang, Jaoaquin Vanstoren, John Mitchell, Kai Shu, Kaid Xi, Kai-Wei Chang, Lifang He, Lift Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji,uman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yamfang Ye, Yinzhi Cao, Yong Chen, and Yue Zhao. 2024. Trustllm: Trustworthiness in large language models.
* Talbott (2006) William Talbott. 2006. Bayesian epistemology. In Edward Zalta, editor, _Stanford Encyclopedia of Philosophy_.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
* Tian et al. (2023a) Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. 2023a. Fine-tuning language models for factuality.
* Tian et al. (2023b)Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023b. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.
* Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting.
* Passend (2023) Olav Benjamin Vassend. 2023. What hinge epistemology and bayesian epistemology can learn from each other. _Asian Journal of Philosophy_, 2(2):1-21.
* Wang et al. (2023a) Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023a. Towards understanding chain-of-thought prompting: An empirical study of what matters.
* Wang et al. (2023b)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models.
* Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
* Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In _Proceedings of the 3rd Workshop on Noisy User-generated Text_, pages 94-106, Copenhagen, Denmark. Association for Computational Linguistics.
* Williamson (2010) Jon Williamson. 2010. _In Defence of Objective Bayesianism_. Oxford University Press.
* Xiao and Wang (2021) Yijun Xiao and William Yang Wang. 2021. On hallucination and predictive uncertainty in conditional language generation. _ArXiv_, abs/2103.15025 

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.
* Yu and Ji (2024) Pengfei Yu and Heng Ji. 2024. Information association for language model updating by mitigating lm-logical discrepancy.
* Zhang et al. (2023) Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Cheng Zhou, Xinbing Wang, and Luoyi Fu. 2023. Enhancing uncertainty-based hallucination detection with stronger focus. _ArXiv_, abs/2311.13230.
* Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback.
* Zhou et al. (2023) Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 5506-5524, Singapore. Association for Computational Linguistics.

 



## Appendix A Results of Confirmation task

### Results of Confirmation task

\begin{table}
\begin{tabular}{c|l|l c c c c c c c} \hline  & **Dataset** & **Metric** & **No_EVI** & **EVI** & **Coincidence** & **Irrelevant** & **Conflict** & **Incomplete** & **Contradiction** \\ \hline \hline \multirow{8}{*}{**GPT-3.5-turbo**} & \multirow{4}{*}{SciQ} & Confidence & 0.671 & 0.781 & 0.785 & 0.594 & 0.638 & 0.723 & 0.764 \\  & & Accuracy ↑ & 0.676 & 0.829 & 0.839 & 0.526 & 0.6 & 0.792 & 0.837 \\  & & ECE ↓ & 0.312 & 0.171 & 0.154 & 0.44 & 0.381 & 0.205 & 0.16 \\ \cline{2-10}  & \multirow{4}{*}{Trivia} & Confidence & 0.834 & 0.864 & 0.894 & 0.699 & 0.759 & 0.843 & 0.849 \\  & & Accuracy ↑ & 0.858 & 0.872 & 0.976 & 0.653 & 0.742 & 0.851 & 0.857 \\  & & ECE ↓ & 0.134 & 0.127 & 0.127 & 0.324 & 0.251 & 0.141 & 0.139 \\ \cline{2-10}  & \multirow{4}{*}{GSMSK} & Confidence & 0.218 & 0.932 & 0.933 & 0.172 & 0.738 & 0.765 & 0.801 \\  & & Accuracy ↑ & 0.098 & 0.961 & 0.852 & 0.068 & 0.028 & 0.677 & 0.755 \\  & & ECE ↓ & 0.777 & 0.046 & 0.148 & 0.725 & 0.939 & 0.299 & 0.222 \\ \hline \multirow{8}{*}{**GPT-40**} & \multirow{4}{*}{SciQ} & Confidence & 0.621 & 0.799 & 0.833 & 0.565 & 0.653 & 0.744 & 0.813 \\  & & Accuracy ↑ & 0.711 & 0.92 & 0.905 & 0.675 & 0.655 & 0.835 & 0.925 \\  & & ECE ↓ & 0.276 & 0.082 & 0.1 & 0.314 & 0.334 & 0.165 & 0.078 \\ \cline{2-10}  & \multirow{4}{*}{Trivia} & Confidence & 0.837 & 0.916 & 0.911 & 0.824 & 0.824 & 0.889 & 0.91 \\  & & Accuracy ↑ & 0.944 & 0.955 & 0.99 & 0.905 & 0.82 & 0.94 & 0.95 \\  & & ECE ↓ & 0.06 & 0.047 & 0.01 & 0.088 & 0.173 & 0.064 & 0.05 \\ \cline{2-10}  & \multirow{4}{*}{GSMSK} & Confidence & 0.354 & 0.865 & 0.54 & 0.299 & 0.372 & 0.755 & 0.842 \\ \cline{2-10}  & & Accuracy ↑ & 0.249 & 0.97 & 0.505 & 0.227 & 0.191 & 0.83 & 0.955 \\ \cline{2-10}  & & ECE ↓ & 0.715 & 0.03 & 0.473 & 0.697 & 0.74 & 0.157 & 0.037 \\ \hline \end{tabular}
\end{table}
Table 2: The result of confirmation task with token probability method. We used 200 samples for GPT-40 due to the cost limit. NO_EVI refers the question with no context which means \(P(H~{}|~{}\theta)\), serving as baseline. Others are the case of \(P(H~{}|~{}E,\theta)\) where evidence appears in the context. EVI refers to the context in which the golden evidence from the dataset is given, while the other evidence types are those mentioned in section 3.2.

\begin{table}
\begin{tabular}{c|l|l l l l l l l l} \hline  & **Dataset** & **Metric** & **No\_EVI** & **EVI** & **Conicidence** & **Irrelevant** & **Conflict** & **Incomplete** & **Contradiction** \\ \hline \hline \multirow{8}{*}{**GPT-3.5-turbo**} & \multirow{4}{*}{Tivria} & Confidence & 0.874 & 0.921 & 0.916 & 0.798 & 0.828 & 0.888 & 0.922 \\  & & Accuracy ↑ & 0.693 & 0.846 & 0.853 & 0.551 & 0.617 & 0.777 & 0.853 \\  & & ECE ↓ & 0.18 & 0.076 & 0.077 & 0.248 & 0.211 & 0.111 & 0.074 \\ \cline{2-10}  & \multirow{4}{*}{Trivia} & Confidence & 0.921 & 0.939 & 0.963 & 0.822 & 0.862 & 0.924 & 0.934 \\  & & Accuracy ↑ & 0.869 & 0.884 & 0.979 & 0.668 & 0.693 & 0.856 & 0.884 \\  & & ECE ↓ & 0.057 & 0.059 & 0.034 & 0.154 & 0.17 & 0.072 & 0.076 \\ \cline{2-10}  & \multirow{4}{*}{GSMSK} & Confidence & 0.422 & 0.986 & 0.977 & 0.377 & 0.838 & 0.86 & 0.848 \\  & & Accuracy ↑ & 0.12 & 0.967 & 0.849 & 0.059 & 0.028 & 0.716 & 0.756 \\  & & ECE ↓ & 0.302 & 0.036 & 0.138 & 0.318 & 0.81 & 0.144 & 0.091 \\ \hline \multirow{8}{*}{**GPT-40**} & \multirow{4}{*}{SciQ} & Confidence & 0.872 & 0.968 & 0.959 & 0.852 & 0.871 & 0.923 & 0.965 \\  & & Accuracy ↑ & 0.694 & 0.934 & 0.924 & 0.708 & 0.698 & 0.84 & 0.933 \\ \cline{1-1}  & & ECE ↓ & 0.18 & 0.06 & 0.102 & 0.149 & 0.114 & 0.132 & 0.066 \\ \cline{1-1} \cline{2-10}  & \multirow{4}{*}{Trivia} & Confidence & 0.845 & 0.973 & 0.973 & 0.943 & 0.918 & 0.966 & 0.97 \\ \cline{1-1}  & & Accuracy ↑ & 0.945 & 0.969 & 0.99 & 0.924 & 0.843 & 0.924 & 0.959 \\ \cline{1-1}  & & ECE ↓ & 0.053 & 0.026 & 0.016 & 0.04 & 0.122 & 0.042 & 0.038 \\ \cline{1-1} \cline{2-10}  & \multirow{4}{*}{GSMSK} & Confidence & 0.506 & 0.958 & 0.684 & 0.481 & 0.529 & 0.875 & 0.957 \\ \cline{1-1}  & & Accuracy ↑ & 0.3 & 0.969 & 0.587 & 0.257 & 0.224 & 0.829 & 0.969 \\ \cline{1-1}  & & ECE ↓ & 0.206 & 0.065 & 0.156 & 0.224 & 0.305 & 0.103 & 0.051 \\ \hline \end{tabular}
\end{table}
Table 3: The result of confirmation task with sampling method. We used 200 samples for GPT-40 due to the cost limit. NO_EVI refers the question with no context which means \(P(H~{}|~{}\theta)\), serving as baseline. Others are the case of \(P(H~{}|~{}E,\theta)\) where evidence appears in the context. EVI refers to the context in which the golden evidence from the dataset is given, while the other evidence types are those mentioned in section 3.2.



Results of p-value for Confirmation task

We consider \(p\leq 0.05\) as statistically significant, while \(0.05<p\leq 0.1\) is regarded as marginally significant.

 

Results of Strength of evidence task

Figure 4: The results of the Strength of Evidence task on the SciQ dataset with token probability method. The blue bar represents the cases where the strength of evidence is high. Specifically, the blue bar indicates the context from more credible sources, more specific, recent, and experimental evidence, while the red color represents less credible sources, less specific, old, and observational evidence.

Figure 5: The results of the Strength of Evidence task on the SciQ dataset with sampling method. The blue bar represents the cases where the strength of evidence is high. Specifically, the blue bar indicates the context from more credible sources, more specific, recent, and experimental evidence, while the red color represents less credible sources, less specific, old, and observational evidence.

 



## Appendix D Results of pvalue for Strength of evidence task

\begin{table}
\begin{tabular}{l|l l l} \hline \multicolumn{3}{c}{**P-values between more reliable evidence and less reliable evidence**} \\ \hline Method & **Verbal** & **Token** & **Sampling** \\ \hline Confidence & 0.015 & 0.758 & 0.038 \\ ACC & 0.486 & 0.217 & 0.222 \\ ECE & 0.401 & 0.048 & 0.817 \\ \hline \end{tabular}
\end{table}
Table 7: The p-values obtained from paired t-tests for confidence, accuracy, and ECE between the less reliable evidence and strong evidence.





## Appendix E Results of Gemini-1.5-flash

\begin{table}
\begin{tabular}{c|l|l c c c c c c c} \hline  & **Dataset** & **Metric** & **High Source** & **Low Source** & **High Spec** & **Low Spec** & **Recent** & **Old** & **Experiment** & **Observation** \\ \hline \multirow{3}{*}{**Gemini-1.5-Flash**} & \multirow{2}{*}{SciQ} & Confidence & 0.981 & 0.822 & 0.976 & 0.944 & 0.965 & 0.917 & 0.974 & 0.933 \\  & & Accuracy \(\uparrow\) & 0.865 & 0.855 & 0.795 & 0.87 & 0.86 & 0.76 & 0.83 & 0.785 \\  & & ECC \(\downarrow\) & 0.119 & 0.065 & 0.181 & 0.084 & 0.105 & 0.168 & 0.158 & 0.148 \\ \hline \end{tabular}
\end{table}
Table 11: The result of the strength of evidence task with the verbal method. We used 200 samples for Gemini-1.5-Flash due to the cost limit.

\begin{table}
\begin{tabular}{c|l|l l c c c c c c} \hline  & **Dataset** & **Metric** & **No\_EVI** & **EVI** & **Coincidence** & **Irrelevant** & **Conflict** & **Incomplete** & **Contradiction** \\ \hline \hline \multirow{4}{*}{**Gemini-1.5-Flash**} & \multirow{4}{*}{SciQ} & Confidence & 0.93 & 0.983 & 0.851 & 0.554 & 0.771 & 0.936 & 0.96 \\  & & Accuracy \(\uparrow\) & 0.656 & 0.86 & 0.885 & 0.66 & 0.62 & 0.79 & 0.87 \\  & & ECC \(\downarrow\) & 0.28 & 0.123 & 0.082 & 0.263 & 0.218 & 0.146 & 0.09 \\ \cline{2-10}  & \multirow{4}{*}{Trivia} & Confidence & 0.909 & 0.881 & 0.791 & 0.45 & 0.644 & 0.855 & 0.854 \\  & & Accuracy \(\uparrow\) & 0.855 & 0.91 & 0.97 & 0.62 & 0.73 & 0.88 & 0.93 \\  & & ECC \(\downarrow\) & 0.087 & 0.063 & 0.179 & 0.252 & 0.234 & 0.086 & 0.076 \\ \cline{2-10}  & \multirow{4}{*}{GSM8K} & Confidence & 0.987 & 1.0 & 0.703 & 0.546 & 0.552 & 0.967 & 0.967 \\ \cline{2-10}  & & Accuracy \(\uparrow\) & 0.165 & 0.97 & 0.64 & 0.15 & 0.07 & 0.74 & 0.965 \\ \cline{2-10}  & & ECC \(\downarrow\) & 0.822 & 0.03 & 0.373 & 0.476 & 0.542 & 0.277 & 0.044 \\ \hline \end{tabular}
\end{table}
Table 8: The result of confirmation task with verbal method. We used 200 samples for Gemini-1.5-Flash due to the cost limit.

\begin{table}
\begin{tabular}{c|l|l c c c c c c c} \hline  & **Dataset** & **Metric** & **high Source** & **Low Source** & **High Spec** & **Low Spec** & **Recent** & **Old** & **Experiment** & **Observation** \\ \hline \multirow{3}{*}{**Gemini-1.5-Flash**} & \multirow{3}{*}{SciQ} & Confidence & 0.944 & 0.955 & 0.947 & 0.942 & 0.943 & 0.936 & 0.942 & 0.955 \\  & & Accuracy \(\uparrow\) & 0.855 & 0.855 & 0.8 & 0.85 & 0.88 & 0.76 & 0.835 & 0.81 \\  & & ECC \(\downarrow\) & 0.145 & 0.145 & 0.2 & 0.15 & 0.12 & 0.24 & 0.165 & 0.19 \\ \hline \end{tabular}
\end{table}
Table 12: The result of the strength of evidence task with the token probability method. We used 200 samples for Gemini-1.5-Flash due to the cost limit.

\begin{table}
\begin{tabular}{c|l|l c c c c c c c} \hline  & **Dataset** & **Metric** & **No\_EVI** & **EVI** & **Coincidence** & **Irrelevant** & **Conflict** & **Incomplete** & **Contradiction** \\ \hline \hline \multirow{4}{*}{**Gemini-1.5-Flash**} & \multirow{4}{*}{SciQ} & Confidence & 0.952 & 0.991 & 0.987 & 0.91 & 0.955 & 0.98 & 0.992 \\  & & Accuracy \(\uparrow\) & 0.656 & 0.898 & 0.911 & 0.57 & 0.617 & 0.799 & 0.904 \\  & & ECC \(\downarrow\) & 0.294 & 0.107 & 0.081 & 0.339 & 0.337 & 0.186 & 0.09 \\ \cline{2-10}  & \multirow{4}{*}{Trivia} & Confidence & 0.929 & 0.969 & 0.929 & 0.885 & 0.907 & 0.969 & 0.99 \\  & & Accuracy \(\uparrow\) & 0.839 & 0.894 & 0.839 & 0.599 & 0.658 & 0.84 & 0.894 \\  & & ECC \(\downarrow\) & 0.11 & 0.076 & 0.11 & 0.293 & 0.248 & 0.134 & 0.096 \\ \cline{2-10}  & \multirow{4}{*}{GSM8K} & Confidence & 0.72 & 0.996 & 0.947 & 0.705 & 0.852 & 0.936 & 0.994 \\ \cline{2-10}  & & Accuracy \(\uparrow\) & 0.19 & 0.97 & 0.633 & 0.151 & 0.061 & 0.721 & 0.97 \\ \cline{1-1}  & & ECC \(\downarrow\) & 0.53 & 0.034 & 0.314 & 0.554 & 0.791 & 0.215 & 0.032 \\ \hline \end{tabular}
\end{table}
Table 9: The result of confirmation task with token probability method. We used 200 samples for Gemini-1.5-Flash due to the cost limit.





\begin{table}
\begin{tabular}{c|c|l c c c c c c c c c} \hline  & Dataset & Metric & High Source & Low Source & High Spec & Low Spec & Recent & Old & Experiment & Observation \\ \hline \multirow{3}{*}{**Gemini-1.5-Flash**} & \multirow{2}{*}{SciQ} & Confidence & 0.974 & 0.986 & 0.978 & 0.981 & 0.97 & 0.964 & 0.973 & 0.982 \\  & & Accuracy ↑ & 0.879 & 0.854 & 0.809 & 0.853 & 0.879 & 0.774 & 0.83 & 0.791 \\ \cline{1-1}  & & ECE ↓ & 0.106 & 0.133 & 0.169 & 0.128 & 0.091 & 0.193 & 0.15 & 0.193 \\ \hline \end{tabular}
\end{table}
Table 13: The result of the strength of evidence task with the sampling method. We used 200 samples for Gemini-1.5-Flash due to the cost limit.



Results of Ablation study on the ratio of golden evidence

\begin{table}
\begin{tabular}{c|l|c c c c c c c c c c c c c} \hline  & Dataset & Metric & Coultect, M+ & Conflict, M+ & Conflict, M+ & Conflict, M+ & Recomplete, M+ & Incomplete, M+ & Incomplete, M+ & Generalized, M+ & Centralect, M+ & Counterfield, M+ \\ \hline \multirow{8}{*}{**GPT-3.5***Info**} & \multirow{2}{*}{Ssd} & Call/Change & 0.964 & 0.855 & 0.865 & 0.824 & 0.896 & 0.888 & 0.87 & 0.834 & 0.8122 & 0.8514 \\  & & Accuracy? & 0.822 & 0.77 & 0.786 & 0.865 & 0.831 & 0.777 & 0.71 & 0.899 & 0.835 & 0.835 & 0.852 \\  & & EC:1 & 0.001 & 0.115 & 0.158 & 0.211 & 0.009 & 0.111 & 0.116 & 0.168 & 0.0412 & 0.072 & 0.07 \\ \cline{2-13}  & \multicolumn{1}{c}{Cashboard} & \multicolumn{1}{c}{0.927} & 0.917 & 0.885 & 0.832 & 0.922 & 0.924 & 0.968 & 0.915 & 0.934 & 0.946 & 0.951 \\  & \multicolumn{1}{c}{Tria} & Accuracy? & 0.864 & 0.829 & 0.776 & 0.873 & 0.886 & 0.856 & 0.833 & 0.832 & 0.844 & 0.840 & 0.863 \\ \cline{2-13}  & \multicolumn{1}{c}{Leff} & \multicolumn{1}{c}{0.000} & 0.009 & 0.239 & 0.171 & 0.800 & 0.007 & 0.008 & 0.008 & 0.008 & 0.008 & 0.008 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.239 & 0.171 & 0.800 & 0.009 & 0.008 & 0.008 & 0.008 & 0.008 & 0.008 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.008 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.012 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.012 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.012 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.012 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.012 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.012 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.074 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback} & \multicolumn{1}{c}{0.000} & 0.009 & 0.009 & 0.012 & 0.009 & 0.009 & 0.009 & 0.009 & 0.009 & 0.094 & 0.094 \\ \cline{2-13}  & \multicolumn{1}{c}{Gallback 

Results of Ablation study on irrelevant evidence

Figure 6: The results of ablation for irrelevant evidence. The blue bar represents the result of no evidence \(P(H)\), serving as a baseline. The red bar results from irrelevant evidence by replacing evidence from other samples within the same dataset explained in section 3.2. The green bar represents irrelevant evidence from another dataset.

 

Experimental Detail

### Hyperparameter

We utilized OpenAI's API to create a dataset containing evidence and conducted inference experiments. Specifically, we used GPT-4-0613 to generate Negated evidence, Coincidental evidence, and Contradictory evidence required for the confirmation task, and gpt-4o-2024-05-13 to create evidence necessary for the strength of evidence. The inference was performed using GPT-3.5-0125 and GPT-4o-2024-05-13 with settings of temperature=1.0 and top_p=1.0.

### Evaluation Detail

According to [8], for the SciQ and TriviaQA datasets, we considered a model's response as correct if its Rouge-L score [9] with the golden label is 0.3 or higher. For GSM8K, only responses that were an exact match with the golden label were considered correct.

For sampling method for measuring confidence, we set the ratio of most frequent response as the confidence. As the datasets are open-ended question, we should consider the synonym of each responses. In order to handle this, we used GPT-4o-2024-05-13 to capture the semantic similarity and calculate the frequency of the most common response.

For measuring p-value, we conducted two-sided paired t-tests for p-value calculations. Instead of using individual samples, we performed the tests on dataset-level metrics by comparing the mean metrics (e.g., accuracy) under different evidence conditions (e.g., No Evidence vs. Golden Evidence). Specifically, we calculated the p-value for accuracy by comparing six results (three datasets x two models) for No Evidence with the corresponding six results for Golden Evidence.

### Dataset

For SciQ and GSM8K, we extracted the samples containing the explanation, including more than 4 sentences to create various proportions of negated sentences in the ablation study. Similarly, for trivia QA, we used the explanation2 including more than 4 sentences and extracting 1000 samples. We generated negated sentences using GPT-4-0613 for negated and contradictory evidence and filtered out samples containing incorrect sentences. Similarly, we used GPT-4o-2024-05-13 for generating Strength of Evidence task and also filtered out the generated strength of evidence that included a wrong template. The total number of samples is shown in Table 17 and Table 18. We used all these samples when inferencing with GPT-3.5-turbo and 200 samples for GPT-4o-2024-05-13.

 



\begin{table}
\begin{tabular}{l|c c c c c c c}  & NO\_EVI & EVI & Coincidence & Irrelevant & Negation & Incomplete & Contradiction \\ \hline \hline SciQ & 1095 & 1095 & 1095 & 1095 & 991 & 1095 & 991 \\ TriviaQA & 1000 & 1000 & 1000 & 1000 & 798 & 1000 & 798 \\ GSM8K & 622 & 622 & 622 & 622 & 618 & 622 & 618 \\ \hline \end{tabular}
\end{table}
Table 17: The number of samples for the Confirmation task dataset.

\begin{table}
\begin{tabular}{l|c c c c c c c}  & High Credible Source & Low Credible Source & High Specificity & Low Specificity & Recent & Old & Experiment & Observation \\ \hline SciQ & 1095 & 1095 & 1093 & 1093 & 1074 & 1074 & 1094 & 1094 \\ \hline \end{tabular}
\end{table}
Table 18: The number of samples for the Strength of evidence task dataset.



Prompt

In this section, we will show the prompt for inference,

### Prompt for Inference

[title=VerbitConferencedFrompt] Provide your best guess and the probability that it is correct (0.0 to 1.0) for the following question based on the evidence. Give ONLY the guess and probability, no other words or explanation. For example Guess: <most likely guess, as short as possible; not a complete sentence, just the guess!> Probability: <the probability between 0.0 and 1.0 that your guess is correct based on the given evidence , without any extra commentary whatsoever; just the probability!> **The question**: {question} ***The evidence**: {evidence} \\ \end{tabular}

 



**

\begin{table} \end{tabular}
[FOOTNOTE:*]Footnote ***Example: ”Biochemical reactions of metabolism can be divided into two general categories: catabolic reactions and anabolic reactions. You can watch an animation showing how the two categories of reactions are related at this URL: http://classes. midlandstech. edu/carterp/courses/bio225/chap05/lecture1.htm.” Revise or negate each sentence in the ****Example with incorrect information yet relevant information. The response ###Negation should have same number of sentence with ##Example. ###Negation:"Biochemical reactions of metabolism are typically classified into only one category: equilibrium reactions. You can view a static image illustrating the isolated function of equilibrium reactions at this URL:[http://classes.midlandstech.edu/carterp/courses/bio225/chap05/lecture2.htm](http://classes.midlandstech.edu/carterp/courses/bio225/chap05/lecture2.htm)." ###**Example**: "An anaerobic organism is any organism that does not need oxygen for growth and even dies in its presence. Obligate anaerobes will die when exposed to atmospheric levels of oxygen. Clostridium perfringens bacteria, which are commonly found in soil around the world, are obligate anaerobes. Infection of a wound by C. perfringens bacteria causes the disease gas gangrene. Obligate anaerobes use molecules other than oxygen as terminal electron acceptors." Revise or negate each sentence in the ****Example with incorrect information yet relevant information. The response ###Negation should have same number of sentence with ###Example. ###**Negation**: "An anaerobic organism is any organism that requires oxygen for growth and thrives in its presence. Obligate aerobes will perish when deprived of atmospheric oxygen levels. Staphylococcus aureus bacteria, which are rarely found in aquatic environments, are obligate aerobes. Infection of a wound by S. aureus bacteria causes the disease known as athlete's foot. Obligate aerobes use molecules such as hydrogen or sulfur as terminal electron acceptors." ###**Example**: "The energy of a mechanical wave can travel only through matter. The matter through which the wave travels is called the medium ( plural , media). The medium in the water wave pictured above is water, a liquid. But the medium of a mechanical wave can be any state of matter, even a solid." Revise or negate each sentence in the ****Example with incorrect information yet relevant information. The response ###Negation should have same number of sentence with ****Example. ###**Negation**: "The energy of a mechanical wave can travel through both matter and vacuum. The space through which the wave travels is termed the conduit. The conduit in the water wave pictured above is air, a gas. However, the conduit of a mechanical wave can be exclusively in a gaseous state, not a solid or liquid." ###**Example**: "What group of animals begins its life in the water, but then spends most of its life on land? Amphibians! Amphibians are a group of vertebrates that has adapted to live in both water and on land. Amphibian larvae are born and live in water, and they breathe using gills. The adults live on land for part of the time and breathe both through their skin and with their lungs as their lungs are not sufficient to provide the necessary amount of oxygen." Revise or negate each sentence in the ****Example with incorrect information yet relevant information. The response ****Negation should have same number of sentence with ****Example. ###**Negation**: "What group of animals begins its life on land, but then spends most of its life in water? Reptiles! Reptiles are a group of vertebrates that has adapted to live mainly on land but also in water. Reptile eggs are laid and hatch on land, and they breathe using lungs from birth. The adults live in water for part of the time and breathe exclusively through their lungs as their skin is not permeable enough to facilitate breathing." ###**Example**:{source} Revise or negate each sentence in the ****Example with incorrect information yet relevant information. The response ****Negation should have same number of sentence with ****Example. ###**Negation**: ```

\end{table}
Table 21: A prompt for negating the each sentence in golden evidence.





**

Question: "What does the pull of the moon's gravity on earth cause?", Answer: tides Make explanation that realize the answer to the question by knowing from unreasonable guess. You should not express that answer from unreasonable guess is correct for the question.

**### Explanation**: "Interestingly, while reviewing some casual photographs taken during a lunar eclipse, a curious pattern emerged that reminded me of a physics class discussion. The photos captured how remarkably calm and aligned everything seemed under the moonlight, which brought to mind the effects of the moon's gravity on Earth. It dawned on me, almost by happenstance, that the moon's gravitational pull is what primarily drives the ocean tides. This realization was unexpected, as the focus of the eclipse viewing was purely for its beauty and not the scientific implications. It's fascinating how a moment of leisure led to a reminder that while the moon's gravity tags at the land, only the water, being fluid, visibly responds, creating the tides we observe."

Question**: "What living species of bird is considered the largest of the raptors?" Answer: golden eagles Make explanation that realize the answer to the question by knowing from unreasonable guess. You should not express that answer from unreasonable guess is correct for the question.

Explanation**: "During a casual discussion at a local cafe in Northern California, after a day of hiking, I was sharing stories and showing photos I had taken of a large bird, Golden Eagle, I encountered. Not being an expert, I half-jokingly guessed it might be one of the larger bird species due to its impressive size and wingspan."

Question**: "What is another term for blood clotting?" Answer: coagulation Make explanation that realize the answer to the question by knowing from unreasonable guess. You should not express that answer from unreasonable guess is correct for the question.

Explanation**:"At a community health fair, while casually discussing various health topics, an attendee mentioned his surprise at how quickly a minor cut he had stopped bleeding. Not being a healthcare professional, I playfully suggested it might be due to his body being exceptionally good at 'coagulating'--a term I recalled from a movie rather than a biology class."

Question**: "What is the protein in red blood cells which transports oxygen around the body?", Answer: hemoglobin Make explanation that realize the answer to the question by knowing from unreasonable guess. You should not express that answer from unreasonable guess is correct for the question.

Explanation**: "During a spirited trivia night with friends, where various odd facts were thrown around, someone challenged the group to name the protein in red blood cells responsible for transporting oxygen. Without any medical background, I haphazardly threw out 'hemoglobin' as a guess, mainly because it sounded scientific and related to blood. My guess was influenced by overhearing a conversation earlier that day at the gym where someone mentioned they were taking iron supplements to improve their 'hemoglobin levels' for better endurance"

Question**: {question}, **Answer**: {answer} Make explanation that realize the answer to the question by knowing from unreasonable guess. You should not express that answer from unreasonable guess is correct for the question.

##### Explanation**:

\begin{table} \end{table}
Table 22: A prompt for making coincidental evidence.



* **Homm Co-Making-credit Deviture**

##question: "What substance does the phillosopher stone change the base material to?"

###answer: "gold"

For this ****question, ****answer pairs, make 3 evidences with difference power of evidence in the aspect of Source Credibility.

###Highly Credible Source: "A leading professor of alchemy at a renowned university published a peer-reviewed paper documenting the transmutation of lead into gold using the Philosopher's Stone."

###Moderately Credible Source: "A respected independent alchemist reported successful transmutations in his personal journal."

###Low Credibility Source: "An anonymous blog post claims to have discovered the Philosopher's Stone and successfully converted lead into gold."

###question: "Compounds with aluminum and silicon are commonly found in the clay fractions of soils derived from what?"

###answer: "volcanic ash"

For this ****question, ****answer pairs, make 3 evidences with difference power of evidence in the aspect of Source Credibility.

###Highly Credible Source:"A peer-reviewed study published in the Journal of Soil Science by researchers from a top-tier university provides detailed analysis and evidence that clay fractions in soils derived from volcanic ash predominantly contain compounds of aluminum and silicon."

###Moderately Credible Source:"A detailed report by a well-known geologist in a respected geology magazine discusses the mineral composition of clay fractions in soils and highlights volcanic ash as a common origin of aluminum and silicon compounds."

###Low Credibility Source:"A gardening enthusiast's blog post mentions that soils rich in aluminum and silicon compounds often come from volcanic ash, based on their personal observations and informal tests."

###question: {question}

###answer: {answer}

For this ****question, ****answer pairs, make 3 evidences with difference power of evidence in the aspect of Source Credibility.

\begin{table} \end{table}
Table 23: The prompt for generating various of evidence according to credibility. We did not use moderate credibility evidence, as it is similar to other evidence.

 

* **Attention:** "What substance does the phillosopher stone change the base material to?"
* **answer**: "gold" For this ###question, ###answer pairs, make 3 evidences with difference power of evidence in the aspect of Specificity and detail.
* **Aff Highly Specific Evidence**: "Detailed records from 16th-century experiments show precise measurements and procedures for transmuting lead into gold using a substance identified as the Philosopher's Stone."
* **Aff moderately Specific Evidence**: "Historical documents suggest that some alchemists reported converting metals into gold, but the details are sparse."
* **Conferential Evidence**: "There are general mentions in ancient texts about the ability to convert base metals into gold."
* **Question**: "Compounds with aluminum and silicon are commonly found in the clay fractions of soils derived from what?"
* **fanswer**: "volcanic ash" For this ###question, ###answer pairs, make 3 evidences with difference power of evidence in the aspect of Specificity and detail.
* **Aff Highly Specific Evidence**:"Geochemical analyses of soil samples from regions with known volcanic activity demonstrate that the clay fractions are predominantly composed of alumino-silicate minerals, confirming that these soils are derived from volcanic ash deposits."
* **Moderately Specific Evidence**:"Scientific studies indicate that soils in volcanic regions frequently contain clay fractions rich in aluminum and silicon compounds, which suggests a derivation from volcanic ash."
* **General Evidence**:"Many references in soil science literature mention that clay fractions with aluminum and silicon are often associated with volcanic ash origins."
* **Conclusion**: {question}
* **fanswer**: {answer}

For this ##question, ###answer pairs, make 3 evidences with difference power of evidence in the aspect of Specificity and detail.

\begin{table} \end{table}
Table 24: The prompt for generating various evidence according to specificity. We did not use moderate specific evidence, as it is similar to other evidence 

* **Question**: "What substance does the philosopher stone change the base material to?"
* **answer**: "gold" For this ***question, ****answer pairs, make 2 evidences with difference power of evidence in the aspect of timeliness. (the older evidence should be before 18th-century)
* **Result**: "A 2022 study published in a scientific journal provides new experimental data supporting the possibility of metal transmutation using a newly synthesized substance resembling the Philosopher's Stone."
* **Result**: "A 17th-century manuscript claims to have witnessed the transformation of base metals into gold using an alchemical process."
* **Result**: "Compounds with aluminum and silicon are commonly found in the clay fractions of soils derived from what?"
* **Result**: "volcanic ash" For this ****question, ****answer pairs, make 2 evidences with difference power of evidence in the aspect of timeliness. (the older evidence should be before 18th-century)
* **Result**: "A 2019 study published in a geochemistry journal confirms that soils derived from volcanic ash predominantly contain clay fractions with high concentrations of aluminum and silicon compounds."
* **Result**Older Evidence**: "A 16th-century agricultural text describes soils from regions with volcanic activity as rich in aluminosilicate clays, derived from the weathering of volcanic ash."
* **Result**: {question}
* **Result**Answer**: {answer}

For this ****question, ****answer pairs, make 2 evidences with difference power of evidence in the aspect of timeliness. (the older evidence should be before 18th-century)

\begin{table} \end{table}
Table 25: The prompt for generating various evidence according to timeliness.

 

* **Question**: "What substance does the philosopher stone change the base material to?"
* **answer**: "gold" For this ***question, ****answer pairs, make 2 evidences with different levels of strength in the aspect of Experimental or Observational Evidence, ensuring that the observational evidence includes direct observations from normal people such as "several witnesses observed."
* **Experimental Evidence**: "Recent laboratory experiments conducted under controlled conditions have demonstrated the conversion of lead into gold using a synthetic version of the Philosopher's Stone."
* **##Observational Evidence**: "Several eyewitness accounts from the 1600s describe seeing alchemists successfully convert metals into gold, though these were not scientifically verified."
* **suggestion**: "Compounds with aluminum and silicon are commonly found in the clay fractions of soils derived from what?"
* **answer**: "volcanic ash"

For this ****question, ****answer pairs, make 2 evidences with different levels of strength in the aspect of Experimental or Observational Evidence, ensuring that the observational evidence includes direct observations from normal people such as "several witnesses observed."

\begin{table} \end{table}
Table 26: The prompt for generating various evidence according to the existence of the experiment.