Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models

## Abstract

Abstract There has been much recent interest in evaluating large language models for uncertainty calibration to facilitate model control and modulate user trust. Inference time uncertainty, which may provide a real-time signal to the model or external control modules, is particularly important for applying these concepts to improve LLM-user experience in practice. While many of the existing papers consider model calibration, comparatively little work has sought to evaluate how closely model uncertainty aligns to human uncertainty. In this work, we evaluate a collection of inference-time uncertainty measures, using both established metrics and novel variations, to determine how closely they align with both human group-level uncertainty and traditional notions of model calibration. We find that numerous measures show evidence of strong alignment to human uncertainty, even despite the lack of alignment to human answer preference.
For those successful metrics, we find moderate to strong evidence of model calibration in terms of both correctness correlation and distributional analysis.

## Introduction

Figure: Figure 1: LLMs and humans have similar uncertainty at inference time and frequently agree on overt selection. However, their cloze probability-based preference orderings are not aligned.
Refer to caption: Images/this.png

A sizeable body of work has developed around the identification and quantification of uncertainty in the outputs of transformer-based large language models (LLMs). Accurate uncertainty quantification (UQ) is an essential element in predicting model hallucinations and maintaining user trust. In service of that, UQ research has largely focused on developing and utilizing uncertainty measurement methods that are well-calibrated to model accuracy. A well calibrated measure is one that predicts well the model’s likelihood of generating a valid answer to the given context. Contexts with high certainty should have a low likelihood of being incorrect and vice versa. A subset of UQ work focuses on measures that are able to be calculated at any time during generation, without additional auxiliary generations. This has often been referred to as inference-time uncertainty quantification. Inference-time measures are uniquely useful in that they can provide a constant signal to the user or to external control modules without significant added computation.

Existing research has not considered whether the investigated UQ measures align with human uncertainty. So, while research has investigated measures with significant calibration, the reported values may not correspond with human uncertainty, making the meaning of the values difficult to parse for users.

Simultaneously, a growing body of research has emerged that seeks to identify human-like behaviors in a variety of LLM tasks and contexts. This has included behaviors as varied as theory of mind (Ullman 2023; Amirizaniani et al. 2024; Strachan et al. 2024), strategic preferences (Roberts, Moore, and Fisher 2024; Duan et al. 2024), and framing effects (Jumelet, Zuidema, and Sinclair 2024; Nguyen 2024). This work seeks to synthesize these two research thrusts by identifying uncertainty measures that are simultaneously calibrated and aligned to human uncertainty behavior. In particular, we investigate whether any of the uncertainty measures vary consistently with human uncertainty on a per-question basis. Given the difficulty in reliably quantifying uncertainty for an individual human, we approximate this by comparing model measures against disagreement among groups of human survey respondents.

By evaluating LLM uncertainty alignment as well as calibration in inference-time UQ measures, this paper identifies a set of UQ measures which may be effective and more intuitively interpretable by human users—enabling important advances in LLM-human interaction and prompting the further study of alignment in model signals beyond overt action. This paper specifically contributes to the existing understanding of LLM uncertainty by:

- Noting that top-p selection in LLM decoding is functionally equivalent to the Bayesian highest density credible set, drawing an important but previously un-noted connection between the fields and inspiring our investigation of top-p as a measure of uncertainty.
- Observing that many entropy-based inference-time uncertainty measures have significant human uncertainty alignment despite moderate choice selection and no preference ordering alignment.
- Developing a novel ground-truth distributional calibration measure based on shift in the Jensen-Shannon distance metric to directly evaluate the impact of certainty on answer distribution.
- Showing that the aligned inference-time measures show evidence of calibration in terms of both correctness correlation and ground-truth distributional calibration.

## Prior Work

Uncertainty Quantification in LLMs is a broad field with numerous notions of uncertainty depending on context, task, and available resources. These are covered in a variety of surveys including Liu et al. (2025); Shorinwa et al. (2025); He et al. (2025). Many of the most successful methods in terms of calibration, like monte-carlo dropout (Shelmanov et al. 2021; Roberts et al. 2024b), rely on multiple generation steps to quantify uncertainty and cannot be readily adapted to quantify per-token inference-time uncertainty levels. Existing work on inference-time uncertainty quantification typically relies on perplexity (Mora-Cross and Calderon-Ramirez 2024; Margatina et al. 2023; Jiang et al. 2021), maximum token probability (Tian et al. 2023; Steyvers et al. 2025; Huang et al. 2025; Shrivastava, Liang, and Kumar 2023), or entropy methods (Kadavath et al. 2022; Huang et al. 2025).

Very few works have explicitly investigated the presence of human-like uncertainty responses in LLMs. This work was inspired by preliminary work by Moore et al. (2025), which investigated human-similarity on a diverse set of uncertainty measures. Their work was limited in that the dataset consisted of less than 40 items and did not consider whether the measures which were aligned were also calibrated. We expand on those results by drastically expanding the dataset size and narrow our focus exclusively to inference-time measures. Our work is also novel in that it is the only extant work, to our knowledge, that simultaneously evaluates any uncertainty measures for both alignment and calibration. Other related work, including Argyle et al. (2023); Huang, Wu, and Wang (2025), have used LLMs and uncertainty-aware procedures to simulate human group responses, but do not seek to establish human-like uncertainty measures.

## Inference-Time UQ

This work is interested in investigating inference-time UQ methods, as inference-time calculation is necessary for model control mechanisms to have a signal to interpret and react. Non-inference-time methods are valuable tools for diagnostics, model comparison, etc., but are ill-suited for time-sensitive application. While the field of LLM UQ research is large and growing, it has focused on measures of uncertainty that have limited or no inference-time capabilities. These commonly include intuitive methods like self-reporting (Zhou, Jurafsky, and Hashimoto 2023; Mielke et al. 2022; Band et al. 2024; Lin, Hilton, and Evans 2022; Tang, Shen, and Kejriwal 2024; Chaudhry, Thiagarajan, and Gorur 2024; Shrivastava, Liang, and Kumar 2023; Tian et al. 2023; Xiong et al. 2023; Belém et al. 2024), multi-inference consistency (Lin, Hilton, and Evans 2022; Kadavath et al. 2022; Chen and Mueller 2024; Manakul, Liusie, and Gales 2023; Zhang et al. 2024), and ensemble variation (Wang, Aitchison, and Rudolph 2023; Roberts et al. 2024b; Gal and Ghahramani 2016; Fomicheva et al. 2020). The remainder of this section will describe the inference-time UQ measures employed here. In all cases, these methods are calculated using the token probability distribution over the vocabulary $V$ given some context $c$, $P(v\in V|c)$.

The simplest inference-time UQ measures rely on relative probabilities of the most probable output token (Jiang et al. 2021; Shrivastava, Liang, and Kumar 2023; Tian et al. 2023). In this work, we refer to this simple approach as the top-1 probability. Prior work typically does not promote this as a UQ measure, instead utilizing it as a classifier feature (Jiang et al. 2021) or as a basis of comparison (Shrivastava, Liang, and Kumar 2023; Tian et al. 2023).

The majority of our candidate measures are entropy-based measures. These measures are based on the Shannon entropy over a probability distribution, $S(P(X))=-\sum_{x\in X}P(x)\log(P(x))$. Higher entropy distributions are taken to be indicative of higher uncertainty because entropy increases as the relative probabilities throughout the full distribution approach uniform. Typically, this is measured across the entire probability distribution, which we herein refer to as the total entropy for disambiguity.

**Inference-Time Model Behavior and Uncertainty Alignment**
| Model | LLaMa-3.2<br>1B | LLaMa-3.2<br>1B Ins | LLaMa-3.2<br>3B | LLaMa-3.2<br>3B Ins | Mistral 0.1<br>7B | Mistral 0.1<br>7B Ins | Mistral 0.3<br>7B | Mistral 0.3<br>7B Ins | LLaMa-3.1<br>8B | LLaMa-3.1<br>8B Ins |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Top Answer<br>Agreement | 0.271 | 0.313 | 0.319 | 0.372 | 0.346 | 0.360 | 0.350 | 0.392 | 0.362 | 0.427 |
| Norm. Kendall $\tau$<br>Distance Mean | 0.486 | 0.446 | 0.511 | 0.484 | 0.457 | 0.496 | 0.463 | 0.441 | 0.486 | 0.477 |
| Norm. Kendall $\tau$<br>Distance Std. | 0.339 | 0.350 | 0.337 | 0.340 | 0.349 | 0.338 | 0.349 | 0.346 | 0.343 | 0.343 |

We further experiment on entropy calculated over a variety of normalized subsets of the total probability distribution. The simplest method of obtaining this subset is using top-k sampling, in which the $k$ highest probability tokens are extracted from the total probability distribution. We normalize this subset, $V^{k}$ by dividing every token probability by the sum of the entire subset. We choose this over softmax because softmax does not necessarily maintain the relative ratio between individual probabilities, which can drastically affect the entropy calculations. We investigate this measure for five values of $k$: $5$, $10$, $25$, $50$, and $100$. Note that the total entropy is a special case of top-k entropy where $k=|V|$. Because our datasets are exclusively multiple choice format, as are many common benchmarks, we also measure the uncertainty as the entropy over the normalized probabilities of the target tokens, corresponding to the first $n$ letters of the capitalized alphabet, where $n$ is the number of provided answer choices. We call this measure the choice entropy.

We also investigate the other common sampling method, top-p sampling. Top-p sampling, also known as nucleus sampling, extracts the most probable tokens such that the cumulative probability of the extracted tokens is maximized and less than $p$ and the number of tokens extracted is minimized (Holtzman et al. 2019). Despite its apparent similarity to the highest density credible sets commonly found in Bayesian notions of uncertainty, few prior works have investigated it’s viability for LLM UQ. As in Moore et al. (2025), we investigate the size of the resulting token set as a measure of uncertainty. We extend that work by investigating more values for $p$: $0.95$, $0.9$, $0.75$ and $0.5$. We also include in our study the entropy over the normalized probabilities of the top-p tokens.

## Human UQ Alignment

Alignment refers to how closely the behavior of an AI system conforms to the desired behavior of the user or developer. While this is most commonly discussed in terms of how well the models interpret and conform to instructions or moral imperatives, it can also refer to how closely the model’s behavior matches human behavior in some context. We define herein uncertainty alignment using the latter notion. That is, an uncertainty measure is aligned for a given model if the uncertainty measure correlates well with uncertainty in humans. In this work, we focus on the easier task of correlating with uncertainty among groups of human subjects, as defined by the level of agreement on multiple choice surveys, rather than attempting to measure correlation at the individual level, though this should be explored in future work.

### Dataset

We use two datasets to investigate UQ alignment. The first is the dataset used in the inspiring work, Moore et al. (2025). This dataset is comprised of $38$ manually collected and formatted questions originally sourced from Pew Research surveys (Pew Research Center 2025). This dataset is clearly limited in size and thus diversity, but provides a useful baseline for comparison and is included for completeness.

The second dataset is a collection of 2998 randomly selected questions obtained from the Roper Center for Public Opinion Research database (2025). The exact methods used to sample from this database are detailed in the appendix. All questions were obtained from human surveys performed during the years 2017-2023. Minor keyword filtering was employed to reduce the number of time-sensitive and personal experience questions. A total of 30571 questions were initially retrieved from Roper. We removed questions with invalid response ratios and sampled 3000 questions from the resulting set uniformly at random without replacement. Each question’s answer choices were shuffled to reduce ordering biases, but every model was presented with the same answer choice ordering for consistency. After removal of two additional questions for invalid answer choice counts, the final dataset was comprised of 2998 questions with an average of 3.77 choices per question.

Figure: Figure 2: Prompt templates for querying model answer preference. We query the model for the full vocabulary probability distribution, from which we extract the chosen answer and the various uncertainty metrics.

Figure: Inference-Time Uncertainty Quantification Correlation with Human Uncertainty
Refer to caption: Images/align_corr_corrected.png

### Methods

Each model was queried for the full token probability distribution using a consistent prompt template, described in Figure [2](https://arxiv.org/html/2508.08204v1#Sx4.F2). In addition, we use the standard cloze test to determine the model’s chosen answer from the options provided. We extract from the probability distributions each of the uncertainty measures described above in the inference-time uncertainty section. Human group uncertainty is obtained by taking the entropy over the response percentages after normalization.

Across all questions, we measure human agreement in three ways. First, we measure overt agreement based on the response ratios and cloze test results to show how often the models and humans agreed on the best answer. For a more fine-grained analysis, we measure the relative preferential alignment between model and human using the normalized Kendall $\tau$ distance (Kendall 1938). This measures the minimum number of pairwise swaps needed to convert the model preference order into the human preference order, normalized by the maximum possible distance. Finally, we measure the correlation across all questions between the human group uncertainty and each of the model uncertainty measures.

### Results

Table [1](https://arxiv.org/html/2508.08204v1#Sx3.T1) displays the results of the overt agreement and order preference analysis. For overt agreement, models show mild agreement, with all models other than Llama-3.2 1B significantly ($p<0.01$) beating the theoretically determined random chance ($\sim 0.265$) based on a one-sided one-proportion z test (Moore, McCabe, and Craig 2016). The test is appropriate as only one proportion is used given the precise random proportion is calculated and the proportion is from a binomial distribution.

The relational analysis shows remarkably little agreement between model and human. Every model shows a consistent mean distance of $\mu=0.476\pm 0.035$ and standard deviation $\sigma=0.343\pm 0.006$, indicating effectively random and widely distributed distance scores. Together, these indicate that the models show moderate agreement with humans on the top token, but not on overall token preference ordering. This finding is at odds with previous work on strategic preference ordering in LLMs (Roberts et al. 2024a), with the primary difference in our approaches being the prompt design and output capture approach. The prior work used a prompting strategy dubbed counterfactual prompting as an alternative to the more common cloze testing used here. In the prior, the output is measured using a consistent canary token whose probability is queried once for each option while in the latter a set of options is presented once and the relative probabilities associated with each option is taken as the relative preference. This might suggest that model alignment is strongly sensitive to specific prompting methodology and preference interpretation. The lack of preference ordering alignment could additionally be explained by the fact that models are trained only on individual target tokens and have no information about what tokens would have been valid but less preferred tokens.

Figure: Inference-Time Uncertainty Quantification Calibration on MMLU
Refer to caption: Images/tmp_heatmap.png

The results for uncertainty alignment, shown in Figure [3](https://arxiv.org/html/2508.08204v1#Sx4.F3), are much more promising. As in the prior work, we see wild variation in correlation across models for every measure. Even still, every model shows significant ($|r|\geq 0.3$) correlation for all top-k measures, including total entropy. The same is true of choice entropy and top-p entropy for all $p\geq 0.75$, though with progressively more instability as $p$ decreases. Top-p size, counter to previous studies, shows weak but significant correlation for all $p\leq 0.9$. The only measures that do not show consistent significant correlation are top-1 probability, top-p ($0.95$) size and top-p ($0.5$) size. It is noteworthy, but currently unexplained that degradation trends are reversed between top-p size and entropy. The top performing measures are identified as those whose correlation exceeds $r\geq 0.5$ for every model. This set includes choice entropy, total entropy, all top-k entropies, and top-p entropy for $p\geq 0.9$. These models are further evaluated for calibration. It should be noted that all of the top performing measures are significant for all models on both datasets, with the exception of those based on top-p entropy.

## Calibration

Calibration is the standard measure by which uncertainty measures are evaluated for LLMs. It refers to the measure’s utility in predicting the model’s likelihood to correctly complete some task. A well-calibrated uncertainty measure should be low when the model is highly likely to answer correctly and it should be high when the model’s likelihood to answer correctly is low or near random chance. We measure calibration on the common mulitple choice question answering benchmark, MMLU (Hendrycks et al. 2020). Note that, unlike the Roper and Pew datasets, each MMLU question always has a constant four available answer choices.

**JSD Uncertainty Distribution Shift for All Models and Measures**
| Measure | LLaMa<br>1B | LLaMa<br>1B (I) | LLaMa<br>3B | LLaMa<br>3B (I) | Mistral<br>7B 0.1 | Mistral<br>7B 0.1 (I) | Mistral<br>7B 0.3 | Mistral<br>7B 0.3 (I) | LLaMa<br>8B | LLaMa<br>8B (I) |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | JSDS | 0.055 | 0.093 | 0.150 | 0.031 | 0.098 | 0.060 | 0.075 | 0.091 | 0.217 | 0.098 |
| choice | p | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|  | JSDS | 0.034 | 0.017 | 0.040 | 0.020 | 0.018 | -0.122 | 0.023 | 0.085 | 0.144 | 0.082 |
| total | p | 0.000 | 0.009 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|  | JSDS | 0.055 | 0.149 | 0.094 | 0.043 | 0.051 | -0.127 | 0.026 | 0.088 | 0.052 | 0.108 |
| top-k<br>5 | p | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|  | JSDS | 0.069 | 0.148 | 0.126 | 0.040 | 0.038 | -0.126 | 0.023 | 0.085 | 0.142 | 0.097 |
| top-k<br>10 | p | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|  | JSDS | 0.061 | 0.112 | 0.099 | 0.035 | 0.033 | -0.125 | 0.022 | 0.083 | 0.158 | 0.091 |
| top-k<br>25 | p | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|  | JSDS | 0.053 | 0.082 | 0.071 | 0.031 | 0.029 | -0.124 | 0.023 | 0.084 | 0.154 | 0.086 |
| top-k<br>50 | p | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|  | JSDS | 0.046 | 0.060 | 0.055 | 0.028 | 0.026 | -0.123 | 0.023 | 0.083 | 0.148 | 0.084 |
| top-k<br>100 | p | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|  | JSDS | 0.016 | 0.112 | 0.076 | 0.046 | 0.082 | -0.099 | 0.041 | 0.063 | 0.160 | 0.061 |
| top-p<br>0.95 | p | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|  | JSDS | 0.132 | 0.000 | 0.118 | 0.038 | 0.087 | 0.052 | 0.032 | 0.022 | 0.089 | 0.062 |
| top-p<br>0.90 | p | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 |

### Methods

Similar to alignment evaluation, each of the MMLU questions are presented to the model using the same prompt template in Figure [2](https://arxiv.org/html/2508.08204v1#Sx4.F2). For each question, the full token probability distribution and cloze test results are recorded. Analysis is split into two phases. In the first phase, a simple measure of calibration is obtained by taking the Spearman correlation between the binary correctness of the cloze test result and the candidate uncertainty measures. Because results can vary wildly within a single model across the various question subjects, we separate by subject during analysis.

#### Jensen-Shannon Distance Shift

We provide a more nuanced analysis of the calibration using the shift in Jensen-Shannon distance. The Jensen-Shannon distance (JSD) is a symmetric and finitely-valued extension of the Kullback-Leibler divergence. It is defined as $JSD(P||Q)=\sqrt{\frac{1}{2}D(P||M)+\frac{1}{2}D(Q||M)}$, where $D$ is the K-L divergence function and $M$ is a mixture distribution of $P$ and $Q$. As JSD is a metric, and thus obeys the triangle inequality, we can use it to directly compare the relative distance between two separate pairs of probability distributions over a shared outcome space (Osán, Bussandri, and Lamberti 2018). That is, if $JSD(P,P^{\prime})<JSD(Q,Q^{\prime})$ and if $P$, $P^{\prime}$, $Q$, and $Q^{\prime}$ represent probability distributions over an identical outcome space, this indicates that $P$ and $P^{\prime}$ are more similar to each other than $Q$ is to $Q^{\prime}$. This metric thus provides a similar, but more nuanced, view of the model accuracy, but allows for robust hypothesis testing. Our hypothesis is that a well-calibrated measure should show a significant change in JSD from the correct distribution when the model goes from certain to uncertain.

We leverage the JSD to examine how closely the distribution of answers given by the model matches the distribution of correct answers at high and low entropy. For each uncertainty measure, we standardize the measured certainties to have a mean of 0 and standard deviation of $1$. We assign the questions to high or low certainty by whether the standardized uncertainty is above or below 0. Questions with a standardized uncertainty of 0 were randomly assigned. From this assignment, we obtain four distributions, $H_{A}$, $H_{M}$, $L_{A}$, and $L_{M}$. $H_{A}$ is the count of each correct answer choice for the questions with high certainty, while $H_{M}$ is the distribution of answers given by the model for those same questions. $L_{A}$ and $L_{M}$ are defined similarly. Finally, we use a permutation test with random uncertainty level assignments to test whether $JSD(H_{M},H_{A})>JSD(L_{M},L_{A})$ to a significant degree. We dub the resulting calibration metric, $JSD(H_{M},H_{A})-JSD(L_{M},L_{A})$, as the JSD shift. We run the permutation test with $1000$ permutation iterations.

### Results

The results of the simple correlation analysis is shown as a heatmap in Figure [4](https://arxiv.org/html/2508.08204v1#Sx4.F4). The ideal measure would appear as bright blue (negative correlation) for all models and subjects. Unlike the alignment case, there is a clear winner in the choice entropy. Across nearly all models and subjects, choice entropy shows mild to moderate correlation with correctness. The primary exceptions are LLaMa 1B, which shows low correlation on all subjects, and a small collection of subjects that show no correlation for any model. In both cases, these are likely indicative of poor performance due to underpowered model or excessive question difficulty.

Outside of choice entropy, all models, with the notable exception of Mistral 0.1 7B Instruct, show negative correlation in most subjects, in particular for top-$k$ entropy. Qualitatively, it appears that there is a non-linear relationship between calibration and size of $k$. The calibration appears to peak at $k=10$ and degrades as $k$ increases. Mistral 0.1 7B Instruct is noteworthy in showing weak positive correlation for all measures except choice entropy.

Figure: Permutation Test of JSD Uncertainty Distribution Shift
Refer to caption: Images/JSD-llama3.2-1b-instruct.png

The results of the permutation test on the JSD shift test for the LLaMa 3.1 8B Instruct model is depicted in Figure [5](https://arxiv.org/html/2508.08204v1#Sx5.F5), with the JSD shift and significance for every model-measure pair listed in Table [2](https://arxiv.org/html/2508.08204v1#Sx5.T2). This figure is typical of all models except Mistral 0.1 7B Instruct. Each dotted vertical line is one uncertainty measure’s JSD shift. For all models, including Mistral 0.1 7B Instruct, all measures show clear significance, with at most one measure per model visually intersecting the JSD shift distribution for random assignment. As is apparent in Figure [4](https://arxiv.org/html/2508.08204v1#Sx4.F4) and reinforced in Table [2](https://arxiv.org/html/2508.08204v1#Sx5.T2), Mistral 0.1 7B Instruct shows significant but negative JSD shift for nearly every measure, suggesting that the model itself is unusually negatively calibrated. Permutation graphs for all models can be found in the Appendix.

## Conclusion

In this work, we found strong initial evidence that many entropy-based uncertainty measures are well-aligned to human uncertainty. This is in spite evidence that the human groups and models rarely agree, both in chosen answer and in answer preference ordering. We identify nine uncertainty measures that show especially strong correlation with human group uncertainty: choice entropy, top-k entropy (for numerous values of k), and top-p entropy (for high values of p). We found that these candidate measures show weak but statistically significant calibration on the MMLU benchmark.

#### Future Work

Future work should seek to find measures that are more highly calibrated without sacrificing alignment. Additional fruitful lines of research could include extending the current research into open-ended contexts—including measures similar and dissimilar to the proposed framework below—and investigating whether uncertainty-aware applications based in human-aligned measures show benefits to user experience or task efficacy by directly conducting human studies. Our work also did not investigate a direct relationship between uncertainty and human-LLM answer agreement, in particular whether answer agreement correlates with uncertainty level. Future work should investigate this relationship to a finer degree. Finally, future work should seek to measure uncertainty alignment at the individual level as well as at the human group level.

The most important area of future work is the introduction of highly calibrated and aligned measures—with top-k 10 being the most promising—into agentic and cooperative software to provide users with an intuitive understanding of model confidence. We hope that such a method will increase trust and improve outcomes in human-LLM cooperation contexts.

#### A Proposed Conceptual Framework for Open Ended Questions

This paper investigated and found human aligned uncertainty measures in a multiple choice context. However, the measures and approach used here can be extended to open answer generation contexts. Consider a language model asked an open ended question. The model generates a response, $R$. The model context including $R$ is then extended with a prompt to the effect, ”Evaluate if the answer given is a true, false, or neutral response to the the question”. This prompt converts the open ended question response uncertainty problem into a multiple-choice, 3 answer choice question—similar to the 3.77 answers per question on average in the dataset used to establish alignment. The measures evaluated in this paper are then directly applicable with little inferential cost as the precomputed token values do not need to be re-calculated as is necessary in other UQ measure approaches discussed in the related work section. This framework is planned to be evaluated in future work.

## Limitations

The most significant limitation to this study is that the experiments were limited to only multiple-choice question contexts. Further research is needed to determine whether the results herein will persist in more open ended contexts. Our work is also limited to a selection of widely used open-weight models with 8 billion or fewer parameters. We are prevented by available compute resources from extending our experiments to larger models, but our results do not show any apparent size dependence for the most aligned and calibrated measures.

While we did not find a model size dependence in top-k measures, this should be evaluated in significantly larger models to determine the relevance to consumer grade large language models.

## Technical Details

All experiments were performed using the computing resources at Tennessee Technological University. The high-performance computer was used, leveraging one A100 GPU with 40GB of VRAM. The code was developed in a Python 3 Jupyter environment using the huggingface toolbox (Lhoest et al. 2021).

Experiments were performed on both the completion and instruct fine-tuned versions of the following models: LLaMa-3.2 1B and 3B (AI 2024), LLaMa 3.1 8B (Grattafiori et al. 2024), and Mistral 7B versions 0.1 and 0.3 (Jiang et al. 2023). All experimentation and analysis source code has been released under the MIT license and is publicly available at https://github.com/KyleAMoore/LLM-UQ-Align-and-Calibrate.

The Pew (Pew Research Center 2025) and Roper (Roper Center for Public Opinion Research 2025) datasets are copyrighted and were accessed through the official portals. Therefore, we are unable to independently release the dataset. Our Roper data collection process can be replicated using details included in the appendix.

## References

- AI (2024)
AI, M. 2024.
Llama 3.2: Revolutionizing edge AI and vision with open, customizable models.
- Amirizaniani et al. (2024)
Amirizaniani, M.; Martin, E.; Sivachenko, M.; Mashhadi, A.; and Shah, C. 2024.
Can llms reason like humans? assessing theory of mind reasoning in llms for open-ended questions.
In *Proceedings of the 33rd ACM International Conference on Information and Knowledge Management*, 34–44.
- Arditi et al. (2024)
Arditi, A.; Obeso, O.; Syed, A.; Paleka, D.; Panickssery, N.; Gurnee, W.; and Nanda, N. 2024.
Refusal in language models is mediated by a single direction.
*Advances in Neural Information Processing Systems*, 37: 136037–136083.
- Argyle et al. (2023)
Argyle, L. P.; Busby, E. C.; Fulda, N.; Gubler, J. R.; Rytting, C.; and Wingate, D. 2023.
Out of one, many: Using language models to simulate human samples.
*Political Analysis*, 31(3): 337–351.
- Band et al. (2024)
Band, N.; Li, X.; Ma, T.; and Hashimoto, T. 2024.
Linguistic calibration of long-form generations.
In *Proceedings of the 41st International Conference on Machine Learning*, 2732–2778.
- Belém et al. (2024)
Belém, C.; Kelly, M.; Steyvers, M.; Singh, S.; and Smyth, P. 2024.
Perceptions of Linguistic Uncertainty by Language Models and Humans.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 8467–8502.
- Chaudhry, Thiagarajan, and Gorur (2024)
Chaudhry, A.; Thiagarajan, S.; and Gorur, D. 2024.
Finetuning language models to emit linguistic expressions of uncertainty.
*arXiv preprint arXiv:2409.12180*.
- Chen and Mueller (2024)
Chen, J.; and Mueller, J. 2024.
Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, 5186–5200.
- Duan et al. (2024)
Duan, J.; Zhang, R.; Diffenderfer, J.; Kailkhura, B.; Sun, L.; Stengel-Eskin, E.; Bansal, M.; Chen, T.; and Xu, K. 2024.
Gtbench: Uncovering the strategic reasoning capabilities of llms via game-theoretic evaluations.
*Advances in Neural Information Processing Systems*, 37: 28219–28253.
- Fomicheva et al. (2020)
Fomicheva, M.; Sun, S.; Yankovskaya, L.; Blain, F.; Guzmán, F.; Fishel, M.; Aletras, N.; Chaudhary, V.; and Specia, L. 2020.
Unsupervised quality estimation for neural machine translation.
*Transactions of the Association for Computational Linguistics*, 8: 539–555.
- Gal and Ghahramani (2016)
Gal, Y.; and Ghahramani, Z. 2016.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning.
In *international conference on machine learning*, 1050–1059. PMLR.
- Grattafiori et al. (2024)
Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*.
- He et al. (2025)
He, J.; Yu, L.; Li, C.; Yang, R.; Chen, F.; Li, K.; Zhang, M.; Lei, S.; Zhang, X.; Beigi, M.; et al. 2025.
Survey of Uncertainty Estimation in Large Language Models-Sources, Methods, Applications, and Challenge.
*HAL Open Science*.
- Hendrycks et al. (2020)
Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020.
Measuring massive multitask language understanding.
*arXiv preprint arXiv:2009.03300*.
- Holtzman et al. (2019)
Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; and Choi, Y. 2019.
The curious case of neural text degeneration.
*arXiv preprint arXiv:1904.09751*.
- Huang, Wu, and Wang (2025)
Huang, C.; Wu, Y.; and Wang, K. 2025.
Uncertainty Quantification for LLM-Based Survey Simulations.
*arXiv preprint arXiv:2502.17773*.
- Huang et al. (2025)
Huang, Y.; Song, J.; Wang, Z.; Zhao, S.; Chen, H.; Juefei-Xu, F.; and Ma, L. 2025.
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models.
*IEEE Transactions on Software Engineering*.
- Jiang et al. (2023)
Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; Lavaud, L. R.; Lachaux, M.-A.; Stock, P.; Scao, T. L.; Lavril, T.; Wang, T.; Lacroix, T.; and Sayed, W. E. 2023.
Mistral 7B.
arXiv:2310.06825.
- Jiang et al. (2021)
Jiang, Z.; Araki, J.; Ding, H.; and Neubig, G. 2021.
How can we know when language models know? on the calibration of language models for question answering.
*Transactions of the Association for Computational Linguistics*, 9: 962–977.
- Jumelet, Zuidema, and Sinclair (2024)
Jumelet, J.; Zuidema, W.; and Sinclair, A. 2024.
Do Language Models Exhibit Human-like Structural Priming Effects?
In *Findings of the Association for Computational Linguistics ACL 2024*, 14727–14742.
- Kadavath et al. (2022)
Kadavath, S.; Conerly, T.; Askell, A.; Henighan, T.; Drain, D.; Perez, E.; Schiefer, N.; Hatfield-Dodds, Z.; DasSarma, N.; Tran-Johnson, E.; et al. 2022.
Language models (mostly) know what they know.
*arXiv preprint arXiv:2207.05221*.
- Kendall (1938)
Kendall, M. G. 1938.
A new measure of rank correlation.
*Biometrika*, 30(1-2): 81–93.
- Lhoest et al. (2021)
Lhoest, Q.; del Moral, A. V.; von Platen, P.; Wolf, T.; Šaško, M.; Jernite, Y.; Thakur, A.; Tunstall, L.; Patil, S.; Drame, M.; Chaumond, J.; Plu, J.; Davison, J.; Brandeis, S.; Sanh, V.; Scao, T. L.; Xu, K. C.; Patry, N.; Liu, S.; McMillan-Major, A.; Schmid, P.; Gugger, S.; Raw, N.; Lesage, S.; Lozhkov, A.; Carrigan, M.; Matussière, T.; von Werra, L.; Debut, L.; Bekman, S.; and Delangue, C. 2021.
Datasets: A Community Library for Natural Language Processing.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, 175–184. Association for Computational Linguistics.
- Lin, Hilton, and Evans (2022)
Lin, S.; Hilton, J.; and Evans, O. 2022.
Teaching models to express their uncertainty in words.
*arXiv preprint arXiv:2205.14334*.
- Liu et al. (2025)
Liu, X.; Chen, T.; Da, L.; Chen, C.; Lin, Z.; and Wei, H. 2025.
Uncertainty quantification and confidence calibration in large language models: A survey.
*arXiv preprint arXiv:2503.15850*.
- Manakul, Liusie, and Gales (2023)
Manakul, P.; Liusie, A.; and Gales, M. 2023.
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 9004–9017.
- Margatina et al. (2023)
Margatina, K.; Schick, T.; Aletras, N.; and Dwivedi-Yu, J. 2023.
Active Learning Principles for In-Context Learning with Large Language Models.
In *Findings of the Association for Computational Linguistics: EMNLP 2023*, 5011–5034.
- Mielke et al. (2022)
Mielke, S. J.; Szlam, A.; Dinan, E.; and Boureau, Y.-L. 2022.
Reducing conversational agents’ overconfidence through linguistic calibration.
*Transactions of the Association for Computational Linguistics*, 10: 857–872.
- Moore, McCabe, and Craig (2016)
Moore, D. S.; McCabe, G. P.; and Craig, B. A. 2016.
*Introduction to the Practice of Statistics*, volume 9.
WH Freeman New York.
- Moore et al. (2025)
Moore, K.; Roberts, J.; Watson, D.; and Wisniewski, P. 2025.
Investigating Human-Aligned Large Language Model Uncertainty.
*arXiv preprint arXiv:2503.12528*.
- Mora-Cross and Calderon-Ramirez (2024)
Mora-Cross, M.; and Calderon-Ramirez, S. 2024.
Uncertainty estimation in large language models to support biodiversity conservation.
In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track)*, 368–378.
- Nguyen (2024)
Nguyen, J. K. 2024.
Human bias in AI models? Anchoring effects and mitigation strategies in large language models.
*Journal of Behavioral and Experimental Finance*, 43: 100971.
- Osán, Bussandri, and Lamberti (2018)
Osán, T. M.; Bussandri, D. G.; and Lamberti, P. W. 2018.
Monoparametric family of metrics derived from classical Jensen–Shannon divergence.
*Physica A: Statistical Mechanics and its Applications*, 495: 336–344.
- Pew Research Center (2025)
Pew Research Center. 2025.
The American Trends Panel.
Data set.
- Roberts, Moore, and Fisher (2024)
Roberts, J.; Moore, K.; and Fisher, D. 2024.
Do Large Language Models Learn Human-Like Strategic Preferences?
*arXiv preprint arXiv:2404.08710*.
- Roberts et al. (2024a)
Roberts, J.; Moore, K.; Fisher, D.; Ewaleifoh, O.; and Pham, T. 2024a.
Large Language Model Recall Uncertainty is Modulated by the Fan Effect.
In *Proceedings of the 28th Conference on Computational Natural Language Learning*, 303–313.
- Roberts et al. (2024b)
Roberts, J.; Moore, K.; Wilenzick, D.; and Fisher, D. 2024b.
Using artificial populations to study psychological phenomena in neural models.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, 18906–18914.
- Roper Center for Public Opinion Research (2025)
Roper Center for Public Opinion Research. 2025.
Roper iPoll.
Data set.
Cornell University.
- Shelmanov et al. (2021)
Shelmanov, A.; Tsymbalov, E.; Puzyrev, D.; Fedyanin, K.; Panchenko, A.; and Panov, M. 2021.
How certain is your Transformer?
In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*, 1833–1840.
- Shorinwa et al. (2025)
Shorinwa, O.; Mei, Z.; Lidard, J.; Ren, A. Z.; and Majumdar, A. 2025.
A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions.
*ACM Computing Surveys*.
- Shrivastava, Liang, and Kumar (2023)
Shrivastava, V.; Liang, P.; and Kumar, A. 2023.
Llamas Know What GPTs Don’t Show: Surrogate Models for Confidence Estimation.
*arXiv preprint arXiv:2311.08877*.
- Steyvers et al. (2025)
Steyvers, M.; Tejeda, H.; Kumar, A.; Belem, C.; Karny, S.; Hu, X.; Mayer, L. W.; and Smyth, P. 2025.
What large language models know and what people think they know.
*Nature Machine Intelligence*, 1–11.
- Strachan et al. (2024)
Strachan, J. W.; Albergo, D.; Borghini, G.; Pansardi, O.; Scaliti, E.; Gupta, S.; Saxena, K.; Rufo, A.; Panzeri, S.; Manzi, G.; et al. 2024.
Testing theory of mind in large language models and humans.
*Nature Human Behaviour*, 8(7): 1285–1295.
- Tang, Shen, and Kejriwal (2024)
Tang, Z.; Shen, K.; and Kejriwal, M. 2024.
An evaluation of estimative uncertainty in large language models.
*arXiv preprint arXiv:2405.15185*.
- Tian et al. (2023)
Tian, K.; Mitchell, E.; Zhou, A.; Sharma, A.; Rafailov, R.; Yao, H.; Finn, C.; and Manning, C. D. 2023.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 5433–5442.
- Ullman (2023)
Ullman, T. 2023.
Large language models fail on trivial alterations to theory-of-mind tasks.
*arXiv preprint arXiv:2302.08399*.
- Wang, Aitchison, and Rudolph (2023)
Wang, X.; Aitchison, L.; and Rudolph, M. 2023.
LoRA ensembles for large language model fine-tuning.
*arXiv preprint arXiv:2310.00035*.
- Xiong et al. (2023)
Xiong, M.; Hu, Z.; Lu, X.; Li, Y.; Fu, J.; He, J.; and Hooi, B. 2023.
Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.
*arXiv preprint arXiv:2306.13063*.
- Zhang et al. (2024)
Zhang, C.; Liu, F.; Basaldella, M.; and Collier, N. 2024.
LUQ: Long-text Uncertainty Quantification for LLMs.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 5244–5262.
- Zhou, Jurafsky, and Hashimoto (2023)
Zhou, K.; Jurafsky, D.; and Hashimoto, T. B. 2023.
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 5506–5524.

## Appendix A Human Comparison Dataset Construction

The Roper dataset is comprised of $2998$ questions obtained from the Roper Center for Public Opinion Research iPoll database (Roper Center for Public Opinion Research 2025). These questions were sampled uniformly at random from an initial set of $30571$ questions pulled from the database. The exact search criteria used to obtain these questions can be found in Figure [6](https://arxiv.org/html/2508.08204v1#A1.F6). The initial set was comprised of all results of this search.

Before sampling, the questions were processed for validity and removal of non-response answer choices. Many questions included non-response answer choices for respondants that had no opinion or preferred to not answer the given question for any reason. We decided to remove these, given that the goal of this research is to investigate the behavior of the model when making a decision, not how likely it is to refuse to make a decision. The latter is a well-established behavior (Arditi et al. 2024), and likely affected by uncertainty, but is outside the scope of this work. We also elected to not remove ”None of the Above” and similar answers that indicate an active rejection of the given answers rather than a passive lack of preference. This is consistent with the questions pulled from Roper iPoll, many of which included both rejection and refusal options. Refusal answer choices were found using manual inspection of the dataset, identifying the following refusal options:

- •
don’t know/refused
- •
don’t know/skipped
- •
don’t know/skippedrefused
- •
no answer
- •
not selected
- •
not selected/no answer
- •
not sure/refused
- •
not sure/skipped
- •
omit
- •
refused
- •
refused/web blank
- •
skip
- •
skipped
- •
skipped on web
- •
skipped/refused
- •
skipped/web blank
- •
web blank

Figure: Figure 6: Search string used to query the Roper iPoll database. Bold indicates field names, while all other text is field content. Keyword exclusions and topics were chosen to minimize personal experience questions because LLMs do not have personal experience histories about which to be uncertain.

After refusal filtering, remaining human answer choice ratios were re-normalized to sum to $100\%$. Questions with fewer than two answer choices after removal of refusal options, totaling $332$ questions, were then removed. Finally, questions for which the total human response ratio sum was invalid were removed from consideration, totaling $572$ questions. Valid answer ratio sum values are those that fall into the range ($100-N$, $100+N$), where $N$ is the number of answer choice options post-refusal-removal, representing the maximum divergence from $100$ given worst-case rounding errors. The resulting set of $29667$ questions were then sampled for $3000$ questions uniformly at random without replacement. Two questions were found after experimentation time to have anomalous choice counts, likely due to those questions having more than $26$ available options, which was not accounted for in the initial processing. These were removed and excluded from all analysis, leaving the reported $2998$ questions. Due to licensing constraints, the final set of questions is not released in the supplementary materials.

## Appendix B Additional Figures

[Uncaptioned image]: Images/JSD-llama3.2-1b.png

[Uncaptioned image]: Images/JSD-llama3.2-1b-instruct.png

[Uncaptioned image]: Images/JSD-llama3.2-3b.png

[Uncaptioned image]: Images/JSD-llama3.2-3b-instruct.png

[Uncaptioned image]: Images/JSD-mistral-7b-0.1.png

[Uncaptioned image]: Images/JSD-mistral-7b-0.1-instruct.png

[Uncaptioned image]: Images/JSD-mistral-7b-0.3.png

[Uncaptioned image]: Images/JSD-mistral-7b-0.3-instruct.png

[Uncaptioned image]: Images/JSD-llama3.1-8b.png

[Uncaptioned image]: Images/JSD-llama3.1-8b-instruct.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: **Questions**: 
1. Analysis of the human agreement is conducted using three metrics: one focusing on measuring choice selection agreement, another focusing on preference ordering alignment, and the third one focusing on the discrepancy in human uncertainty and model’s uncertainty. 
      1.a.) Given the focus on measuring alignment between human group-level uncertainty and various UQ methods’ uncertainty, it is unclear why the first two metrics are necessary (the metrics are independent of any UQ method and instead fully rely on the model distribution). Could the authors motivate the relevance of including such analysis in the paper?
      1.b.) To measure uncertainty alignment between UQ methods and human uncertainty, the paper proposes to measure the linear correlation between the UQ method and the entropy of the human distribution per answer. While using entropy provides an aggregate view of human uncertainty it loses information about which answer choice the model is more confident about. For instance, for a multiple choice question with 4 answer choices, the two distributions have the same entropy but refer to two completely different settings: [0.0, 0.0, 0.25, 0.75],  [0.75, 0.25, 0.0, 0]. In other words, if I understand the setup correctly, the use of entropy as the proxy for human group-level uncertainty followed by pearson correlation may be an overoptimistic measure of alignment. This is further validated by lines 215-238 which state that models are only moderately aligned in the top-token but have different multiple choice ordering. Can the authors please clarify whether this is a problem and how they tackled it (e.g., post-hoc manual analysis)?
3. In Section 5.1. The paper mentions the use of Spearman correlation between binary correctness and UQ measures (lines 274-275). However, due to the discreteness of the correctness variable, my intuition is that this is an ill-suited metric. Can you motivate this metric choice?


**Clarity**: 
1. The paper positions itself as “inference-time uncertainty in LLMs”, defining this class of methods as “measures that are able to be calculated at any time during generation, without additional auxiliary generations” and “inference time measures are uniquely useful [...] without significant added computation”. However, in my understanding such definition does not exclude training time approaches – approaches which are not discussed in this paper. I suggest the authors further clarify the definition or that consider training-time approaches, i.e., approaches that approach calibration through fine-tuning models.
2. Table 1 caption (page 4): add information about what bold faces mean.
3. Lines 272-277 mention the analysis being split in two phases but only one phase is discussed: the analysis using Spearman correlation.
4. Line 432 mentions “model itself may be unusually negatively calibrated”. Please explain what negatively calibrated implies, since in the original classification definition calibration values can only be between 0 and 1.


**Supporting arguments**:
1.  Lines 98 refer to the “limited or no inference-time capabilities [...] like self-reporting [...] and multi-inference consistency”. Please clarify what “inference-time” capabilities are and provide adequate citations to back such arguments.


**Typo**:
- Line 153: “surveys .” → “surveys.”
- Legend title in Figure 2: “Measures” → “Models”
- Missing legend in FIgure 5. It’s difficult to attribute each colored line to a different baseline.

**Suggestion**: 
- Figure 2 is great! But super dense! One possible suggestion if you do consider this, is to select the best UQ method out of the top-k entropy ablations, the best for top-p entropy and best for top-p size and plot those. Leaving the other variations to the appendix.
- Formatting of Table 2 is off. I recommend formatting it like Table 1.

**Additional limitations**:
- The paper should also mention the fact that this analysis is only applicable to open-source models or closed-source APIs providing access to top-k and/or top-p access.
soundness: 2
strengths: - S1. Interesting angle on uncertainty quantification (UQ) research in LLMs, exploring whether the miscalibration of LLMs implies that models are in fact reflecting human uncertainty over answers.
- S2. Overall well-written and organized. Figure 3 provides a summary view over the different MMLU subjects.
- S3. Main results are backed by hypothesis testing (Figure 2 and Table 2).
summary: **Summary**: 
proposes to measure the alignment between various logit-based uncertainty quantification (UQ) methods and human group-level uncertainty, which is operationalized using entropy of population answer choices. Across 10 LLMs and 2000+ multiple choice questions of recent surveys, results reveal that most logit-based UQ methods are (linearly) strongly correlated with human uncertainty ($|\rho| geq 0.3$) – with choice entropy, total entropy and top-k approaches systematically ranking higher across models and datasets. In the MMLU dataset, analysis considering three different calibration metrics reveal that choice entropy is systematically better calibrated being more closely related to the correctness of the model.


**Contributions**:
- Novel perspective on the uncertainty quantification problem, focused on measuring model calibration with respect to human uncertainty (as opposed to correctness).
     - However, a similar problem is studied in Moore et al (2025), which first investigated the alignment of additional sampling measures (including sampling measures). The main difference seems to be the evaluation of both human alignment and calibration. Note that calibration and human alignment are measured separately (in two different settings)..
- Use of a Jensen Shannon Distance Shift as a measure of calibration.
weaknesses: - W1. **Limited novelty**: while the paper expands on the differences between prior work (in lines 82-88) it appears incremental (increasing number of datapoints and carrying calibration analysis). Perhaps the authors can highlight differences in findings or how their extended analysis to calibration differs from findings in prior work.
- W2. The definition of “inference-time” is too broad and not sufficiently motivated: the arguments provided to narrow the experiments to logit-based approaches are  also applicable to training-time approaches to UQ. As such, training-based approaches to model calibration could also be suitable approaches to be assessed in this work.
- W3. **Experiments are conducted exclusively in 3 multiple-choice formats**: Pew Research Center and Roper Center for Public Opinion Research (2025) surveys for measuring uncertainty alignment and MMLU to measure calibration. While there’s value in evaluating uncertainty alignment in multiple-choice settings, it does not necessarily generalize to more realistic settings where users interact with LLMs in open-ended generations.
- W4. Results are obtained using a single prompt: given LLMs’ sensitivity to slight changes in prompts, one may wonder about the generalization of these findings to different prompts (e.g., 0-shot vs few-shot prompting).
- W5. Measurement of alignment between UQ methods and human group-level uncertainty has limitations, since it does not consider differences in the ordering of the answer choices. (See Questions for more details).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. **Generalization to open-ended tasks:** Can you validate the proposed 3-way reduction empirically (few-shot rubric, judge variability), and compare against entailment-based or reference-guided scoring?  
2. **Template & decoding effects:** How robust are alignment/calibration results to prompt variants, temperature, and nucleus/top-k settings?  
3. **Combined signals:** Do simple fusions (e.g., choice entropy + top-k entropy) yield better Pareto fronts for risk–coverage/ECE?  
4. **Cost accounting:** Please report wall-clock, memory, and throughput for per-token computations across models/subjects; include guidance for real-time use.  
5. **Edge cases:** Analyze subjects/models with **anti-calibration** (positive correlation) to identify linguistic or dataset artifacts.  
6. **Scale & APIs:** Replicate key plots on larger and API models; verify whether trends (choice/top-k entropy dominance) persist.  
7. **Human side:** Report inter-survey reliability and how class-imbalance in human choices affects “alignment” correlations.
soundness: 3
strengths: - **Originality.** This paper explicitly targets **human-aligned** uncertainty (beyond correctness calibration) and connects **top-p** selection to Bayesian highest-density sets; it also proposes **JSD shift** as a distributional calibration diagnostic.  
- **Quality.** Careful separation of **alignment** vs **calibration**; broad metric family; multi-model evaluation; subject-wise analyses; multiple complementary criteria (correlation, ECE, JSD shift).  
- **Clarity.** Clear prompt template, dataset construction details, and heatmap/table visualizations that make the mixed “agreement vs ordering” story easy to parse.  
- **Significance.** Practical, **low-overhead** signals (token-probability–based) that can drive runtime control/abstention in black-box deployments; highlights **choice entropy / top-k entropy** as strong default options.
summary: This paper investigates inference-time uncertainty measures for large language models and asks two questions: (i) how well such measures align with human group uncertainty (estimated via survey-response disagreement), and (ii) how well they are calibrated to correctness. Using roughly 3k multiple-choice items from Roper (plus a small subset from Pew) and several 1B–8B open-weight models (base and instructed), this paper compares families of token-probability signals, including top-1 probability, choice entropy, total entropy, top-k entropy, and top-p metrics. The study introduces a distributional calibration check based on Jensen–Shannon-distance shift, alongside per-subject Spearman correlations and ECE-style calibration. Main findings: multiple entropy-based measures—especially choice entropy and top-k variants—show moderate to strong alignment with human uncertainty across models; the same measures also show evidence of correctness calibration on MMLU; preference-order alignment remains weak even when top-answer agreement is above chance; a pathway is outlined for extending these signals to open-ended generation via a reduction to a three-way judgment.
weaknesses: - **Multiple-choice scope.** Evidence is limited to MCQ; the open-ended extension is conceptual and untested.  
- **Prompt/decoding sensitivity.** Alignment differs from prior “counterfactual prompting” studies; results may depend on the **cloze** template and decoding choices.  
- **Model coverage.** Only ≤8B open-weight models are included; larger/API models are referenced but not comprehensively stress-tested.  
- **Metric/threshold choices.** Heuristic thresholds (e.g., |r|≥0.3) and standardization-based ECE binning may affect conclusions; alternatives could be reported.  
- **Compute profiling.** End-to-end **latency/memory** impacts of computing metrics at scale are not fully quantified.  
- **Failure analysis.** Notable negative/anti-calibration cases (e.g., a 7B instruct variant) merit deeper diagnosis.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: Main: 
1. The main question I have is, as I pointed out above, how well the findings generalize to larger and more diverse models beyond the small open-sourced models.

Minor 
1. The paper finds a surprising lack of alignment in preference ordering between the models and humans, which contrasts with some prior work. The authors suggest this might be due to the cloze testing prompt format. Could the authors elaborate on this? Would CoT prompting or other prompt engineering techniques potentially improve preference alignment?
2.  The JSD shift results for the Mistral-0.1 7B Instruct model are intriguing, suggesting it might be "anti-calibrated." Do the authors have any hypotheses for why this specific model exhibits this behavior? Could it be an artifact of its instruction-tuning process?
soundness: 2
strengths: The core problem of evaluating whether an LLM's uncertainty corresponds to human uncertainty is important for building more transparent and trustworthy AI systems. The methodology is sound and described clearly. The creation and use of a large-scale dataset from the Roper Center is a significant contribution. The finding that uncertainty alignment can exist independently of preference alignment is a particularly interesting and non-obvious result.
summary: This paper investigates the alignment of inference-time uncertainty measures in large language models (LLMs) with both human group-level uncertainty and traditional calibration metrics. The authors evaluate a variety of entropy-based and probability-based uncertainty measures on a large dataset of survey questions. They find that several measures, particularly those based on entropy over various subsets of the token distribution (e.g., choice entropy, top-k entropy), show strong alignment with human uncertainty, even when the models' answer preferences do not align with human preferences. The paper also introduces a novel distributional calibration measure, the Jensen-Shannon Distance (JSD) shift, to assess how well model uncertainty predicts changes in the answer distribution. The results indicate that the human-aligned uncertainty measures are also moderately to strongly calibrated.
weaknesses: The study's primary limitation is 
1. The models used are mainly small open-sourced non-reasoning models. Experiment on more diverse and larger models, including closed-source SOTA models such as GPT-4, Gemini, Claude etc., would strengthen the claims, as uncertainty measures are more relevant in widely deployed SOTA models.
2. The method mainly focuses on multiple-choice questions. While this is a necessary simplification, the true test of these uncertainty measures will be in open-ended generation tasks. The proposed conceptual framework for extending the method to open-ended questions is a good first step, but it remains to be seen how effective it will be in practice.
3. The writing style can be improved for example, the contribution section in the introduction can be better formatted.

---

confidence: 3
contribution: 1
initial_rating: 2
presentation: 2
questions: See weaknesses above.
soundness: 2
strengths: 1. To my knowledge, this is the first work that extensively studies top-k and/or top-p of the logit-distribution in terms of calibration and human alignment.
2. A reasonable range of models is used in the experiments.
summary: The paper evaluates various LLM output distribution choices - entropy over the full vocab, top-k entropy, top-p entropy, and ‘choice’ entropy (entropy over the tokens corresponding to the answers in a finite set, e.g. a b c for MCQ questions) - for (i) human-alignment (agreement to human preferences and entropy), measured on a large survey dataset of US public opinion, and (ii) calibration, measured on MMLU.

The authors find that human alignment exists to a moderate degree at the top-token, but the ordering of non-top tokens does not correlate to human preference ordering. Furthermore, the degree of entropy of the models on each question correlates well in general with the entropy of human responses. Finally, the authors show that choice entropy has the best calibration on ECE, though all measures are moderately well calibrated. 

Overall, the paper’s analysis suffers from several weaknesses detailed below; and the contribution, even if these were to be rectified, is marginal.
weaknesses: 1. Regarding human alignment, the experimental design does not seem to be sound. The authors primarily use two datasets, which are public surveys conducted in the US, to gather human opinions. These datasets consist of questions asked over a wide range of dates (2017-2023), and are often on topical issues such as politics. Therefore, the human distribution is not stationary; however, all the LLMs tested are off-the-shelf open-source models, and therefore will have static cutoffs and remain stationary over the time period. Some questions will not make sense based on the cutoff in question, either due to the question already being resolved by the cutoff date, or because the premise of the question is not relevant at the time of the cutoff date yet.
2. Furthermore, the actual impact of the current design is limited. The current design only looks at the aggregate human distribution of responses and how closely LLM logit distributions align with these out of the box – and this is alignment turns out to be weak w.r.t. the top token. Instead, the authors could have directly stated a goal such as using LLMs to replace humans in population surveys (a line of work which does have prior art and interest); and then conducted interventions to try and improve the simple baseline scores reported (such as prompting variations, etc).
3. The paper does not provide examples of questions from the above human survey datasets, so I had to go track these down myself to find out what they look like.
4. Regarding calibration, the authors detail some motivation for why they don’t simply use out-of-the-box normalised entropy, but this should be conducted as an ablation rather than simply stated.
5. ‘Global’ entropy as used in the paper comes with the concomitant risk of distributional shift due to domain change, but this is not commented on nor experimented with by the authors.
6. The analysis of calibration is restricted to simply the MCQ setting (which the authors acknowledge), even though it is easily extensible to the open-ended setting. Furthermore, only a single dataset is used, MMLU (though I acknowledge that this single dataset consists of multiple subject topics), and so out-of-distribution analysis – an important element of calibration analysis – cannot be, and is not, done here.
7. The calibration section has no comparisons to baselines at all despite the plethora of previous works examining e.g. ECE on MMLU. It is therefore not clear what the intended contribution of this analysis is.
8. One of the contributions listed in the introduction is the claim that top-p sampling in LLMs is equivalent to the notion of the highest density credible set in Bayesian statistics. The authors state that this is “an important but previously un-noted connection between the fields”. However, there is no justification or exposition in the paper to support this statement. If interpreted at face value, the statement is also not much of a contribution - it is a fairly trivial insight.

---

confidence: 2
contribution: 2
initial_rating: 4
presentation: 2
questions: In addition to the questions/comments in my Weaknesses section: 

- It’d be nice to show more on the human uncertainty, e.g., in the Appendix. Not exactly clear how MUCH uncertainty there is in the human data. Scatterplots, for instance, would be more revealing than bar graphs, e.g., a point for each query/trial (or at least a subset) to look at alignment. 
- Were the survey datasets included in the LLM training data? They were all before 2023 so could have been? Does that influence uncertainty alignment? I realize it's out of scope for any rebutall period, but the paper would be much stronger with a new human eval as well to really assess alignment with models (for questions that are gaurenteed out of the models' training distribution).
soundness: 2
strengths: The authors take on an important problem --- how human-aligned are models' uncertainty. I really enjoyed and appreciate the authors' emphasis on compute-efficient measures of evaluation. I also appreciate the breadth of metrics the authors consider. The work is quite comprehensive. I enjoyed Figure 2 and would have loved to see more of a deep dive into its results (see below).
summary: How aligned are LLMs' uncertainty to people's judgements of uncertainty? In this work, the authors compare a people's uncertainty judgements to models' judgements. The authors propose and investigate several different measures of uncertainty calibration.
weaknesses: While I believe the paper has potential to be very strong, the current version was structurally challenging to follow. The motivation for the work (to my understanding) centered around evaluating models' uncertainty relative to people. However, this is only really done in Section 4. Section 5 is then not grounded in human data at all? I felt Section 5 came "out of the blue" and disrupted the story of the paper. 

One idea would be to break up Fig 2 into more parts, expand Section 4, and substantially limit Section 5 in this piece. Or, flip the order so that the bulk of the emphasis is on the human evaluation. Structurally, the current paper is unfortunately highly confusing.

There are also no error bars in the main results, making it hard to assess how general and reliable the particular measures are.


Decision: Reject