OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Data
Authors: Alana Marzoev
Jillian Ross
Jacob Andreas
Venue: ICLR 2026 Poster (2026)
Keywords: probabilistic estimation
reasoning
uncertainty
calibration
Primary area: datasets and benchmarks
TL;DR: Language models (LMs) excel at reasoning on tasks with clear answers and complete information. Yet many real-world applications are open-ended and uncertain, and require reasoning about incomplete or noisy data.

Abstract: Real-world settings where language models (LMs) are deployed --- in domains spanning healthcare, finance, and other forms of knowledge work --- require models to grapple with incomplete information and reason under uncertainty. Yet most LM evaluations focus on problems with well-defined answers and success criteria. This gap exists in part because natural problems involving uncertainty are difficult to construct: given that LMs have access to most of the same knowledge as humans, it is non-trivial to design questions for which LMs will struggle to produce correct answers. As a result, LM performance on reasoning under uncertainty remains poorly characterized. To address this gap, we introduce \textsc{OpenEstimate}, an extensible, multi-domain benchmark for evaluating LMs on probabilistic estimation tasks that require models to synthesize knowledge from pretraining and express predictions as Bayesian priors. We assess these priors for accuracy and calibration. Across six frontier models, we find that LM-elicited priors are worth the equivalent of about five samples from the underlying data distribution, and that posteriors computed using LM priors tend to be more accurate than those computed using a naive prior. At the same time, the relationship between model accuracy and confidence is weak across the board, indicating the value of developing new methods to improve calibration. The \textsc{OpenEstimate} benchmark thus offers a challenging evaluation for frontier LMs and a platform for developing models that are better at probabilistic estimation and reasoning under uncertainty.

OPENESTIMATE: EVALUATING LLMs ON REASONING UNDER UNCERTAINTY WITH REAL-WORLDS DATA
Alana Renda, Jillian Ross, Jacob Andreas
MIT CSAIL.
(marzoev, jilliann, jjda ajda)miit... @mi edu
ABSTRACT
Real-world settings where language models (LMs) are deployed - in domains spanning healtheare, finance, and other forms of knowledge work require models to grapple with incomplete information and reason under uncertainty. Yet most LM evaluations focus on problems with well-defined answers and success criteria. This gap exists in part because natural problems involving uncertainty are difficult to construct: given that LMs have access to most of the same knowledge as humans, it is non-trivial to design questions for which LMs will struggle to produce correct answers. As a result, LM performance on reasoning under uncertainty remains poorly characterized. To address this gap, we introduce OPENESTIMATUD an extensible, multi-domain benchmark for evaluating LMs on probabilistic estimation tasks that require models to synthesize knowledge from pretraining and express predictions as Bayesian priors. We assess these priors for accuracy and calibration. Across six frontier models, we find that LM-elicited priors worth the equivalent of about five samples from the underlying data distribution, and that posteriors computed using LM priors tend to be more accurate than those computed using a naive prior. At the same time, the relationship between model accuracy and confidence weak across the board, indicating the value of developing new methods to improve calibration. The OPENESTIMATE benchmark thus offers challenging evaluation for frontier LMs and a platform for developing models that are better at probabilistic estimation and reasoning under uncertainty.
INTRODUCTION
Language models (LMs) have demonstrated strong performance across a broad range of reasoning tasks. However, most existing evaluations confined to problems with clearly defined answers and assume access to complete, unambiguous information. In contrast, many real-world applications in which LMS deployed are characterized by open-endedness and uncertainty.
For example, consider financial analyst assessing the total addressable market of an early-stage investment. To perform this task, the analyst must draw on accumulated knowledge of the competitive landscape and industry dynamics to construct coherent model of the domain and produce an informed initial estimate. Because the setting inherently uncertain, the analyst's beliefs are best expressed not as point estimate but as a probability distribution over possible outcomes-a inn Bayesian terms, a prior- which captures both a central estimate and the analyst's confidence in Generating sound priors requires not only probabilistic reasoning skills, but also the ability to synthesize heterogeneous, noisy, and sometimes opaque sources of evidence into a structured format for downstream inference.
This use case is not unique in its requirements-analleee including healthcare, public policy, and scientific discovery. However, despite the ubiquity of applica- tions involving uncertainty, existing benchmarks seldom test their ability to generate accurate and well-calibrated Bayesian priors in realistic contexts. Some past work (Xia et al. 2024 Wong et al.) 2025) has studied procedures for eliciting probabilistic models from LMs. but most specify the task as a mathematical exercise with fully specified inputs (Paruchuri et al. 2024), or as forecasting questions that are bounded and whose outcomes eventually leak into traming data (Karger et al. (024)).
problems exists across variety of domains,
Designing a faithful evaluation of this capability is nontrivial. A good benchmark must be grounded: it should require the model to draw background knowledge from pretraining to form high-quality

---

priors. At the same time, it must avoid information leakage: if the ground tround-truth answer already exists in the training data, the benchmark tests memorization rather than reasoning.
To address this gap, introduce evaluation procedure based derived conditional statistics: summary statistics computed by filtering large, public observational datasets using randomly sampled conditions. Concretely, ask the model estimate the true value of statistic derived from a public dataset-foo for example, "the average funding raised by non-tech companies outside the US with more than 10 employees" (from Pitchbook (PitchBook Data (024)), or "the average weight of US adults with diabetes and blood mercury levels within a specified range" (from NHANES (Centers for Disease Control and Prevention, 2018). ). Because the filtering criteria (conditions) used to derive these statisties are drawn randomly from large space of possible attribute combinations, the resulting quantities are empirically verifiable yet unlikely to appear verbatim pretraining corpora.
We use this procedure to construct OPEMESTIMATE, a benchmark for evaluating LMs' probabilistic estimation capabilities. The benchmark comprises of 178 such statistics across three domains and can be readily extended to new ones without labor-intensive data collection. Models are presented with natural language descriptions these statisties and are asked to express their estimates of the true value of the statistics as Bayesian priors, which are evaluated in terms of accuracy (whether predicted distributions concentrate near the ground truth) and calibration (whether stated confidence levels align with observed frequencies).
Using OPENESTIMATE, we evaluate the quality of priors elicited from frontier LMs, and find that these models are far from omniscient: in terms of accuracy and calibration, they often perform better-and and often worse- than estimates derived from only a handful of samples from the underlying population. At the same time, these priors could still be useful in practice: posteriors computed using LM priors tend to be more accurate than those computed using uninformative priors.
More broadly, the ability synthesize background knowledge about a topic of interest into well- calibrated Bayesian priors is prerequisite for deploying LMs in high-stakes environments ranging from portfolio construction to policy analysis. OPENESTIMATE reveals that current models fall well short of this bar, and we hope it serves as both a diagnostic tool and a catalyst for developing LMs that can function not just as knowledge retrievers, but as reliable probabilistic reasoners. To support future research and reproducibility. we release our code, benchmark dataset, and evaluation framework
THE OPENESTIMATE BENCHMARK
In this section, we describe the design of the OPENESTIMATE benchmark. We begin by describing our procedure for defining derived conditional statistics as estimation targets across domains (Section We then explain how models are prompted to specify their priors by selecting and parameterizing probability distributions (Section 2.2). Finally, we outline the evaluation metrics used to assess the accuracy and calibration of the resulting priors (Section 2.3
2.1
2.1 DERIVED CONDITIONAL STATISTICS AS ESTIMATION TARGETS
Evaluating LMs' probabilistic estimation skills requires asking questions for which the answers are known but don't 't appear pretraining data. This is difficult to achieve in practice: most of human knowledge is contained in pretraining corpora, so coming up with new questions would require collecting new data experimentally, which is costly and time-consuming. OPENESTIMATE sidesteps this problem by defining derived conditional statistics: summary statistics computed by filtering large-scale observational datasets on specific conditions and aggregating over a target attribute. Because the filtering conditions are chosen randomly, the resulting statisties are empirically verifiable yet unlikely to correspond to well-documented facts in pretraining corpora.
We begin by selecting existing data sources spanning three broad domains: social sciences (Glassdoof labor economics), industrial settings (NHANES(Centery) for Disease Control and Prevention 2018), public health).
Data 2024), finance), and medicine
2

---

Domain
Datasel
# marginal
#1 cond
# cond
#3 cond
Total
Example
Labor Economics
Glassdoor
20
6
43
Midpoint salary
Finance
Pitchbook
17
20
20
61
Total funding
Human Health
NHANES
14
20
20
20
74
Total cholesterol
Table 1: Distribution of statisties across domains. Columns indicate the number of marginal statistics and conditional statistics with one, two, or three conditioning attributes.
From each dataset, we construct two types of statistics. Marginal statistics are computed using every row of the dataset-- in NHANES, for example, this looks something like "the mean weight of adults in the US". Conditional statistics, on the other hand, are computed over filtered subsets of this dataset where the filtering is done by applying up to three additional conditions- for instance, "the mean weight of adults (1) who diabetes diagnosis, (2) who take medication for depression, and (3) have cholesterol above a given threshold.
We sample these conditions at random from the empirically observed values in the dataset. Following Xia et al. (2024), we require that each additional ccondition shift the marginal statistic by least 5%, ensuring that the statistics reflect meaningful variation across subpopulations rather than minor fluctuations due to sampling noise.
The full generation procedure is described in Algorithm 1 and depicted in Figure | the generated statistics for each domain are reported in T
A summary of
Algorithm 1:
Sampling N₃ marginal (k = 0) and conditional (k = 1,2,3, 3) statistics
Input: data D, auxiliary attributes A, counts [N threshold т, n minimum sample size
Output: set V of statistics
V - 0. +
// tracks which attributes have already been used
for k E {0, 1, do
while number of variables with attributes < N do // aę is set of l attributes // keep rows matching attributes in a
sample i distinet attributes ag C A
D' < filter D by a
if D' then continue
// skip if filtered sample is too small
n* < mean (d, : d D')
estimate mean on D' // estimate standard error on D' // unconditional mean on full D
se" + SE(R" D¹)
Pp < mean[d, d, d D
if IA- Mao and ||= "ee,""""""""")
Hol se" and a & S then
add (a,, see) ) to V add ag to
// store valid statistics
// store attributes avoid reuse
return V
While some of the resulting statistics may be contained within pretraining corpora (for example, the over- all diabetes prevalence in the United States is a widely reported on figure), many others far less likely to have been explicitly documented. In particular, the conditional statistics for example, "the mean weight of adults with diabetes who are over 40, have elevated cholesterol, and take medication for depres- sion," or "the median deal size for companies a specific sector with a given number of employees" represent random combinations of attributes that are unlikely have been reported on directly. That said, a model with strong domain knowledge should still be able to reason toward accurate estimates- for in- stance, by drawing on adjacent knowledge of how diabetes prevalence varies with age, how deal sizes differ across industries. By randomly sampling these conditions, we generate a large set of estimation targets that remain grounded in real-world observational data yet avoid testing shallow factual recall.
2.2 SPECIFYING BELIEFS AS BAYESIAN PRIORS
How should we elicit LM estimates about the likely values of these statistics? The simplest approach would be prompt models to produce point estimates, but these capture only first-order the estimate is close to the true value- and say nothing about whether the model has appropriate uncertainty about its answer. By instead requiring estimates be expressed as Bayesian
3

---

Proept What average
Osservationll Plchbook
fasding unsod bs venture bapied
Deried Average Funding Companies
sigma samesendding you Biyva an prior on the valaa
Rospenso SChiin Thought Mu S10M Sigrra SPM
Figure Statistic generation and prior elicitation pipeline. We construct derived statistics from large-scale observational datasets (e.g., PitchBook), specify them as estimation targets, and prompt language models to provide Bayesian priors on their true values.
priors (probability distributions over the variable of interest), we can evaluate both the accuracy the model's 's central estimate and the calibration of its uncertainty. Thus, in OPENESTIMATE we instead require estimates be specified as Bayesian priors (probability distributions on the variable of interest).
Models are provided with a brief natural language description of the statistic of interest and are instructed select and parameterize the functional form of the target distribution accurately. For all experiments in this paper, the models selected either Gaussian, Beta, or log-normal distribution parameterize:
X ~ N(,, o²), X Beta(o, B), or X ~ (jg o²),
We hypothesize that these three forms are chosen by LMs because they arise frequently in our domains of interest- Gaassians for continuous, symmetric quantities like wages; Betas for proportions like disease rates; and log-normals for right-skewed quantities like startup valuations. These distributions represent the models' Bayesian prior or belief about the true value of the statistic and can be combined with samples from the underlying dataset produce posterior distributions.
2.3 EVALUATION METRICS
Given prediction from the LM in the form of a Bayesian prior, how should we evaluate its quality? focus on two complementary dimensions performance:
Accuracy: The degree to which the model assigns high probability density regions close to the empirical ground-truth value.
Calibration: The consistency between the model's 's stated uncertainty and empirical frequen- cies. A model is well-calibrated if events assigned probability p occur with long-run frequency p. such that nominal coverage levels of prediction intervals match their realized coverage.
2.3.1 ACCURACY
To assess accuracy, we evaluate whether the model places the mean of its predicted prior close to the ground-truth value of the statistic.
To quantify this, we first compute the mean absolute error (MAF) between the mean of the prior, (n), and the empirical ground-truth value sr; estimated from the full dataset for each of the variables in the dataset:
1 Σ k4 mcan(p.). 11
MAELIM
To interpret these errors across statistics with different units, we report LM predictions relative to a statistical baseline derived from a small number of samples from the true distribution pulled from the underlying dataset. Starting from naive flat priors (a 1, 3 I for Beta distributions: draw a random sample Á of size Di 5 from the relevant μ = 0, x² 10a for Gaussians), corresponding to a sample of e.g. patients or job postings), sub-population (D' in Algorithm I
from which we can compute a posterior P1(44 D)-
We then compute the statistical baseline MAE the expected error across such samples:
MAF Epll mean(ā((-) B))
4

---

We summarize performance using the enror ratio, defined as the LM's relative to this baseline:
MAELIM
Error Ratio
MAEyaselie
An ratio below indicates that the LM'ss prior more accurate than small, noisy sample from the population whose properties are being estimated.
We also consider the win rate of the LLM prior to the statistical baseline, which is the percentage of the time that the model's ' estimate is closer to the ground truth than the statistical baseline.
Win Rate (LLMM prior > baseline)
Σ (MAELLMM, < MA Basselinee i)
In addition to the N 5 baseline used for computing MAEs, we report win rates against baselines with varying numbers of samples.
Finally, we evaluate the usefulness of these priors in combination with data by computing an LLM posterior:
p(u D) & ((p) p(D μ4)
(1)
(as in the statistical baselines, but replacing the naïve prior with p). As with priors, we evaluate the win rate of these posteriors relative to statistical baselines.
Together, these two dimensions provide more complete picture of accuracy: the error ratio tests the average error of models relative to the statistical baselines whereas the win rate determines how consistently the LLMs outperform the baselines.
2.3.2 CALIBRATION
A model is well-calibrated if the probabilities it assigns correspond to empirical frequencies: events predicted to occur with probability p should occur about p percent of the time. our setting, this means that the ground-truit -tuuth value should fall into each predicted quantile with the correct long-run run frequency,
To measure this, we compute the continuous ranked probability score (CRPS), which penalizes both miscalibrated predictions and overly dispersed distributions. CRPS measures the distance between the predicted cumulative distribution function and the ground truth y without binning:
/_
CRPS(F,8)
(F(x)) T(x 2 v)) dr
where T(x 2 y) is the indicator function. Lower values indicate better predictive performance.
As with MAE, compare LM performance statistical baseline computed from small samples, where p D) is the posterior distribution obtained from sample D of size [D]:
CR CRPShaseline = En
CRPS( 1),
We then report the CRPS ratio:
CRPS
CRPS Ratio
paseliee
EVALUATION
Our evaluation is divided into two parts. In Section 3.1 we evaluate the zero-shot performance
of current language models under standard inference settings. In Section 3.2 we zero in the best-performing models and analyze how changing inference-time settings like system prompt, temperature, and prior elicitation method affect prediction quality.
5

---

Ratios by Domain (95% CI)
Glassdoor
Pitchbook
NHANES
Sereiie 20-51
302
1.07
108
2,35
1.24
1,35
124
125
Mini
GPF 40
Owen3 2350
es Minl
Laana 708
03 Min!
04 Mimi
Uama Top
Oweed 2358
Gen to
Owen3 2350
Uaama 708
oin
64
Figure MAE eroor ratio of LLM prior a naive statistical baseline computed using a uninformative prior and five examples from the true distribution. Most models are no better than five examples; some are significantly worse. There isn't 't statistically significant gap in performance between most model families.
3.1 ZERO-SHI SHOT EVALUATION
In this section, we focus on zero-shot performance under standard inference settings. We do not apply fine-tuning, retrieval augmentation, or prompt engineering beyond directly asking the model parameterize the distribution of variable. To contextualize the LMs' performance, we compare to four statistical baselines that use N E [5, 10, 20, 30] examples that are computed using the procedure described Section 1.3.1
We evaluate six state of ofr language models, including three reasoning models 3) Meta Llama 3.1 8B, Meta Llama 3. 70B (Grattafiori et al. 2024), OpenAl GPT-4 (Achiam al. 03-mini (OpenAl 2025a), OpenAl 04-mini (OpeAA 2025b), and Qwen3-235B-A222 2025). We exclude Llama 3.1 8B after fails to correctly interpret units. We evaluate each model medium temperature reasoning effort corresponding to 0.5 for GPT-4, "medium" for mi-mini and 04-mini, 0.5 for Llama 3.1 70B Instruct Turbo, and 0.6 for Qwen3-23515 2-53-333--222.. We use standard system prompt and prior elicitation prompt which are described in full in Appendix A.1 A.1
| 2023), OpenAl (Yng et al.
Domain
Sample Size
% Prior Better
% Posterior Better
Glassdoor
5
37.0%
71.4%
10
21.7%
69.0%
20
13.0%
68. 1 1%
30
8.7%
70.5%
Pitchbook
5
50.8% %%
69.6%
10
50.8%
76.5%
20
49 49.2%
80.1 %
30
50.8%
81.6%
NHANES
5
74.3% 3%
70.4%
10
59.5%
65. 1 1%
20
47.3%
56.6%
30
37.8%
50.4%
Table 2: Win rate of the LLM prior relative to an N-sample statistical baseline, and win rate of an LLM posterior (LLM prior + N samples) relative to a statistical baseline (uninformative prior + N samples).
Accuracy. Fixing the model to o4-mini, we compare the win rate of LLM priors against statistical baselines computed using N E (5, 10, 20, 30) data points sampled from the true distribution. This addresses the question: "how many data samples is the LLM prior worth?" We then form LLM posteri- ors by updating the LLM prior with the same examples used to compute each baseline, and compare
Here, reasoning models are defined as models that have undergone a dedicated training step that involves reinforcement learning for chain-of-thought

---

the win rate of the LLM posterior against the corresponding statistical baselines. This comparison asks whether starting from an LLM prior leads to better posteriors than starting from an uninformative prior.
As shown in Table 2 find that the standalone LLM priors generally outperform the five-sample baseline in ~000700 of cases, though win rates rapidly drop off with larger numbers of samples. However, even though these priors are often inaccurate in isolation, they can be effectively combined with data: LLM posteriors outperform or match statistical baselines with naive priors.
Next, we compare the accuracy of different model families across domains by evaluating MAE relative to the five-sample statistical baseline. The results are shown in Figure We find relatively little varia- tion between most models (with the exception of Llama-70B), and that again, most models have average eπrors that are no better than five examples; some are significantly worse. This suggests that while the LM priors are often consistently better than the statistical baseline, they are worse in terms of average ab- solute error. On the whole, these results suggest that OPENESTIMATE is challenging for frontier models.
Calibration. Next, we assess model calibration Larger models and reasoning models tend to be better calibrated than smaller, ren-onnsnning models, but again, no single model family consistently outperforms the rest; specific rankings are domain dependent.
Model
Glassdoor
NHANES
Pitchbook
GPT-4o
3.31
1.86
1.10
Llama-3-70B
4.56
2.76
1.13
Llama-3-8B
10.56
19.17
2.74
Qwen3-235B
2.50
1.65
1.04
03-mini
3.17
1.35
0.99
04-mini
2.42
1.17
1.01
Table 3: CRPS Ratio by Model Family Across Domains (vs. 5-Sample Baseline)
presents CRPS ratios comparing each model family the five-sample baseline. Reasoning models (o3-mini and 04-mini) achieve the best overall performance. Performance varies considerably by domain: in Pitchbook, all models perform comparably to the baseline, while in NHANES, smaller models struggle significantly: Llama-3-8 1-maa-3-BB performs 20 times worse than the baseline. Overall. model size and reasoning capabilities appear most critical in the NHANES domain, while even smaller models achieve reasonable performance in Pitchbook.
"We exclude the statistical baselines from Figure in this analysis because the baselines derive their posteriors from the same dataset used compute the ground-truth values. Therefore, larger sample sizes produce extremely tight distributions centered on the ground -ruth mean, which leads the ground truth almost always fall in the middle quartiles (c.g., second or third).
Glassdoor
Pitchbook
NHANES
GPT 4o
0.4
37.8
55.7
6.1
35.1
13.4
18.0
33.4
33.8
21.6
20.0
24.6
Liama 70B
11.7
26.5
22.2
39.6
28.9
14.4
9.5
.2
35.4
20.3
21.1
23.2
8D
Ground Trutts in Ouamile
Liama 8B
5.2
0.4
0.0
94.3
44.6
15.1
8.2
32.1
45.4
11.6
17.8
25.1
Qwen3 2358
9.6
40.4
38.7
11.3
32.8
19.0
16.7
31.5
28.4
25.7
21.4
24.6
-00
o3 Mini
8.7
50.4
37.8
3.0
32.8
20.7
17.7
28.9
27.8
22.7
22.4
27.0
o4 Mini
13.5
30.0
34.8
16.7
30.2
23.8
19.7
Perfect Calibration
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
1
3
1
3
1
3
Quartile
Quartile
Quartile
Figure 3: Heatmap describing the deviations from perfect calibration of each approach. Bolded
values are statistically significant according a per-quartile binomial test (p < 0.05). All approaches systematically overestimated across domains (Quartile is greater than 25%). In some instances, there was high rates of both over and under-estimation (Quartile 1 and 4 are greater than 25%).
7

---

Glassdoor
Pitchbook
NHANES
103
C7743
Number standard deviations
Cumulative distribution function displaying the percentage of ground truth values that fall within na standard deviations away from the mean of the prior, where a is the standard deviation of the prior. The dashed line represents perfect calibration for Gaussian. The best performing models have 80% of the ground truths within 1-2.5 standard deviations from the prior mean. There is overconfidence Pitchbook and NHANES but underconfidence in Glassdoor.
Figure 4:
We study whether this miscalibration is systematic across model families. As shown in Figure 3 we find that all model families exhibit tendency towards systematic overestimation. In Pitchbook. overestimation is compounded by high rates underestimation as well, with both tails overweighted.
Next, we evaluate how tightly models concentrate their uncertainty by examining the cumulative distribution of ground-truit truth values relative the predicted priors (Figure We find the best models cover 80% of the ground truth values within two to three standard deviations of the mean. However, performance is domain-des -deraindddeeeeeett in Glassdoor and NHANES, the best models cover over 80% of ground-truth values within two standard deviations, while in Pitchbook, three standard deviations are required. This suggests that even the strongest models vary substantially how they express uncertainty across domains.
Finally, we analyze whether model-reported uncertainty is a reliable guide to predictive accuracy (Figure 5 by comparing the standard deviation ratio to the error ratio. Ideally, models are low error and well-calibrated. In the Glassdoor domain, models appear reasonably well-calibrated relative to the five-sample statistical baseline, but consistently less accurate than this baseline. In contrast, models in Pitchbook are consistently more confident and less accurate than this baseline. Results in NHANES fall in between these extremes: models generally achieve lower error than in Glassdoor, but their uncertainty estimates are less well well-calibrated, with several models exhibiting either under- or dispeiipern Taken together, these results indicate that the relationship between uncertainty and accuracy is once again strongly domain- memain-deeendent.
We also assess whether predictive uncertainty aligns with accuracy by examining the rank correlation between the two for each model family. A stronger correlation between predictive uncertainty and accuracy would indicate that uncertainty is good indicator of accuracy. However, the reality mixed: uncertainty is good indicator of accuracy in NHANES but not necessarily in Pitchbook Glassdoor.
3.2 ABLATIONS
We investigate how inference-time settings inffuence the quality of elicited priors, focusing three factors: (i) temperature or reasoning effort, (ii) system prompt, and (iii) elicitation protocol. To isolate their effects, evaluate both reasoning model (OpenAl o4-mini) and non-reasoning model (OpenAl gpt-4o). The full set of results is shown in Appendix A.2 None of the settings tested has consequential impact on performance, indicating that more sophisticated approaches to improving accuracy and calibration are needed.
4 RELATED WORK
Our work intersects with three major lines of language model research: evaluating probabilistic reasoning as a mathematical skill, structuring probabilistic reasoning for better estimation, and applications to forecasting.
8

---

Uncertainty vs Accuracy by Model (colored by Spearman correlation)
16-
24
Gura
34
Gre
17.5
94
Hoin Ervor Pacto (reaative)
22-
Misin e nurta (rodattv
Moai Emor Pudio Rellavve
15.2
-12
12.5
120
100
54
-02
JUME 124
0.
Mean Standardd Deviatton
Figure Relationship between uncertainty and accuracy across domains. Each point shows a model's error ratio versus its standard deviation ratio relative to the N = 5 baseline. Colors indicate the Spearman correlation between predictive uncertainty and accuracy within a single model's predictions, addressing the question of whether given model tends to be comparatively more confident when it's more accurate. These correlations differ more so by domain than by model.
Evaluating probabilistic reasoning. One line of research examines how well LMs perform at problem-solving tasks involving structured probabilistic models. For Eaxample,Paruchhie evaluate models' probabilistic reasoning given simple idealized distributions; |Nafar et al. (025
al. (024))
tests models' ability to provide probabilistic estimates given Bayesian network; and al. (023)
examine the models' causal reasoning given probabilities. Collectively, these studies frame proba- bilistic reasoning
mathematical exercise with clearly defined inputs and well-specified outputs. By contrast, our benchmark targets real-world estimation problems, where the relevant information must be inferred rather than provided and the ground truth itself may be ambiguous or unavailable.
Structuring probabilistic reasoning. Another line of work proposes structures for LM-based probabilistic reasoning improve performance. Using "guesstimation" questions similar ours, Xia et al. (2024) prompt LMs to propose relevant random variables and moment constraints, and then fits a log-llnnar distribution that satisfies these constraints. |Feng al. (2024) take a similar approach, and evaluate multi-step process in which LMs brainstorm relevant factors, make coarse probabilistic assessments, and construct an approximate Bayesian network for inference. Huynh et al. (025) use LLMMs to generate synthetic counterfactual outcomes by sampling pseudo-observations. constructing empirical distributions. These approaches extend beyond single-variable reasoning by introducing latent structure and explicit intermediate steps. However, the focus for Xia et al. (2024) and Feng
et al. 2024) is answering discrete multiple-choice questions, while Huynh et al. (2025) focuses on augmenting small datasets for downstream causal inference tasks rather than directly evaluating the quality of L.LM-MMeneaaee distributions.
Like approach. Sepby et al. (2025) elicit parametric Bayesian priors from L.MSs. However, they evaluate priors by comparing them to human expert elicitation existing psychology studies or to historical observational data in specific settings (e.g., precipitation and temperature in particular cities in December). By contrast, we specifically construct derived variables- cross-sections tabular data-across diverse domains; directly evaluate accuracy and calibration relative to estimated ground truth; and we systematically evaluate how model family and inference time settings impact results.
aggregations and
Language model- -ased forecasting, Recent studies have also evaluated LMs' forecasting capabilities KKrger et al. 2024 Halawi et al. 2024 Ye et al. 2024, Chang al. 2024 Schoenegger et al. 2025). These works also test whether models can synthesize heterogeneous evidence into well-calibrated estimates, but they focus on making predictions about real-world future events. In contrast to our benchmark, the outcomes of forecasting questions are, by design, highly likely to appear in LMs' training data after they resolve; they thus perpetually become "stale" and must be replaced with new questions, noted by Karger et al. (2024). By focusing questions that require reasoning about fine- -gainnd cross of tabular datasets, rather than future events, OPENESTIMATE questions are designed to remain challenging over time.

---

5 LIMITATIONS AND FUTURE WORK
While OPENESTIMATE provides first step toward evaluating uncertainty in open-domain estimation, several limitations remain that point to directions for future work. Ground truth values in OPENES- TIMATE estimated from finite samples, and therefore might exhibit estimation error. Moreover, while OPENESTIMATE was constructed to reduce systematic information leakage, leakage still can occur varying degrees. In terms of scope, the current benchmark is limited questions derived from three datasets across three domains; expanding to new domains would lead to a more thorough evaluation of priors. In terms of evaluation, we focus our attention on zero-shot methods without retrieval or fine-tunings studying training-ting time interventions for uncertainty awareness and domain adaptation would be complementary next step in future work.
CONCLUSION
We introduced OPENESTIMATE, benchmark and evaluation framework for assessing language models on open-ended probabilistic estimation with real-world tabular data. The benchmark (i) defines a realistic task where models must express beliefs as full probability distributions (Bayesian priors), (ïi) elicits priors from the LMs through several protocols, and (iii) evaluates performance in terms of accuracy and calibration against statistical baselines. By focusing on derived conditional statistics from domains such public health, labor economics, and finance, OPENESTIMATE probes the ability for models to reason under uncertainty while limiting direct factual lookup and information leakage.
ACKNOWLEDGMENTS
Research was sponsored by the Department the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number 1--750019-2-1000 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies. either expressed or implied, of the Department of the Air Force the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.0877,, 2023.
Centers for Disease Control and Prevention. National health and nutrition examina- tion survey (NHANES), 2017-2018. U.S. Government, National Center for Health Statistics, 2018. URL https: // www. cdc gov/nchs//haans/ Data set, accessed via
https://ww.. cdc. http://wwwwdd..ooo//////////t/ee gov/nchs//hanes includes questionnaires, datasets, and documentation.
He Chang, Chenchen Ye, Zhulin Tao, Jie Wu, Zhengmao Yang. Yunshan Ma, Xianglin Huang, and Tat-Seng Chua. A comprehensive evaluation of large language models temporal event forecasting. arXiv preprint arXiv:2407. 2024.
Yu Feng, Ben Zhou, Weidong Lin, and Dan Roth. BIRD: A trustworthy Bayesian inference framework for large language models. arXiv preprint arXiv:2404. 2494. 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level forecasting with language models. Advances in Neural Information Processing Systems, 37: 50426-50468. 2024.
10

---

Nicolas Huynh, Julianna Piskorz, Jeroen Berrevoets, Max Ruiz Luyten, and Mihaela van der Schaar. Improving treatment effect estimation with LLM-based data augmenta- tion. In 1st ICML Workshop Foundation Models for Structured Data, 2025. URL httppppeeeeeeeeee//oooorrndddd/d
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman- -einer,, Mrinmaya Sachan, et al. Cladder: Assessing causal reasoning in language models. Advances in Neural Information Processing Systems, 36:
31038-31065, 2023.
Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E Tetlock. Forecastbench: A dynamic benchmark of Al forecasting capabilitiess arXiv preprint 0rXiv:2409.19832 2024.
Aliakbar Nafar, Kristen Brent Venable, Zijun Cui, and Parisa Kordjamshidi. Extracting probabilistic knowledge from large language models for Bayesian network parameterization. arXiv preprint 17X1::2505.15918, 2025.
OpenAl. 03-mini system card. Technical report, January 2025a. Published January 31, 2025.
OpenAl. o3 and o4-mini system card. Technical report, April 2025b. Published April 16, 2025.
Akshay Paruchuri, Jake Garrison, Shun Liao, John Hernandez, Jacob Sunshine, Tim Althoff, Xin Liu, and Daniel McDuff. What are the odds? language models are capable of probabilistic reasoning. arXiv preprint arXiv:2406. /2830, 2024.
PitchBook Data. Pitchbook database. Accessed via Wharton Research Data Services (WRDS), 2024.
http:htts////dddwwwwa whar toennnoo..uenn..... edu/ Accessed December 29, 2024.
Philipp Schoenegger, Peter S Park, Ezra Karger, Sean Trott, and Philip E Tetlock. Al-augmented predictions: LLM assistants improve human forecasting accuracy, ACM Transactions on Interactive Intelligent Systems, 15(1):1-25, 2025.
David Selby, Yuichiro Iwashita, Kai Spriestersbach, Mohammad Saad, Dennis Bappert, Archana Warrier, Sumantrak Mukherjee, Koichi Kise, and Sebastian Vollmer. Had enough of experts? Quantitative knowledge retrieval from large language models. Stat, 14(2):770054 2025.
Lionel Wong. Katherine M. Collins, Lance Ying, Adrian Weller Cedegao E. Zhang, Tobias Gersternberg, Alexander K. Lew Timothy O'Donnell, Joshua B. Tenenbaum Jacob D. Andreas, and Tyler Brooke-Wilson Modeling open-world cognition as on-demand synthesis probabilistic models. arXiv:2507. 1254,,7,,,
Shepard Xia, Brian Lu, and Jason Eisner. Let's think var-by-var: Large language models enable ad hoc probabilistic reasoning. arXiv preprint arXiv:24/2.02081,, 2024.
An Yang. Anfeng Li. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, al. Qwen3 technical report. arXiv preprint S..Iv::505509388 2025.
Chenchen Ye, Ziniu Hu, Yihe Deng, Zijie Huang, Mingyu Derek Ma, Yanqiao Zhu, and Wei Wang. Mirai: Evaluating LLM agents for event forecasting. arXiv preprint 24ox::2400.012311 2024.

---

A
APPENDIX A
A.1 ZERO-SHOT ESTIMATION
We lested Llama 8B but excluded from our analysis because it incorrectly followed instructions pertaining units and had an average error that was orders of magnitude larger than the other models due to this mistake.
System Prompt.
Glassdoor
You helpful assistant that questions about the labor market.
Pitchbook
You are helpful assistant.
NHANES
You are helpful assistant that can answer questions about human health.
A.2 ABLATIONS
Elicitation Protocol.
Direct
You are statistical
expert tasked with constructing prior distribution for variable. Your goal is to choose the most appropriate distribution type and estimate its parameters.
Your estimates should reflect uncertainty about the population-level parameter, not the variation across individual observations.
Here is the variable you need to model:
[[variable]]
((units_descriptionn)
Available Distribution Types: Normal (Gaussian), Lognormal, Beta
Instructions:
1. Reasoning:
First, provide detailed reasoning
explaining how you arrived at your specific parameter values. Address: What range do you expect the population parameter to fall in and why? How certain/uncertale you about this parameter? How do your chosen parameter values translate meaningful quantities in the original scale (e.g., median, mean, quantiles, credible intervals)? Why is this distribution type appropriate for this variable?
2. Output: After your reasoning, provide
your answer using EXACTLY these XML tags based which distribution you choose:
If you choose Normal:
<mu>value</mu>> <sigma>value<siimm>>
If you choose Lognormal:
12

---

<mu>value</mu> <sigma>value<<sigmaa
CRITICAL: and sigma parameters in LOG-SPACE, not real real-space!
Key relationships to real-space values:
MEDIAN (real-space) exp(mu)
MEAN (real-space) exp(mu sigma 2/2)
MODE (real-space) exp(mu sigma"2 2))
How to set mu: First decide what you think
the MEDIAN value should be (in the original units), then set mu = in(median).
Examples: If median should be 30 dollars, then 1n(30) 3.4 approximately. median should 100 employees, then In(100) approximately. median should 1000 dollars, then mu 1n(1000) 6.9 approximately.
How to set sigma:
sigma controls the spread in space (typical values: 0.2 to 1.0). sigma 0.3 gives roughly 95 percent credible interval (exp(mu-0.6) exp (mu+0. 6)]. sigma 0.5 gives roughly a 95 percent credible interval of [exp(1u11.0 exp (mu+111... .8]]
Common mistake to avoid:
WRONG: Setting mu 30 when you mean the value is 30 dollars (This gives median = exp(30) = 10 trillion dollars!)
CORRECT:
Setting 1n(30) = 3.4 approximately when you mean the value is 30 dollars (This gives median exp(3.4) 30 dollars)
Always verify: Calculate exp( Does this match
your expected median? Calculate exp(mu + sigma 2222). Does this match your expected mean? If these wildly different from what you expect, you have made error!
If you choose Beta:
<alpha>value<<alpph>> <beta>value</beta>>
Critical Unit Check: Pay close attention to units. If the variable says in millions USD, you need to work in millions! For example, I think the typical company has raised about 3.5 million dollars. In millions, this is: 3.5, NOT 3500000!
Now, please analyze the variable and provide your reasoning followed by your distribution choice and parameters.
Quantile
You are statistical expert tasked with constructing prior distribution for variable. Your goal is to choose the most appropriate distribution type and express your uncertainty about the param parameters true value using quantile estimates.
Your estimates should reflect uncertainty about the population-level parameter, not the variation across individual observations.
Here is the variable you need to model:
((variable))
(
Available Distribution Types:
13

---

Normal (Gaussian):
For var variables that can be positive or negative, symmeticc around the mean
Lognormal:
For strictly positive variables, often right-skewed (e.g., prices, sizes, counts)
Beta: For variables bounded between 0 and (e.g., proportions, probabilitiee
Instructions:
1. Consider the context of the variable, including its meaning and any relevant information that informs your beliefs.
2. Choose the
most appropriate distribution type based on: The natural bounds of variable (can it be negative? is it bounded be tween 0 and 1?). The expected shape of uncertainty (symmetric vs. skewed?). The nature of the quantity being estimated.
3. Estimate the following percentiles the parameters true
value: 5th percentile (only a 5 percent chance the true value is below this). 25th percentile. 50th percentile (median, your best estimate of the true value). 75th percentile. 95th percentile (only percent chance the true value is above this).
4. Begin your analysis by showing your thought process inside <parameter_es teatteresstmaaion_prroooo
tags. Include the following elements: Explicitly state the type of parameter ter being estimated (e.g., population mean, propor proportion). Explain why you chose particular distribution type. List any known facts data points about the variable. Consider and list possible data sources methods for estimating this parameter. Brainstorm factors that might influence the parameters value. Note potential biases or limi fations in the available information.
State any assumptions you are making. Consider how the parameter might have changed
different subgroups. Provide your quantile estimates with a brief explanation for each. Include relevant facts or context about the variable. Justify your choices. Emphasize population parameter uncertainty (not individual variability). Reflect what your estimate spread indicates about your certainty. Consider any plausible edge cases alternative scenarios.
time
5. After your analysis, provide your final answer in the following format:
ctnttribbutioo_tyeeeeorrrrrrr Lognormal, or Betalk/distrrbbttion.tyys
<q5>[5th percentile value]</q5>
<q25>[25th percentile value]</q25>
<q50>[50th percentile (median) value]</q50>
<q75>[75th percentile value]</q75>
<q95>[95th percentile value]</q95>
<justification>
[Brief summary of your reasoning, including why you chose this distribution type] </justificatio>>
<confidence__eve
[Description of how certain or uncertain you are, and why] </confidence_
Examples:
1. Normal Distribution Example: Variable: Average temperature in a city during summer Units: Degrees Celsius
<distribution iisstrrtttoo.kyyeemmmma/s/ssrrrr <q5>22</q5> <q25>24</q25>
14

---

<q50>26</g50>
<q75>28</g75>
<q95>30</q95>
<justification>
Normal distribution appropriate because temperature can
be positive or negative and uncertainty about the mean is approximately symmetric. Based on historical climate data and considering year-to-year variation. The spread reflects uncer tainty in long-term averages due to climate variability.
</justification>
<confidence__evee
Moderately confident. Climate data is well-documented but climate change introduces some uncertainty about current averages. </confidence__evee
2. Lognormal Distribution Example: Variable: Average home price in metropolitan Units: Thousands of USD
<q5>280</g5>
<q25>350</q25>
<q50>420</q50>
<q75>520</q75>
<q95>680</q95>
<justification>
Lognormal distribution is appropriate
because home prices are strictly positive and typically right-skewed. Based on recent market data and regional economic indicators. The asymmetric spread (wider the high end) reflects the possibility of higher prices in desirable areas.
s/jusiffiattion
seenfidence_eevlll
Somewhat uncer Housing markets volatile and influenced by many factors including interest rates and local economic condi tions. </confidence_level>>>
3. Beta Distribution Example: Variable: Proportion of customers who complete purchase after adding items cart Units: Proportion (0 to 1)
35550.55</55> <q25>0.62</q25>
<q50>0, 68</q50>
74</q7>>
<q95>0.80</q955
<justification> Beta distribution is appropriate
because this is a proportion bounded between 0 and 1, Based on industry benchmarks for e-commerce conversion rates and typical cart abandonment patterns. The spread accounts for variation across different product categories and costomrr segments.
</justification>
Moderately confident. Conversion rates well-studied
e-commerce, can vary significantly by industry and website design. </confidence__evel
15

---

Critical Unit Check: Pay close attention to units. variable says in millions USD, you need to work in millions. For example, I think the typical company has raised about 3.5 million dollars. In millions, this is 3.5, not 3500000
Remember to tailor
your analysis to the specific variable and units provided, focusing on uncertainty about the population-level parameter rather than individual variability.
Veran-Vriance
You are statistical expert tasked with constructing
a prior distribution for variable. Your goal is to choose the most appropriate distribution type and estimate its parameters using mean standard deviation.
Your estimates should reflect tncert about the population-level parameter, not the variation across individual observations.
is the variable you to model:
{(variable))
Available Distribution Types:
Normal (Gaussian):
For variables that can be positive negative, symeeric around the mean
Lognormal:
For strictly positive variables, often right-skewed (e.g., prices, sizes, counts)
Beta: For variables bounded between and (e.g., proportions, probabilities)
Instructions:
1. Consider the context of the variable, including what it represents and any relevant information or assumptions that inform your beliefs.
2. Choose the
most appropriate distribution type based The natural bounds of the variable (can negative? is it bounded tween 0 and 1?). expected shape uncertainty (symmetric vs. skewed?). The nature of the quantity being estimated.
3. Estimate the following quantities:
Best guess (mean): : your estimate of the most likely value of the population-level parameter. Standard deviation: a numerical expression of your uncertainty about the true value, not the variability across individual observations.
4. Begin your analysis by showing your thought process inside paraameerr.etiimatioopprrr.
tags. Include following elements:
Clearly state
the type of parameter being astimated (e.g., population mean, true proportion). Explain why you chose particular distribution type. List any known facts, data points, or previous estimates about the variable. Consider possible data sources, analogous populations, or related studies that inform your belief. Identify key factors that might influence the value of the parameter. Note any limi tations, uncertainties, or assumptions in your reasoning. Reflect how the parameter might differ across subgroups or change over time. Provide your best guess (mean) and your estimate of the standard deviation. Justify your choices with reference to the context, data, and assumptions. Emphasize that your uncertainty pertains to the population parameter, not indi individul variation. Reflect what
16

---

magni tude your standard deviation implies about your confidence. Consider plausible edge cases or outliers that helped you calibrate your uncertainty
5. After your analysis, provide your final answer in the following format:
sdistribution_tye NNraa, Lognormal, or Betalk/dissrrbbutionn.yyy <mean>[Best guess for the true value]</mean> <std_dew>[Standard deviation representing your uncertainty3kkstdd.ee
<justification>
[Brief summary of your reasoning, including
why you chose this distribution type and what informed your parameter estimates]
</justification
<confidence_lewee level> [Explanation of how confident or uncertain you are, and why] s/confidenc__level
Examples:
1. Normal Distribution Example:
Variable: Average height of adult males in country Units: Centimeters
<mean>115<</man>> siid_dde>2..<<stte
<justification>
Normal distribution is appropriate because
height can theoretically take value and is approximately symeeric around the mean. Based on global averages, previous studies in similar populations, considering factors like nutrition genetics. The standard deviation reflects uncertainty due to potential sampling biases and regional variations.
</justification
<confidence _evel>>>>
Moderately confident. While height is well-studied, variations between regions and over time introduce some uncertainty. </confideenc__eeell
2. Lognormal Distribution Example:
Variable: Average annual revenue of small businesses in a region Units: Thousands of USD
<distribution <mean>250<</mean>
sstd__dev110</sttd.dd _eev
<justification>
Lognormal distribution
is appropriate because revenue is strictly positive and typically right-skewed, with some businesses earning significantly more than the median. Based on industry reports and regional economic data. The standard deviation reflects substantial uncertainty due variation
industries and economic conditions.
</ </justification>
<confidence_leww
Somewhat uncertaine tain. Business revenue varies widely by industry and economic conditions, available data may not fully representative.
</confidence__eeel)
3. Beta Distribution Example:
17

---

Variable: Proportion of people who prefer tea over coffee a city Units: Proportion (0 to 1)
<distribution Suttrrotttooo.pooeeecaaddrrrrrrt <meen>>.6<</man>> <std_dev>0. 05</std_dev>
<justification>
Beta distribution is appropriate because this is proportion bounded between and 1. Estimated based on local cultural preferences, limi ted survey data, and comparison with similar cities. The standard deviation accounts for potential biases in available data and variations across different demographics. </justification>
cconfidence_led
Somewhat uncertain. Beverage preferences can vary significantly based on factors like age, cultural background, and local trends, which are not fully known. </confidence__evel>>
Critical Unit Check: Pay close attention to units. If the variable says in millions USD, you need to work in millions. For example, I think the typical company has raised about 3.5 million dollars. In mi millions, this is 3.5, not 3500000.
Remember: you are modeling beliefs about the parameter, not the spread of raw data. Your standard deviation should reflect how much uncertainty you have about the single true value that governs the population, not the spread of outcomes across individuals.
Provide your analysis and final answer based on the given variable and units description. Your final output should consist only of the formatted answer and should not duplicate rehash any of the work you did in the thinking block.
Additional Results.
We calculate expected calibration enror as a complimentary metric for calibration. Let Qij be the j-th quartile bin of p.. We define â Σ 1(4 E Qiy}. Formally, we compute the quartile expected calibration error (ECE) as:
ECE Σ |4j - 0.25| j=1
Lower values indicate better calibration, with ECE quartile granularity).
corresponding perfect calibration (at
Figure 6: Effect of elicitation protocol (direct, guantile, mean-variance) on error ratio, expected calibration error (ECE), CRPS ratio, and uncertainty (standard deviation) across reasoning and non-reasoning models, relative direct elicitation.
18

---

J
Figure 7: We examine the impact of changing temperature reasoning effort accuracy, calibration, and certainty.
Figure 8: We examine the impact of changing the system prompt reasoning effort accuracy, calibration, and certainty.
19


Reviews:
[{"review_id": "L5Y51MQM6m", "reviewer": "Reviewer_oFRa", "summary": "This paper introduces OPENESTIMATE, a benchmark designed to evaluate LLMs on their ability to generate well-calibrated Bayesian priors for real-world quantities in domains such as labor economics, finance, and public health. Models are asked to express uncertainty as Gaussian or Beta distributions, and their outputs are assessed for accuracy and calibration against empirical data.\n\nExperiments across six frontier LLMs (including GPT-4 and Qwen3-235B) show that current models are generally inaccurate and overconfident, often performing no better than using five random samples from real data. While some reasoning models (e.g., o3-mini, o4-mini) better capture probability mass near true values, calibration remains poor and domain-dependent. Ablation studies reveal that elicitation method (how uncertainty is prompted) affects results more than temperature or system prompt settings.\n\nOverall, the study finds that LLMs’ probabilistic reasoning is weak but not random, showing structured, domain-aware uncertainty that could serve as a foundation for improving AI systems that reason under uncertainty.", "questions": "1. The authors could provide a discussion of the differences across models, such as distinctions between open- and closed-source models, and how model size may affect performance on the benchmark.\n\n2. Clarify the motivation and reasonableness of the zero-shot task setting, and whether the model outputs can truly be interpreted as Bayesian priors.\n\n3. Can you justify the choice of focusing only on three social-science domains or expand to other fields.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Novel Benchmark Design: The paper introduces OPENESTIMATE, a first-of-its-kind benchmark that evaluates LLMs on probabilistic estimation using real-world tabular data. Unlike prior work focused on deterministic or forecasting tasks, it systematically measures both accuracy and calibration of Bayesian priors.\n\n\n2. Comprehensive Empirical Evaluation: It provides a cross-domain assessment (labor economics, finance, and public health) across multiple frontier models (GPT-4, Llama 3.1, o3/o4-mini, Qwen3). The inclusion of strong statistical baselines gives the results clear interpretability and robustness.", "weaknesses": "1. The task definition appears to lack clear motivation. Under a zero-shot setting, the LLM has no prior exposure to the specific datasets used in the benchmark, making it unclear how it could produce meaningful estimations. Moreover, it is debatable whether the model’s outputs can truly be considered “priors,” since they effectively reflect the posterior knowledge embedded during pretraining. The way an LLM is trained or fine-tuned likely has a substantial influence on these results, and this issue should be explicitly acknowledged and discussed.\n\n2. The paper focuses on three domains: labor economics, private markets, and public health, but it is unclear why these specific areas were chosen over other possible domains. The authors should clarify the rationale for selecting datasets exclusively from the social sciences and explain why natural sciences, medicine, or engineering data were not considered. In addition, it would be helpful to articulate what types of real-world reasoning or uncertainty these three domains are intended to represent, and what general conclusions can meaningfully be drawn from results confined to these areas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "okafyvbr5u", "reviewer": "Reviewer_7ciD", "summary": "The paper builds a benchmark dataset OPENESTIMATE, which evaluates LLMs' ability to provide a prior probability distribution over uncertain variables. The variables are constructed from real-world datasets on labor economics, private markets, and human health. LLMs are prompted to specify a Gaussian or Beta distribution for the variables. Results show that state-of-the-art LLMs perform poorly on this benchmark; for example, the LLM-generated priors are often less accurate than posteriors formed from 5 real samples.", "questions": "1. In Figure 2b, how can the expected calibration error (ECE) be as large as 10? By definition, the maximum value that ECE can take is $|1-0.25| / 4 = 0.175$. Are the numbers supposed to be *percentage* ECEs?\n2. In constructing the derived variables, the paper conditions only on attributes that alters the target statistic by at least 5%. Would this create overly difficult estimation targets with high variability, and thus make the benchmark more difficult than common real-world tasks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper studies a practically important problem, namely LLMs' ability to give a good prior over random quantities in the real world.\n2. The paper gives a reasonable approach to generate derived variables that are unlikely to have been explicitly documented in LLMs' pretraining data, by conditioning on attributes that change the target statistic sufficiently.\n3. The performance metrics for accuracy and calibration are reasonable and intuitive.", "weaknesses": "1. My major concern is that the paper uses only the Gaussian and Beta distributions, which may be highly misspecified probabilistic models for the real-world data. This raises the question of whether the LLMs' poor performance is true reflection of its probabilistic reasoning skills, or an artifact of being forced to specify an inappropriate model. I think an important evaluation would be to let an LLM propose a distribution form on its own, and see if the accuracy and calibration improve.\n2. The derived variables are constructed by conditioning on randomly sampled attributes. While it can effectively avoid data leakage, I wonder if the derived variables are always practically relevant, e.g., ones that a real-world analyst would actually ever estimate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8SBDYUaeJB", "reviewer": "Reviewer_5LwK", "summary": "The paper introduces OpenEstimate, a benchmark for testing large language models on probabilistic estimation tasks. Instead of giving point estimates, models must express beliefs as Bayesian priors (Gaussian or Beta distributions) for real-world quantities drawn from datasets in labor economics, finance, and public health. Results show that even advanced models like GPT-4 and LLaMA 3 are poorly calibrated and overconfident, performing no better than simple statistical baselines built from a few real samples.", "questions": "Do you expect that supervised or reinforcement learning using OpenEstimate could improve uncertainty calibration?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper fills a clear gap by evaluating LLMs’ ability to reason under uncertainty, focusing on probabilistic estimation rather than deterministic prediction.\n\n2. The study spans multiple domains and models, using clear metrics for accuracy and calibration and comparing results against interpretable statistical baselines.", "weaknesses": "1. The paper mainly reports performance differences without probing why models fail to calibrate uncertainty.\n2. Restricting distributions to Gaussian and Beta forms limits realism for complex or multimodal uncertainty.\n3. The benchmark is not used to improve models or study uncertainty learning, missing the opportunity to explore how training on such data could enhance self-calibration.\n4. Only zero-shot inference is tested. There’s no attempt to improve performance through structured prompting, self-consistency, or retrieval, which would make the benchmark more actionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LziGa14WoE", "reviewer": "Reviewer_H3RD", "summary": "This paper introduces a new benchmark designed to evaluate the ability of LLMs to perform probabilistic estimation, as opposed to merely extracting point estimates from such foundation models. The **OpenEstimate** benchmark involves asking LLMs to generate parametric Bayesian priors, specifically parametrized as Normal or Beta distributions for a set of of \"derived variables\" from real-world datasets in various fields.\n\nThe evaluation is twofold: firstly **accuracy**, which uses a normalized mean absolute error to verify whether the location of the mode of the distribution is correct, while accounting for the probability mass of the 'ground truth' distribution at that point.\nSecondly, the authors measure **calibration**, a kind of empirical earth-mover's distance between the observed frequencies and the predicted distribution.\n\nExperiments with several recent general-purpose LLMs suggest no systematic patterns by any particular LLM on any particular task, but that most models tend to be overconfident and do worse than 5 draws from the empirical distribution.", "questions": "1. The central claim is that LLMs appear to be 'overconfident'. How can we be sure that this is not an artifact of the experimental design? The restriction to Gaussian priors for heavy-tailed or skewed data (e.g. funding) is likely to enforce model mis-specification.\n\n2. Why split the data into four coarse bins to calculate ECE instead of standard, more informative methods for assessing calibration, such as quantile--quantile plots?\n\n3. What is the definition of 'reasoning' and 'non-reasoning'. Can you justify placing powerful GPT models in the 'non-reasoning' category?\n\n4. What is the justification for using the *mode* of the distribution for the MAE calculation?\n\n5. What is the formula or citation used for the correction term from the Jeffreys prior used in the scale-adjusted log-probability metric?\n\n6. How does this work compare with others on eliciting Bayesian priors from LLMs (see above)?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper tackles an important and topical problem of evaluating LLM quantification of uncertainty, which is an emerging and until recently underexplored area of research. Generating full, parametrized priors is a relatively novel idea compared to many other works in the literature, which rely on less 'statistical' approaches, such as simple point estimates or series of questionnaires.\n\nThe proposal of \"derived variables\" is a good idea, as it brings the benchmark closer to measuring the utility of priors on realistic tasks, rather than estimating superficial summaries.\n\nIf the code and benchmark datasets are made available, this could be a useful resource for other researchers and practitioners. [NB: the code/data was not visible to reviewers at this stage.]\n\nThe motivation for the paper is well articulated and overall the writing and presentation of the article are clear. The figure labels are readable (with caveats, see below).", "weaknesses": "The most significant weakness of the paper is the restriction to normal and beta distributions for continuous variables and proportions, respectively. While this is noted in the limitations section, it undermines the validity of the benchmark. For any continuous variable with a skewed distribution or heavy tails---which is indeed likely to include those in one or more of the chosen benchmark datasets---the prior will be fundamentally mis-specified. The use of the mode for MAE should be better justified; while it coincides with the median/median of the normal distribution, this is not true more generally.\n\nOn a related note, the beta distribution has more than one parametrization, and so (without seeing the prompt, which is missing from the paper), it will be impossible to disentangle the performance of the LLM from its 'understanding' of the choice of parametrization. Indeed, the first line of the appendix hints at an issue that might be related, and which may have been resolved using 'function calling' capabilities to constrain the format of the output.\n\nThe choice of **expected calibration error** (ECE), based on four coarse bins seems like reinventing the wheel. The construction of this metric is ad-hoc, lossy and non-standard for this type of analysis. A graphical method such as a QQ plot would have provided a far more rigorous and informative assessment of calibration. Splitting data into quarters (incorrectly referred to in the paper as 'quartiles', but quartiles/quantiles are *points*, not *intervals*; a common mistake) seems arbitrary and discards information within those bins. The number of models being compared is small enough to allow for a visual method, and if not then other, more granular numerical metrics are available, as well as standard statistical tests for comparison of empirical and expected distributions.\n\nPoor visualization. Bar charts with error bars, sometimes called \"dynamite plunger plots\", should never be used, and hide the true distribution of performance across the different tasks. Box plots or simple dots and error bars would be better. See Drummond & Vowler (2011; doi:10.1111/j.1476-5381.2011.01251.x).\n\nThe distinction between 'reasoning' and 'non-reasoning' models in the ablation study needs to be more clearly defined and justified. What is the basis for classifying, e.g. `GPT-4o` as 'non-reasoning'?\n\nSome related works need to be mentioned in the literature review. This is not the first work to propose eliciting parametric Bayesian priors from LLMs, see for example Selby et al (2025; doi:10.1002/sta4.70054), which explores the problem of evaluating the quality of LLM priors on real-world datasets. Data augmentation approaches (i.e. sampling pseudo-observations from the LLM and then looking at the empirical distribution) should also get a mention: see Huynh et al (2025; https://openreview.net/forum?id=2Q3gFNbpAr).\n\nFinally, there are one or two minor typos: e.g. \"empricial\" at bottom of page 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)