Uncertainty Quantification for LLM-Based Survey Simulations
Authors: Chengpiao Huang
Yuhang Wu
Kaizheng Wang
Venue: ICML 2025 poster (2025)
Keywords: synthetic data
large language models
uncertainty quantification
simulation
Primary area: general_machine_learning->evaluation
TL;DR: Uncertainty Quantification for LLM-Based Survey Simulations

Abstract: We investigate the use of large language models (LLMs) to simulate human responses to survey questions, and perform uncertainty quantification to gain reliable insights. Our approach converts imperfect, LLM-simulated responses into confidence sets for population parameters of human responses, addressing the distribution shift between the simulated and real populations. A key innovation lies in determining the optimal number of simulated responses: too many produce overly narrow confidence sets with poor coverage, while too few yield excessively loose estimates. To resolve this, our method adaptively selects the simulation sample size, ensuring valid average-case coverage guarantees. It is broadly applicable to any LLM, irrespective of its fidelity, and any procedure for constructing confidence sets. Additionally, the selected sample size quantifies the degree of misalignment between the LLM and the target human population. We illustrate our method on real datasets and LLMs.

Uncertainty Quantification for LLM-Based Survey Simulations
Chengpiao Huang' Yuhang Wu?
Kaizheng Wang'
Abstract
survey question, and collecting the LLM's i responses. Compared to traditional survey methods that recruit and query real people, LLM simulations offer significant ad- vantages in terms of time and cost efficiency, enabling the generation of large-scale synthetic responses with minimal effort.
We investigate the use of large language models (LLMs) to simulate human responses to survey questions, and perform uncertainty quantification to gain reliable insights. Our approach converts imperfect, LLM-simulated responses into confi- dence sets for population parameters of human responses, addressing the distribution shift be- tween the simulated and real populations. A key innovation lies determining the optimal num- ber of simulated responses: too many produce overly narrow confidence sets with poor coverage, while too few yield excessively loose estimates. To resolve this, our method adaptively selects the simulation sample size, ensuring valid average- case coverage guarantees, It is broadly applicable to any LLM, irrespective of its fidelity, and any procedure for constructing confidence sets. Ad- ditionally, the selected sample size quantifies the degree of misalignment between the LLM and the target human population. We illustrate method on real datasets and LLMMs.
However, a growing body of evidence suggests that LLMs are not perfectly aligned with the human population, and in some cases, the misalignment can be substantial (Aher et al., 2023; Santurkar et al., 2023). This raises critical concerns about the reliability of insights derived from LLM- generated data. remains a challenge how to properly simulate human responses using LLMs and how to account for their imperfections when using the simulated samples to make inference about the true human population.
We propose to address this challenge through the lens of uncertainty quantification. Specifically, we seek to construct confidence sets for population statistics of human responses based on LLM-generated data. A central question this process is:
How many synthetic samples should be generated?
On one hand, generating too many samples risks overfitting the synthetic distribution, which may deviate from the real human population. On the other hand, generating too few samples yields overly large and uninformative confidence sets. The optimal sample size depends on the discrepancy between the synthetic and real populations - a quantity that unknown practice. This necessitates data-driven approach determine the appropriate number of simulated responses.
1. Introduction
Large language models (LLMs) have demonstrated remark- able capabilities in mimicking human behaviors. Recent studies have leveraged LLMs to simulate human responses various domains, including economic and social science experiments (Aher et al., 2023: Horton, 2023; Chen al., 2023; Bisbee et al., 2024; Huang et al., 2024; Yang al., 2024; Ziems al., 2024), market research (Brand et al.,
2023; Gui & Toubia, 2023; Goli & Singh, 2024; Wang et al.,
Main contributions. In this paper, we develop general framework to address these challenges. Our key contribu- tions are as follows:
2024), education (Zelikman et al., 2023; Lu & Wang, 2024), and so on. The typical simulation procedure consists in prompting an LLM with a real or fictional persona as well
'Department of Industrial Engineering and Operations Re- search, Columbia University DDecision, Risk, and Operations Division, Columbia Business School 'Data Science Institute, Columbia University. Correspondence to: Kaizheng Wang <kaizheng wang@ columbia.cuu.........
(Formulation) We provide rigorous mathematical framework for uncertainty quantification in LLM- based survey simulations.
(Methodology) We propose a flexible methodology that transforms simulated responses into valid confidence sets for population parameters of human responses. Our approach adaptively selects the simulation sample
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Uncertainty Quantification for LLM-Based Survey Simulations
size based the observed misalignment between the LLM and human populations. It is applicable any LLM, regardless of its fidelity, and can be combined with any method for confidence set construction.
We write an (b,,,) if on = (b,,) and on Q(b,,). The notation Bernoulli(p) denotes the Bernuulli distribution with mean p. The notation N (u, denotes the normal distribution with mean μ and variance o².
Related works. Our work relates to research on assess- ing the fidelity of LLM simulations and measuring their alignment with real human populations. Prior studies have explored similarity metries between synthetic and human distributions (Santurkar et al., 2023; He- Yueya et al., 2024; Doninguez-Olmedd et al., 2024; Durmus et al., 2024; Calderon al., 2025) and Turing-type tests (Argyle al., 2023; Mei et al., 2024) evaluate LLM reliability. While these approaches provide valuable insights into LLM mis- alignment, they do not offer methods for leveraging imper- feet LLM simulations to draw reliable conclusions about human populations. In contrast, our work provides a princi- pled approach for constructing confidence sets that account for the inherent discrepancies between LLM--generttdd and human responses.
2. Warm-up: Binary Response Simulation
To motivate our problem and methodology, we will start with simple setting where an LLM simulates binary re- sponses to a survey question. In Section 3, we will present the general problem setup and the general methodology.
2.1. Motivating Example: Educational Test
Suppose a school wants estimate the proportion μ E [0, 1] of students that can answer newly designed test question correctly. It will not only provide insights into student progress but also evaluate the question's 's effectiveness in differentiating among students with varying levels of under- standing. Such information can guide the school in tailoring teaching strategies to better address student needs.
Additionally, our work connects to the line of work model-free statistical inference, including conformal infer- ence (Vovk et al., 2005; Shafer & Vovk, 2008; Bates et al., 2021; Angelopoulos et al., 2024; Kim et al., 2024) and prediction-powered inference (Angelopoulos et al., 2023). At a high level, these methods use labeled data from the true distribution to calibrate imperfect point predictions from an arbitrary black-box model and then construct valid set estimates. Our approach follows similar spirit. The "fea- tures" and "labels" in our setting correspond to the survey questions and their population statistics of human responses, respectively. However, our labels are not directly observ- able. As a result, the labeled calibration data needed for these methods is not available. Moreover, for every simu- lation sample size k. one can produce point prediction of the label using& k synthetic responses generated by the LLM. As the optimal sample size is not known priori, there are infinitely many candidate point predictions to choose from. This makes it difficult to apply existing statistical inference methods.
The most direct approach is to give the test students and collect their results y!, Un E {0. 1}. where indicates whether student i answers the question correctly. A point estimate for μu is the sample mean ū = 1 a E (0, 1), we can construct a confidence interval for jr:
n Ei Vr. Given
[ e (1
00 (1-2) 2
3 x Vn
(1- 2
(1)
Vn
where V 5( 7) is the sample standard deviation, and + is the cumulative distribution function (CDF) of (O, 1). By the Central Limit Theorem (CLT), this interval has asymptotic coverage probability 1 - a as n - oc. As a different approach, one can also use Hoeffding's 's concen- tration inequality (e.g.. Theorem 2.8 in (Boucheron et al., 2013)) construct a finite-sample confidence interval
log (2/a)
log(2/a)
1
+
(2)
Outline. The rest of the paper is organized follows. Section 2 studies binary response simulation as a warm-up example. Section 3 presents the general problem setup and methodology. Section 4 illustrates our proposed method on real datasets. Section concludes the paper.
2n
2n
which has at least (1 a)) coverage probability for every n E Σ4. For simplicity, we will stick to (1) in this section.
Alternatielly, the school may use an LLM to simulate stu- dents' responses to the question. Compared with directly testing on real students, this approach is more time-efficient and cost-saving. If we prompt the LLM k times with ran- dom student profiles, then it generates k synthetic responses, which leads to synthetic outcomes y."h :..., 12 E {0, 1}.
Notation. We use Z4 to denote the set of positive integers. For n E Z4, define [n] = .t...... For a, b E IR, define a A 6 = min(a, 6} and a V b = max(a, b). For non-negative sequences {a, 1 and (b, 1 1. we write an = O(b)) if there exists C > 0 such that for all n, it holds that an S Cbn. We write an ((b,)) br = (a,)).
We may also compute the sample mean syn Di= E yi syn
2

---

Uncertainty Quantification for LLM-Based Survey Simulations
and the CLT-based confidence interval
1. The choice o offers valuable information for future simulation tasks on the appropriate number of syn- thetic samples generate, so as to produce reliable confidence intervals. It also helps avoid generating excessive samples and improves computational effi- ciency.
syn 5k
(k)
M/
syn
C.
1
a 2
v k
c. kk
-- (1 -1- | (3)
Uk
+
Vk
2.
The width of (>y((k) provides an assessment of the alignment between the LLM and the human popula- tion. A wide confidence interval indicates high uncer- tainty of its estimate of the true μ., and thus a large gap between the synthetic data distribution and the true population.
where s/tT / 01 Thth and c is a scaling parameter. Such dilation by e necessary; without it, whenever the LLM-generated data deviates from the stu- dent population (even by the slightest amount), the interval IP9 (k) may never achieve (1 - a) coverage regardless of k. We give an example in Appendix B.I.
3. The sample size T: reflects the size of the target popula- tion that the LLM can represent. We make an analogy using the classical theory of parametric bootstrap. Sup- pose a model is trained via maximum likelihood esti- mation over l i.i.d. human samples. When performing parametric bootstrap for uncertainty quantification, the bootstrap sample size usually set be the training sample size k. Thus, our simulation sample size k re- veals the LLM as "being made up of" h people from the population. We provide a visualization in Figure 4 of Appendix A. The larger & is, the more diversity that the LLM appears to capture. In contrast, small k could imply the peculiarity of the LLM compared to the major population.
Due to the misalignment between the LLM and students, the distribution of the synthetic data {u. J==1 ", may be very
different from the true response distribution. In this case, the sample mean ynT can be a poor estimate of μ., and Isy (A)) is generally not valid confidence interval for μ. In particular, as k + oo, the interval concentrates tightly around the synthetic mean E[y]"] and fails to cover the true mean μ. On the other hand, when k is small, the interval becomes too wide to be informative, even though it may cover z with high probability. We provide illustration in Figure 1.
1.0
0.8
Remark 2.1 (Comparison with existing works). Existing works typically measure LLM misalignment using inte- gral probability metrics and f-divergences (Santurkar et al., 2023; Dominguez et al., 2024; Durmus et al., 2024), which do not carry operational meanings themselves and can be hard interpret. In contrast, our simulation sample size k provides actionable guidance and is easy to understand.
0.6
0.4
0.2
2.2. Methodology for Selecting Simulation Sample Size
0.0
We now introduce our method for choosing a good simula- tion sample size k. It makes use of similar test questions for which real students' results are available. If such data is available, then we can compare LLM simulations with real students' results on these questions, and use to guide the choice of k:
Figure 1. The widerage-width trade-off for the simulation sample size k. The true distribution is Bernoulli(0.4) and the synthetic distribution is Bernoulli(066) The horizontal axis the simula- tion sample size k. The red dotted horizontal line plots the true mean μ = 0.4. The blue curve plots the sample mean y2" the synthetic data, and the blue shaded region visualizes the confidence interval (K)), for E [40). When is too small (say & S 6), the interval (k) is too wide. When & is too large (say l 2 18), (1)) fails to cover j.
Specifically, we assume access to test questions similar to the question of interest. For example, they can come from previous tests or question bank. For j E [m], the j-th test question has been tested on n, real students, with test results D - We also simulate LLM responses 5yyn ( Mi=1 to the j-th test question, and Daym = {u.T*) J to the test question of interest. Here K E E4 is the simulation budget.
Main insights. Our goal in this work is to develop a princi- pled approach for choosing a good simulation sample size R; so that Zoym (k) is a valid confidence interval for μ while hav- ing a small width. Solving this problem has the following important implications.
k
For each question j E [m], we form confidence intervals
3

---

Uncertainty Quantification for LLM-Based Survey Simulations
similar (3) using the synthetic data D aiming to cover the true proportion My of students that can answer the j-th question correctly:
2.3. Theoretical Analysis
In this section, we present a theoretical analysis of our pro- posed method. To do so, we first describe the setup in Section 2.1 and Section 2.2 in mathematical terms.
C. syn j.k
I (R:) =
ayn jikk
+ -1 (1-2
2 2
kk
The student population can be represented by a distribution P over space 2 of possible student profiles, say, vectors of background information, classes taken, grades, etc. To simulate student responses from the LLM, synthetic student profiles are generated from a synthetic student population payn over 2, and then fed to the L.M..
syn
+
c.
J,k
syn
-1-1) (4)
].
V
where 07,0 = Σ syn is the sample mean of the first k samples in n and y v 01 ) is the es- timated standard deviation. We also set the convention I (0) = - (0) = [0, 1]. as nothing can be said about the true parameter without data. We will pick & E {0,, } such that Iz5"(A (7) covers 24 with high probability. We ex- pect this choice of to be also good for (k)),,, as the test questions are similar.
We use &b and { to denote the test question of interest and the m similar ones, respectively. Students' performance on test questions is characterized by a performance func- tion F: a student with profile = E Z answers a question e correctly with probability F(z, .b) E [o, 1]. The average student performance on the test questions / and (155) are then μ F22, vb) and μ4 E 15),, re- spectively. In addition, the LLM generates synthetic stu- dent performance from a synthetic performance function Fsy: when prompted with a synthetic profile g E Z, the LLM answers a question &e correctly with probability psyn (= (b) E [0. 1].
Ideally, we would like pick k such that (1 coveraae is achieved empirically over the test questions:
1
11g & IT"" S a.
(5)
The collection of the real dataset Dj = {yg can be thought of as drawing ny i.i.d. student profiles (= 1 'P and then sampling Vj.i ~ Carmoulli(P(z.:: v)))) for each E [ng]]- Similarly, the generation of the synthetic syn R
As the true ( are not available, we use the real data
1mmh {D)} to compute the sample means y, ny as proxies for nz. The empirical miscoverage can be ap- proximated by
1 Σ Vj,1
dataset D = {o M==1 can be thought of as drawing
i..d synthetic profiles {2 Ji=l paan and then sampling vj.4 Bernoulli(/<syncC Syn
K
1 Σ 10, & (T (k)).
f. vby)) for each i E [K]. For we adopt a similar notation (= for the synthetic profiles. We note that when collecting real or synthetic samples, the performance functions never ap- pear explicitly. They are introduced only to facilitate the problem formulation.
G(k)
(6)
Dann = {u, k
Our criterion for selecting A is given by
K max {0 S S K: G(1) S a/2 vi Vis k}. (7)
Note that ie is well-defined because G(0) 0.
Finally. we assume that the test questions are drawn ran- domly from a question bank, and that the datasets are inde- pendent.
The choice of the threshold a/2 in (7) can be explained as follows. By CLT, when ny is large, F(2; 2 ūj) & 1/2. Suppose My & I (l)), then μJ is either on the left or the right of 175m(()), In the former case, Øj is on the left of ny with probability around 1/2, which implies y, & I (k). Similarly, in the latter case, Fj is on the right of (i with probability around 1/2, and then Øj & 1 (k). Roughly
Assumption 2.2 (Randomly sampled questions). The ques- tions v,, $/, ..., pm are independently sampled from a distri- bution over a space V.
Assumption 2.3 (Independent data). For each j E [m]] conditioned on vyy, the datasets D, and j are in- dependent. Conditioned on $2...., Vmm the dataset tu- ples (Di, ) (Din, Dex") are independent. Finally, (1), DDvn') is independent of {w,, D,, D2yn) ")
speaking, the frequency of having Dj E I I.5m (k)) is at least half of the frequency of having 2j & I (K). In other words, the lower bound
1 - M 1 Mm $ (k)} G(k) 2
m
We are now ready state the theoretical guarantee of our approach. Its proof is deferred to Appendix B.2. We note that the assumption P(ù; < Mj I Φb)) = 1/2 is a CLT approximation and is for mathematical convenience only.
(8)
2
m j=1
approximately holds. Substituting (8) into (5) yields the
threshold a/2 for choosing R.
Theorem 2.4 (Coverage guarantee). Let Assumptions 2.2
4

---

Uncertainty Quantification for LLM-Based Survey Simulations
and 2.3 hold. Assume that P(ū S by 1b) = 1/2 for each j E [m]] Fix a E (0, 1). Then the simulation sample size io
sollowing holds with probability at least 1 8:
2
defined by (7) satisfies
1 2 min
K, n,
5A
P P(
21-a (
2
When this happens, the selected confidence interval Izyn (k)
has width (max{ (oaxx[-1/- -1/2,, , 1/2 A)).
The probability is taken with respect to the randomness of {1b,, Dj, )) &b and D
Theorem 2.6 implies that the interval Iunm (k) is the shortest possible. To see this, suppose that the simulation budget K is large, then with high probability, I has width ( (A, n- n// This is the optimal width because of the following reasons. First, in the worst case, any I (k) that covers the true mean with high probability must have width 2( in order to address the distribution shift be- tween the real and simulated responses. Second, as n real human responses can identify the true mean up to an error of Onn-1/2), then any valid [ (k) must also have width This shows the sharpness of the chosen sample size and the confidence interval Iayn (k).
On average, the chosen simulation sample size ie leads to a confidence interval I (k) that covers the true mean je with probability at least 1 - a - Ol /Lm) As m + co, the aforementioned lower bound converges to 1 -1--0 a.
2.4. Sharpness of Sample Size Selection
We have seen that the chosen interval Z (k) has good coverage properties. In this section, we complement this result by showing that the interval is not overly conservative. To simplify computation, we slightly modify the setting.
Example 2.5 (Gaussian performance score). Consider the setting in Section 2.3 with the following modifications. On test question & E V, the performance (e.g., score) of real student follows a Gaussian distribution with mean (=>pP(>z e) and variance 1, instead of Bernoulli dis- tribution with mean (z, Similarly, the perfor- mance of the LLM follows a Gaussian distribution with mean po (2 ()) and variance 1. Moreover, the confidence intervals (zy((A) defined in (3) and 1,5m((( (k) de-
3. General Setup and Methodology
In this section, we study the more general setting where sur- vey responses and confidence sets can be multi multi-dimensional.
3.1. Problem Formulation
Let Z be profile space, P probability distribution over Z which represents the true population, and payn a synthetic distribution over 2 used to generate synthetic profiles.
fined in (4) are changed to
Let y be a collection of survey questions, and y be the space of possible responses the survey questions. When person with profile = E 2 is asked a survey question v E V the person gives response following distribution 2( - 1 2, v)) over V. We are interested in the distribution of the population's 's response to the survey question th, which is given by R( 1+) fz 2 - z, vb) P(z)). In particular, we seek to construct a confidence set for some statistic ((1) of R( | vb. which can be multi-dimensional, say in Rd. Below
C
(k) =
Uk
UR
syn
Vk
Vk
C
C
(k))(k)=
Vj.k
syn
DEK
sy
Vi
respectively, where C' = 2 (1 a/4). For simplicity, we suppose that the real datasets have the same size: nij- = n for all j [mi. Finally, we define
revisit the educational test example in Section 2 in this framework. More examples are provided in Appendix A.
A = sup EE>p v)- - EEr (2 }. NEW
Example 3.1 (Educational test evaluation). In the educa- tional test evaluation example in Section 2, each v e Z is a student profile, each & E Ti is a test question, the response space is y {0, 1}. and Q(
In Example 2.5, the quantity A measures the discrepancy between the distributions of the real students' performance and of the simulated students' performance. The following theorem presents lower bound on the chosen simulation sample size k. Its proof can be found in Appendix B.3.
2, v)
Bernoulli(F(2, vb)). The statistic ((v) is the probability of student answering the question correctly: B Eg-p [2,, 4)].
Theorem 2.6 (Sharpness of chosen sample size). Consider the setting of Example 2.5. Let ie be chosen by the procedure (7). Choose& E (0, 1). There exists a constant C' > 0 determined by a such that when m > C' loog(n/8) the
We consider constructing the confidence set by using simu- lated responses from an LLM. Given a profile 2, a survey question 1 and a prompt p. the LLM simulates response
5

---

Uncertainty Quantification for LLM-Based Survey Simulations
py from a distribution Quyn( . 2, p) which aims to
each E [m]. This choice of l will also be good for (k), thanks the i.i.d. assumption on the survey questions.
mimic 2(- 2, 2.))). We can generate i.i.d. synthetic pro- files {2²") Ji=1 ~ payn, then feed them into the LLM along with u and The LLM then generates synthetic responses (ur" syn K 1i=1- where v;" ~ Q8y"( syn v).p). Here K E Z4
Ideally, we would like to pick k such that (1 - a) coverage is achieved empirically over the m survey questions:
is the simulation budget.
m Σ 1(0(v)) & 55" Sa. a. TTL
1
(1I)
Using the simulated samples D { construct a family of candidate confidence sets such as the one-dimensional CLT-based confidence interval (3). More generally, the statistics literature has developed variety of approaches such as inverting hypothesis tests (Casella & Berger, 2002), the bootstrap (Efron, 1979), and the empirical likelihood ratio function (Owen, 1990). We will assume access to a black-box procedure C that takes as input a dataset D and outputs a confidence set C(D) C Rd. Then, we can construct family of confidence sets ((5m(())) by
K J==1- we can
However, the population-level quantities {0(wy)) are not available, so we must approximate them by the real data {- In Section 2, we have taken the approach of constructing unbiased point estimates, but it does not directly extend to the more general case.
Instead, we will use the real data {Da} to construct confidence sets for {8(b))) Choose a confidence level 7 E (0, 1). For each j E [m], we use D, to construct a confidence set Sj that satisfies
(km C (Lv, )==)
(9)
We also set (0) = Rd, so (())) E (0) always. We will not impose any assumptions on the quality of the confi- dence sets produced by C.
P P(B(W)) E Sj 27 2%
(12)
These confidence sets are easy to construct as the samples in Dy follow the true response distribution. When ((v))) e
As the LLM may not be a faithful reflection of the true human population, we will make use of real data to choose good confidence set from (SSy ( We assume that we have collected real human responses from m surveys 2/..... Vm E V. For each j E [m]] we have responses Dj = { Th from ny i.i.d surveyees {2 ~ P, with yy.a ~ 2(- 22,8, 4)y).
S, the condition S, S S) (k) is sufficient for ((1))) (pn) Equivalently, when ((v))) E Sj. the condition
((v)) & (k) must imply S, & (k). Thus, we take
1 Σ 11,, z S (b) L(k) = m j=1
(13)
proxy for the empirical miscoverage. Since ((vly) S, with probability y. then the frequency of having S, g S (k)) is at least times the frequency of 0(w,) & St (k).
We also simulate LLM responses these m survey ques- tions. For each; j E [mi]. we feed i.i.d. synthetic profiles [ Ti=1 ~ -psyn, the question 1/y and the prompt p into the LLM, which then simulates responses Dmm {353 syn K syn ²j.a (). The datasets {D,}
{2
Roughly speaking,
with syn ~ @3ym(-
1
m
1((())) & S5 (4)) (0))).
and (D J==1 will be used to select a confidence set from
L(k) 27:
(14)
m
(SSH) ( km We make the same Assumptions 2.2 and 2.3
as in Section 2.
Combining (11) and (14) leads to the following criterion for selecting k:
We are now ready to formally state our problem. Problem 3.2 (Uncertainty quantification). Given (0, 1), how to use {D, and (D Im to choose io E [K] such that
T max (OSkSK: : 1(1) S ya, vi S sk.. (15)
Note that : is well-defined because L(0) = 0. The full procedure for sample size selection is summarized in Algo- rithm 1.
IP (
E e (k))
3.2. General Methodology for Sample Size Selection
We now present the coverage guarantee for our method, which shows that the chosen confidence set Styn has coverage probability at least 1 - a - 0()/mm. Its proof is deferred Appendix B.4.
We now present our general methodology for Problem 3.2.
For each j E [m]. we form confidence sets similar to (9) using the synthetic data D
Theorem 3.3. Let Let Assumptions 2.2 and 2.3 hold. Fix e (0,1). The output i of Algorithm I satisfies
5 (k) = C (10 HL), Wk E [K].
(10)
We also set (0) = Rdd. We will pick &e E (0, 1,.., K} such that (k))((k) is good confidence interval for ((b,,) for
p(((w) E (ST)(()) 21 21-a-1 a1-a-1-1
2m
6

---

Uncertainty Quantification for LLM-Based Survey Simulations
Algorithm 1 Simulation Sample Size Selection
Panel', and contains the general US population's responses to survey questions spanning topies such as science, politics, and health. After pre-processing we have 385 unique ques- tions and 1,476,868 responses to these questions from at least 32,864 people. These questions have 5 choices corre- sponding to ordered sentiments which we map to sentiment scores -1, -11,-3:00 0, 3: 1. Each question has at least 400 re- sponses. For each response, we have information on the surveyee's political profile, religious affiliation, educational background, socio-economic status, etc., which we use to generate synthetic profiles. See Appendix C.1 for more details. We consider the task of constructing a confidence interval for the US population's 's average sentiment score for survey question. This the setup in Example A.3.
Input:
Survey questions with real and simulated re- ه= preseribed miscoverage probability a, confidence set construction procedure C, confidence level 7,, simulation budget K.
sponses {(by, Dj, Din))))) }
j = l..... m do
Use and " to construct synthetic confidence sets km
{ [k)) by (10).
Use Dj construct confidence set S, satisfying (12).
end for Define
L(k)=
Σ m j=1
15S 2 (k))
The second dataset is the EEDI dataset created by (He- Yueya al., 2024), which was built upon the NeurIPS 2020 Education Challenge dataset (Wang et al., 2021). It consists of students' responses to mathematics multiple-choice ques- tions on the Eedi online educational platform². The dataset contains 573 unique questions and 443,433 responses to these questions from 2,287 students. All questions have four choices (A, B, C, D). Out of these questions, we use ques- tions that have least 100 student responses. Excluding questions with graphs diagrams, we are left with total of 412 questions. For each student, we have information on their gender. age, and socioeconomic status, which we use to generate synthetic profiles. See Appendix C.2 for more details, We consider the task of constructing a confidence interval for the probability of student answering question correctly. This similar the setup in Example 3.1.
Set h max (0 < k S K L(i)= S ya, vi s k)
Output: Sample size k
worth noting that our method achieves this coverage without any assumptions the qualities of the LLM and the procedure C for confidence set construction. Nevertheless, the size of the chosen confidence (*x((k) depends these factors. If there is large alignment gap between the LLM and the human population, then Sayn (k) will inevitably be large.
4. Numerical Experiments
Confidence set construction. Our method can be built upon any arbitrary confidence set construction procedure C. We use a construction procedure based on Hoeffding's concentration inequality (e.g.. Theorem 2.8 in (Boucheron et al., 2013)), as it has valid coverage guarantee for any finite sample size. Given a E (0, 1) and responses {v." syn 12=1- we construct the confidence interval
In this section, we apply our general method in Sec- tion to LLMs over real datasets. The code and data are available https: /githb...... 11m-survey- i mul ation.
4.1. Experiment Setup
LLMs. We consider LLMs: GPT-3.5-Turbo (gpt- 3. 3.5-b, GPT-4o (gpt-4o), and GPT-4o-mini topt-4o-ming (OpenAl, 2022; 2024a;b); Claude 3.5 Haiku
log(2/a)
cMV
2k
(Anthropic, 2024); Llama 3.1 8B (Llama- 1aama-3-BB-a0 -3-BB-
log(2/a)
syn yk
+ cM
(16)
2k
Turbo) and Llama 3.3 70B
1.mma-3.3700
Instruct-Turbae (Dubey et al., 2024); Mistral 7B
where =
8Ith is the sample mean, M > 0 is an upper bound on the range of the responses, and c > is scaling constant.
Milstra--B-Innttrue-- 3) (Jiang al., 2023): DeepSeek-V3 (DeepSeek-v3) (Liu et al., 2024).
Hyperparameters. c = 22 and = 0.5. For the EEDI dataset, we set the
We consider a E (0.05 l E [10)),
Datasets. We use two datasets for survey questions, each corresponding to one uncertainty quantification task. The first dataset the OpinionQA dataset (Santurkar et al., 2023). It was built from Pew Research's 's American Trends
american-trends-pane
/nttps://eed com/

---

Uncertainty Quantification for LLM-Based Survey Simulations
simulation budget K 50 and take M 1 since the responses are binary. For the OpinionQA dataset, we set the simulation budget K = 100 and take M = 2 since the responses range within [,,,]] 1].
4.3. Experiment Results
For both datasets, we evaluate three metries as a varies: the miscoverage probability proxy (18), the selected simulation sample size k, and the half-width of the synthetic confidence interval (k)()).. We consider 100 random train-test splits of the questions. For compactness, we present results on the miscoverage probability proxy and k. and defer the results on the half-width of the synthetic confidence interval as well as more experiment details to Appendix D. We omit Llama 3.1 8B for the EEDI dataset experiment because it fre- quently failed to answer EEDI questions in required formats. As a baseline, we also include naive response generator (random) that chooses an available answer uniformly at random.
4.2. Experiment Procedure
We now describe experiment procedure for applying the method in Section 3 to each dataset. Denote the dataset by {(1)) D;)} where 1 is survey question and Dj- = } J is a collection of human responses. For each E [J], we simulate K responses Dinn from an into
LLM. We then randomly split 9- [(2,, (D),, a training set g0r = {(D), (D)))) and a testing set gi = {(D), ()))) with [0*1: : gr = :
Selection of simulation sample size. We apply the ap- proach (15) with the training set 9r' to select a simulation sample size k. For the confidence set Sj in (12) constructed from the real data Dj. we use the standard CLT-based confi-
Coverage validity. In Figure 2, we present histograms of p-values for the hypothesis test [[[(k)] S a against E [(]] > a across various LLMs and a's over the Opin- ionQA and EEDI datasets. The p-values are computed using a one-sided z-test over the 100 random splits. As can be seen from the histograms, all p-values are reasonably large, indicating that the hypothesis E [[))) S a cannot be re- jected (e.g., at the 0.05 significance level) for any LLM and a across both datasets. These experiment results verify the theoretical guarantees in Section 3, showing that the miscoverage rate is effectively controlled by our method.
dence interval:
8j
+ 7
=
4
n
2
+ 8j --1
1 + 7 2
Øj
(17)
Vnj
where 83 =
n. En Uy. and sy = v (1 - V,). Since nj
is at least 100, S, has approximately y coverage probability. In Figure 5 of Appendix C.3, we provide a visualization for the selection of k.
Selected simulation sample size. In Figure 3, we plot the average to over the 100 random splits for various LLMs on the OpinionQA and EEDI datasets, respectively. The error bars represent 95% confidence intervals.
Evaluation of selected sample size. We use 2he to eval- uate the quality of the chosen simulation sample size k. As the true population parameter ((1)) is unavailable, the true coverage probability ((((()) E styn(k)) (k)) cannot be computed. However, we can apply the same idea as (13) in Section 3 to compute a proxy for the miscover- age level. For each survey question j E Tre. the se- lected sample size i leads to the synthetic confidence set (STM(Z) = Cty
In general, a larger &o means that the LLM has a stronger simulation power. On the OpinionQA dataset, GPT-4o has the best performance. On the EEDI dataset, DeepSeek- V3 has the best performance, followed by GPT-4o and Claude 3.5 Haiku. Interestingly, on the OpinionQA dataset all LLMs elearly outperform the random benchmark, while on the EEDI dataset only DeepSeek- V and GPT-4o seem to outperform the random benchmark. Moreover, LLMs exhibit uniformly higher on the OpinionQA dataset than on the EEDI dataset, suggesting higher fidelity in simulating subjective opinions social problems than in simulating student answers mathematics questions.
1 s Ji=z). We form the confidence set Sy from real data D, as in (17) and define
Ī(k) 1
1
Σ JE.T.n
11s, 2 (K)).
(18)
Jrel
The experiment results demonstrate the importance of a disciplined approach to using synthetic samples. The ease of LLM-based simulation makes it tempting to generate a large number of responses per question. However, our results show that there is great heterogeneity in the simulation powers of different LLMSs over different datasets: the largest to is below 100, while the smallest & could be in the single digits. This means that there is real peril in using too many
The proof of Theorem 3.3 shows that, for every k e [K] and survey question j,
7- P((()) o (k)) s SP(S, & Sg (k)) = E[E([())
Thus, if E [(x) S a, then (((()) & SSmm < a must hold. To this end, we will test hypothesis Ho :
E [[[.(K)] S a against its alternative H1 [T[((K)] > a.
8

---

Uncertainty Quantification for LLM-Based Survey Simulations
gih-1--ttrte
4pt40 min
Uama
Mistrer78
Deep Seecly
Frequency
random
132
20
8.5
0.25
0.25
0.55
0.50
p-value
12
3.5
11
cleudeec---heicc
Liam 708
Mistral-78
Deeptweke
random
9-
Frequency
6.85
0.75
6.85
02
0.3
0.4
0.5
v-vuuue
Figure 3. Average Ie for various LLMs over the OpinionQA dataset (top) and the EEDI dataset (bottom).
Figure 2. Histograms of g-values for the hypothesis test E [[k]] < a against B [[k] > a across various LLMs and a's over the OpinionQA dataset (top) and the EEDI datasct (bottom).
sets with good coverage. Second, would be interesting to see if our approach can combined with debiasing meth- ods give more informative confidence sets. Finally, as prompt engineering is known to have crucial effects on the quality of LLM generations, it is worth investigating the impacts of prompts on the selected simulation sample size k, and how prompt engineering can be leveraged to improve the fidelity of LLM simulations.
synthetic samples and being overly confident in the results.
5. Discussions
We developed general approach for converting imperfect LLM-based survey simulations into statistically valid confi- dence sets for population parameters of human responses. It identifies simulation sample size which is useful for future simulation tasks and which quantifies the fidelity of I.LM simulations. Numerical experiments on real datasets verified coverage guarantees our approach, and revealed that existing LLMs exhibited higher fidelity in simulating opin- ions to social problems than in simulating student answers to mathematics questions.
Acknowledgements
Chengpiao Huang and Kaizheng Wang's research is sup- ported by an NSF grant Institute seed grant SF-181 Columbia University.
and a Data Science
Impact Statement
Several future directions are worth exploring. First, our approach does not explicitly minimize the size of the predic- tion set. A natural question is whether we can incorporate a size minimization procedure to produce smaller confidence
This work develops methodologies for uncertainty quantifi- cation for LLM-simulated survey responses. We believe that it can facilitate more reliable LLLM-simulate experi-

---

Uncertainty Quantification for LLM-Based Survey Simulations
ments in areas such as social science, economies, marketing, and education. On the other hand, we point out that the theoretical guarantees for our method hold only on average over the randomness of the survey questions and the real and simulation responses, and do not necessarily hold for every random instance. Therefore, we recommend caution when applying the confidence intervals to decision-making
URL https://doi. 1093/ / /acprof: 0S0/97801995352555 000.0011.
Brand, J., Isracli, A., and Ngwe, D. Using LLMs for mar- ket research. Harvard Business School Marketing Unit Working Paper, (23-062), 2023.
Calderon, N., Reichart, R., and Dror, R. The alterna- tive annotator test for LLM-as-a-jjudg:: How to statis- tically justify replacing human annotators with LLMs. arXiv preprint arXivv25001110990 2025. URL https: http://aabb//01111000.9
References
Aher, G. V., Arriaga, R. I., and Kalai, A. Using large language models simulate multiple humans and repli- cate human subject studies. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pp. 337-371. PMLR, 23-29 Jul 2023. https:: /ttp///ooodiicceee ngs mlr p.rsss////// v202 2.02/aher23a html.
/
Casella, G. and Berger, R. Statistical inference. CRC press, 2002.
Chen, Y. Liu, T. X., Shan, Y., and Zhong, S. The emergence of economic rationality of GPT. Proceedings of the National Academy of Sciences, 120651):62316205120 2023. doi: 10.1073/pnas. 2316205120, URL https: /tt:///wwww./... http://wwwaaaa////////..a/../..t .rr.oo/g/oooooo /doi/ abs/10. Abbs/10.1033ppas .2lass.23111 231 1205222...
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zmicc, T. perdiction-powette inference. Sci- ence, 382(6671 (:60))1))666-6666 676, 2023. doi: 10.1 166sccccceee adi6000. URL htRL:ttp: /www. science. org/doil abs/ htt.11////ccene..aii..... 6000.
Dominguez-Olmeddo R., Hardt, M., and Mendler-Dünner, C. Questioning the survey responses of large language models. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL ht https: / //openreiiee net. forum? 7d
Angelopoulos, N., Bates, S., Fisch, A., Lei, I.., and Schuster, T. Conformal risk control. In The Twelfth International Conference on Learning Representations, 2024. URL sttps://openzzevirrs wtw.eewww.e//ooounnneee id=33XGfHLt2g.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv preprint arXi::240722178. 2024.
Anthropic.
Introducing computer use, new Claude 3.5 Sonnet, and Claude 3.5 Haiku, 2024. URL
Durmus, E., Nguyen, K., Liao, T., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Saaffeld-Dodde Z., Herman- dez, D., Joseph, N., Lovitr, L., McCandlish, S., Sikder, O., Tamkin, A., Thamkul, J., Kaplan, J., Clark. J., and Ganguli, D. Towards measuring the representation of subjective global opinions in language models. In First Conference on Language Modeling, 2024. URL httt::: / //openreview. net forum? 2id 2dmmmiii2222111111
https: http://wwwwww..occcccc wwt.://c///////e/ceeeeeeeeeeeeee modelss-and-cooputez-eeoo
Argyle, L. Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., and Wingate, D. Out of one, many: Using language models to simulate human samples. Political Analysis, 31 (3):337-351, 2023. doi: 101 20007///...2233..
Bates, S., Angelopoulos, A., Lei, L., Malik, J., and Jor- dan, M. Distribution-free risk-controlling prediction sets. J. ACM, 68(6). September 2021. ISSN 0004-5411. doi: 10. 14.1445//478835. URL 1145/347855
Efron, B. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1 26, 1979. doi: 10.
1214/aos/1 http/////1116344// 176344552. URL /tttt:////oo...... ooo//00 2214/aos/1176344
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., and Larson, J. M. Synthetic replacements for human survey data? The perils of large language models. Political Analysis, 32(4): 401-416, 2024. doi: 10.1017/pan.20244 2024.5.
Goli, A. and Singh, A. Frontiers: Can large language mod- els capture human preferences? Marketing Science, 43(4):709-722, 2024. doi: 10. 10.1287/mkse.2023.000 URL stttps://doi..r 0306.
1287 222.mmcc..............
Boucheron, S., Lugosi, G., and Massart, P. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 02 2013, ISBN 9780199535255. doi: 10. 10.10099aappoooooo//771999993333
Gui, G. and Toubia, O. The challenge of using LLMs to simulate human behavior: A causal inference perspective. arXiv preprint arXiv:2312.15522 2023.
10

---

Uncertainty Quantification for LLM-Based Survey Simulations
He- Yueya, J., Ma, W. A., Gandhi, K., Domingue, B. W. Brunskill, E.,, and Goodman, N. D. Psychometric align- ment: Capturing human knowledge distributions via lan- guage models. arXiv preprint arXiv:2407.15645, 2024.
1214/ao//1 URL hwtphh:///oo..ccoo 1214/aos/11763477
Ramesh, A., Paviov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 18-24 Jul 2021. wRt /ttp:///////eeeeeeeeeee ngs .oig..m.... ml press/ 1139/ramesh2la. html.
Horton, J. J. Large language models as simulated economic learn from homo silicus? Technical report, National Bureau of Economic Research, 2023.
agents: What
Huang, Y., Yuan, Z., Zhou, Y., Guo, K., Wang, X., Zhuang, H., Sun, W., Sun, L.., Wang. J., Ye, Y., et al. Social science meets LLMs: How reliable are large language models in social simulations? arXiv preprint arXiv:2410.23426, 2024.
Santurkar, S., Durmus, E., Ladhak, F,, Lee, C., Liang, P., and Hashimoto, T. Whose opinions do language models reflect? In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 29971-30004 PMLR, 23-29 Jul 2023. v202 http//anturkarr233...
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7B. arXiv preprint 10.0v22310.06825, 2023.
Kim, J., O'Hagan, S., and Rockova, V. Adaptive uncer- tainty quantification for generative AL. arXiv preprint arXiv:2408. 08990, 2024.
mlr press/
Shafer, G. and Vovk, V. A tutorial on conformal predic- tion. Journal of Machine Learning Research, 912):37 421, 2008. URL http: // //wrr//mrr.///////mmm .rrg/ wwtppapers///eeoeeetootoot..erst http//appaas////// / shaferOBa. html. htmll.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. DeepSeek-V V3 technical report. arXiv preprint 1xXi:22412119433 2024.
Lu, X. and Wang, X. Generative students: Using LLM- simulated student profiles to support question item eval- uation. In Proceedings of the Eleventh ACM Con- ference on Learning Scale, L@S '24, pp. 16-27, New York, NY, USA, 2024. Association for Comput- ing Machinery. ISBN 9798400706332.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in a random world, volume 29, Springer, 2005, URL https: / //oii http:/ooooooo....////// http///..107/b106777 007.11666..
Wang, M., Zhang, D. J., and Zhang. H. Large language mod- els for market research: A data-augmentation approach. arXiv preprint arXiv:2412.1936, 2024.
doi: 10.1145/ 3657604.3662031. URL https: http//ooo.oggg////// 604. 3145/355609.36603..
Wang. Z., Lamb, A., Saveliev, E., Cameron, P., Zaykov, J., Hernandez-Looato. J. M., Turner, R. E., Baraniuk, R. G., Craig Barton, E., Peyton Jones, S., Woodhead, S., and Zhang, C. Results and insights from diagnos- tic questions: The NeurlPS 2020 education challenge. In Escalante, H. J. and Hofmann, K. (eds.), Proceed- ings of the NeurIPS 2020 Competition and Demonstra- tion Track, volume 133 of Proceedings of Machine Learning Research, pp. 191-205. PMI.R, 06-12 Dec 2021. URL httst WtRttttss//ppooeeeeeeeeee http///pppeeeggggee////e// mir.press/ v133 2233/aan22a. html.
Mei, Q., Xie, Y., Yuan, W., and Jackson, M. O. A Turing test of whether AI chatbots are behaviorally similar to humans. Proceedings of the National Academy of Sci- ences, 12199):2211922512,,, 2024. doi: 10.1073/pnas. 2313925121. UtL wwww www.wppaaa.... . orrg/rrggggo /oi/
OpenAl. GPT-3.5 Turbo, 2022. URL https://
form. www.:ai..oppai..ccmmmmmm com/ 3- 1-5tgpt-3-55-uut
OpenAI. Hello GPT-4o, 2024a. URL htt.htsss////enaaa -tt.looppppoo////
Yang, K., Li, H., Wen, H., Peng. T.-Q.. Tang, J., and Liu, H. Are large language models (LLMs) good social pre- dictors? In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computa- tional Lingaistics: EMNLP 2024, pp. 2718-2730, Miami, Florida, USA, November 2024. Association for Com- putational Linguistics. doi: emnlp.153. URL https: / /tt///aaaaaantooooyaaaaaa .orgg 2024. findi ngs-- 153/.
OpenAI. GPT-4o mini: Advancing cost-efficient in- telligence, 2024b. URL https://openai. com/ Index/gpt-4e andd-ainnidddniinnnngnnn cost- efficient-intelllenn
Owen, A. Empirical Likelihood Ratio Confidence Regions. The Annals of Statistics, 18(1):90- 120, 1990. doi: 10.

---

Uncertainty Quantification for LLM-Based Survey Simulations
Zelikman, E., Ma, W., Tran, J., Yang, D., Yeatman, J., and Haber, N. Generating and evaluating tests for K-12 students with language model simulations: A case study on sentence reading efficiency. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2190-2205, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/11/023. emnlp-main.135. URL https: / ac anthhlogy.. org/2023 emnlp main.135/.
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., and Yang. D. Can large language models transform computational social science? Computational Linguis- tics, 50(1))237-291, 03 2024. ISSN 0891-2017. doi: 10.1162/coli a 00502. URL https://doi.orge: 1162/coli 20000
12

---

Uncertainty Quantification for LLM-Based Survey Simulations
A. Illustrations and Examples
first present a visualization of the relation between the simulation sample size and the LLM's simulation power in Figure 4.
LARGE
RESPONDE RESPONSES
Figure 4. An interpretation of the simulation sample size i as the size of human population that an LLM can represent. Generating outputs from the LLM can be thought of as "resampling" from & human samples that make up the LLM. The figure is generated by DALL-E (Ramesh et al., 2021), and borrows ideas from the Mechanical Turk, a chess- playsng machine from the 18th century with human player hidden inside.
Below we provide more examples for the general problem framework in Section 3. The survey responses can be real-valued multi-dimensional.
Example A.1 (Market research). Suppose company is interested in learning its customers' willingness-to to--eessss-t-p------ pay (WTP) for product, which the highest price customer is willing pay for the product. Then, each 2 can represent customer profile (e.g., age, gender, occupation), each survey question is about certain product, and customer's 's response is noisy observation of the customer's s WTP. Then R( 3/) is the distribution of the customer population's WTP. We
may take ((()) the T-quantile of the WTP distribution R(- 1b), for some 7 E (0, 1):
0 (bbb inf {q [0, oo) P P (u S 5a) 2 r].
An LLM can be used to simulate customers' WTP for the product.
Example A.2 (Public survey, multi-diimnssonnl), Suppose an organization is interested in performing a public survey in city. Each survey question v is a multiple-choice question with 5 options. An example "How often do you talk your neighbors?", with 5 choices "Basically every day". "A few times week", "A few times month", "Once month". and "Less than once month". Every E Z is a person's profile (e.g., age, gender, occupation), the response space y is the standard orthonormal basis {e in R where y = indicates that person chooses the i-th option. We can take ((15 = 1t [y] E RR5, which summarizes the proportion of people that choose each option. An L.M can be used to simulate people's s answers to the survey question.
Example A.3 (Public survey, one-dimensional), Consider the setup in Example A.2. When the 5 choices in a survey question correspond to ordered sentiments, we can map them to numeric scores, say, v = (-1, - 3. 1) Then the statistic ((bb) (v, ((+b) reflects the population's average sentiment for the survey question vb.
B. Proofs
B.I. Failure of Exact CLT-Based Intervals under Distribution Shift
Consider a true distribution N 1) and a synthetic distribution N Nurm 1). Suppose we draw A i.i.d. synthetic samples {z, )= ~ Nu 1) and construct the standard (1 - a) confidence interval for μ.:
④ 0-1(1- (11 a/2)
(1-1122)
T (k)
= 5x
Ts
+
Vk
Vk
13

---

Uncertainty Quantification for LLM-Based Survey Simulations
where 3y 1/2 Σ Vi. Let A 1 p, which represents the discrepancy between the true distribution and the synthetic
distribution. Then
p P(UE ( =P | 11l ② (1-1a)) a/2) (1-0/2)
Vk
P (VV((z= - - vk(u -1 51111 (1///))
(VKS-
1 801(1111--11 - aa/2) S vi(āa - -100 S ViA + x (1-(1--122) a/2)) 1
Note that Vh(Tx )) (0, 1). Thus, whenever A +0.
P P < - at
failing to attain (1 a) coverage probability, regardless of the sample size k.
B.2. Proof of Theorem 2.4
We will prove the following stronger guarantee.
Lemma B.1 (Conditional coverage), Consider the setting of Theorem 2.4. Let 8 E (0, 1). With probability least 6,
PP P(NEZ I2M 8)) 21
2 log(110))
(19)
m
By Lemma B. we obtain
p I3m (mm
= E ( E (3) ( ]:)]
dl a1 M M
M k) > t) dt
(- m (t (1 - a)) a))²){ dt
1 -
cxpp
0
2
1-a
2 (1 (1 a))²) a
2 1
exp
2 21-a-1
We will now prove Lemma B.I. Define e = /2log(1/8)/m and a deterministic oracle sample size
k = inf {K k [K] IP & Im) > ae)
(20)
If k inf 0 does not exist, then there is nothing to prove. Now suppose that E [K] exists. We will prove that with probability at least 1 - 6, it holds that G(k) > a/2. When this event happens, we have ie < k, which implies P(u & (k)) R)) s <a + and thus (19), thanks to the independence of & and (,, Dev").
By Hoeffding's inequality (e.g., Theorem 2.8 in (Boucheron et al., 2013)) and the conditional independence of (Di, DPY"), .... (Dm, Dzym) given (I1,,.., ..., vn),
1
( 2m
P
G(k)
P
$ I
(m)(
(k))
2 8-8.
(21)
m
14

---

Uncertainty Quantification for LLM-Based Survey Simulations
We now bound P(ū @ _ (k)) For each j E [m]] and k [K],
1 {ū _ (B)) {ūg < My and Mj min 2nth (k)) + {0s 2 My and μi m
By the conditional independence of Dj and D given V3
P Ui < μuj and μ < min (k)
E
IP Üj = رن wb)) IP [r) < min (k) {35)
1 E 2
p 1gg < min (3) wt))
}
2 وم min I3 (k))
Similarly.
1 (p113 max I7 (k)). P ū 2 Mij and μy > max I,ym (k)) 2
Therefore,
1 (P(1s < min (ktmk)) P(u; max (z)m
P Ug I (m))
2 2
1 p u & [k)r 2 2F(H & (1))
(22)
When the event in (21) happens,
(1g(1/8)
G(k)
Σ
وi $ (k)
m
2m
j=1
(log(1/8)
p 14 q (Tm)
(by (22))
2
2m
(by definition of k)
2
This completes the proof.
B.3. Proof of Theorem 2.6
By Hoeffding's 's inequality (e.g., Theorem 2.8 in (Boucheron et al., 2013)) and union bound, the following happens with probability at least 1-6. 8:
G(k) <
1 p
Σ
(k)) +
log(n/5)
Wk < min n, K.
C 2
(23)
2m
54
We now show that the right hand side of (23) is at most a/2 for m large. For all j E [m]] and & E [K].
P U & 1 (k) = TP
syn
Uz
Vil
Since ūy N(p,, 1/n) and 7ith
N(ug syn 1/ks), then
1 1
95 U N
[1j
11
syn
k.
15

---

Uncertainty Quantification for LLM-Based Survey Simulations
When k S min(n, K, ( )
we have
P P(U & IT P
[3) V2.kk (P) - 12 1
C
Vk
ClVK- k -
4C/(5Vh)
= 2Φ
$22
(A (A I C/((V))
VK- +n-
Vk- + k-1
2/2C
4/2 (11 - -a/4))
(4/20b
2Φ
2Φ
24
< 2
5
5
Let
44/20-L (a/4)
a' 24p 2
8
5
then g 0 and
P
0j $ I: (k)
E.
(24)
2
When m >> log(n/))((EE), substituting (24) into (23) yields that for all S (n, K, (6x) )),
m
log(n/0)
G(k)
Σ
P Ug $ 1 (k))
m j=1
2m
<(2 a - {)
+ + 2
When this happens, we have k > (n, K, ( (6 )²).
B.4. Proof of Theorem 3.3
We will prove the following stronger guarantee. Lemma B.2 (Conditional coverage). Consider the setling of Theorem 3.3. Let 5 E (0, 1). With probability at least 1 - 6,
P P(((0) E s (k) K) 2 1-aa-- 7-1 log(1/8)
(25)
2m
By Lemma B. we obtain
P ((())) E Stm (k)) = = |P (P(O(W) E (k) Ri]
P (P((()) ())) x)) > 1) dl
2
| - exp
(22m7 (1 (1 - a))²}] a
-l-a
(22m7² (t - (1 - a))) de
> 1 a
exp
a
m
> 1 21 2m
We now prove Lemma B.2. Define = --11/ log(1/6) and deterministic oracle sample size 2m
T: = inl { K [K]: P (((() & (tm(()) > a + e
(26)
16

---

Uncertainty Quantification for LLM-Based Survey Simulations
If k inf 0 does not exist, then there is nothing to prove. Now suppose & [K] exists. We will prove that with probability
least 1 -, it holds that (kk > ya. When this event happens, we have 1 < k, which implies ((O()) & &St"((a) (k) R) S a+t
and thus (25), thanks the independence of 7: and (19) "D
By Hoeffding's inequality (e.g., Theorem 2.8 in (Boucheron et al., 2013)) and the conditional independence of (D1,Di")) (Dm: D given (#/1,.., vn),
m m
log (1/8) 2m
L(k)
P S - 5g (p)
2 8-88
(27)
We now bound P(S, g sym((b) For each j E [m] and E [K],
1 {5, & S (k)} 2 1 (0(5)) E Sj and (1bb)) & (g))
By the conditional independence of Dj and Dj given رز
P(S, 2 5T
2 E P (((())))))))))))){()) E S and ((1,) & (p)((+) [))]
( 4₁) -P((())) & 5f (A) [5)]
Ff7 P P(((())) & 57 v))]
p (0())) & (k)
y P((()(()) & 50m (k)
(28)
Therefore, when the event in (27) happens.
m
log(1/8)
L(K)= 2
Σ P
((S S z (k)) -
m j=l
2m
log (1/8)
2 7. IP (((())) & ssn((k))
(by (28))
2m
> 7a.
(by definition of k)
This completes the proof.
C. Details of Numerical Experiments
C.1. The OpinionQA Dataset
Selection of survey questions. The original dataset is categorized into topics such as health, crime/security, and political issues. Ideally, we would want to consider questions from the same category to ensure that they are similar enough. However, the category with the most questions has fewer than 200 questions. We thus consider pooling all questions. The dataset has 1,442 survey questions in total, which is too large for our computational resources. We selected subset of questions as follows. First, while the number of choices ranges from 2 to 19, most questions have choices. To give fair comparison and for simplicity, we only consider questions with 5 choices. Second, not all questions have choices that can be clearly ordered in sentiments, such as the following one:
Who do you think has the most responsibility to reduce the amount of made-up news and information? 1. The government, 2. Technology companies, 3. The public, 4. The news media, 5. None of these, 6. Refused.
We asked 4P to determine a question's 's choices can be ordered in sentiments and we keep those that have GPT-4o's affirmative answer. This leaves us with 546 questions. To compensate for the loss of similarity by pooling questions across
17

---

Uncertainty Quantification for LLM-Based Survey Simulations
various topics and to further reduce our computational cost, we selected 400 questions that "most similar" to each other by embedding the question statements using OpenAl's -ewe--ebeeddiieeee ng-3- 1, calculating the mean, and selecting the 400 questions with the smallest Euclidean distance to the mean. Out of these 400 questions, 15 questions have various issues with their choices by manual inspection, so we exclude them. This leaves us with 385 questions. All these questions happen to have at least 400 responses.
Example questions. The questions in the OpinionQA dataset span a wide range of topics, including health, crime/security, and political issues. Some example questions are as follows:
How much, if at all, do you think wages and incomes are contributing to your opinion about how the economy is doing?
great deal fair amount Not too much 4. Not at all Refused
Regardless of whether you would want to move, how likely is it that you will move to a different community at some point in the future?
Very likely
2. Somewhat likely 3. Not too likely Not at all likely 5. Refused
How much, if anythhing would you be willing to change about how you live and work to help reduce the effects of global climate change? Would you be willing to make:
1. A lot of changes 2. Some changes
3. Only a few changes 4. No changes at all 5. Refused
Excluding surveyees with missing information, each of the 385 questions we consider has at least 400 responses. Since there was no information on the surveyees' identification, by dropping repeated profiles we can only say that there are at least 32,864 surveyees. Each surveyee is described by 12 features. Their corresponding categories are listed in Table
Profiles.
Table Categories of surveyees' features in the OpinionQA dataset.
Feature
Options
US citizenship
'Yes', 'No'
Region
'Northeast', 'Midwest', 'South', 'West'
Sex
'Male', 'Female"
Age
'18-29', '30-49', '50-64", '65+'
Marital
'Married', 'Divorced', 'Separated", 'Widowed', 'Never been married'
status
Race
'White", 'Black', 'Asian', 'Hispanic', 'Other' 'Less than high school', 'High school graduate',
Educational
'Some college, no degree", 'Associate's degree',
background
'College graduate/some postgrad', 'Postgraduate"
'Less than $30,000', '$30.000-$50.000 '$50.000-$75.000,
Income
155,000-1100,000, $$100,000 or more'
'Protestant', 'Roman Catholic', 'Mormon', 'Orthodox', 'Jewish', 'Muslim', 'Buddhist', 'Hindu', 'Atheist', 'Agmostic', 'Other'. 'Nothing in particular'
Religious
affiliation
Religious
'More than once a week', "Once a week'. "Once or twice a month",
attendance
'A few times a year', 'Seldom', 'Never' 'Republican', 'Democrat', "Independent', "Other"
Political party
Political
'Very conservative', "Conservative', 'Moderate', 'Liberal', "Very liberal'
ideology
18

---

Uncertainty Quantification for LLM-Based Survey Simulations
Synthetic response generation. We generate synthetic profiles by bootstrapping the 32,864 unique real profiles. We then generate synthetic answers by prompting LLMs to pretend that they are surveyee with the synthetic profile and answer the question. An example prompt is as follows:
Pretend that you reside in the US and you are a US citizen from the West region of the country. You are female, your age is between 18 and 29, and you are single. In terms of race, you are white. In terms of education, you atlended college but did not graduate. Your annual income is less than $30,000. Religion-wise, you do not belong to any particular religion, and you never attend religious services. Politically, you are affiliated with a political party that is not Democratic or Republican, and you consider your political ideology to be liberal. Please answer the following question:
How much, if at all, do you think what happens to black people in the country overall affects what happens in your own life? ['1. lot', '2. Some', '3. Not much', '4. Not at all', '5. Refused').
Please provide your answer choice (a single number from I 5) in double square brackets.
In our experiments, LLMs usually directly gave answers in the required format, e.g., '[[2]]'
C.2. The EEDI Dataset
Example questions.
Some example questions from the EEDI dataset are as follows:
What number belongs in the box? 7
A)9 B)-5 C)-6 5
If you multiply a square number by 9, you get a square number. Is this statement:
A) always true B) sometimes true
C) never true
D) impossible say
Which calculation is equal to -20?
+ A) 2 x (-2) - (-4) 4 B) 28 (-4) x2 (-5)T2++ (-44)= +1
Profile distribution. Excluding students with missing information which take up less than 10% of the total population, there 2,111 students who answered least one of the 412 questions. Each student is described by three features: gender, age, and whether not they are eligible for free school meals premium pupil. Gender is represented by 1 or 2, where I corresponds to female and 2 corresponds male. The students" ages are rounded to integers from 11 and 14. Whether or not a student is cligible for free school meals is represented by 0 or I, where corresponds to not eligible and corresponds to eligible. The distribution of these students' features is presented in Table 2.
Table 2. Summary statistics of students' features in the EEDI dataset.
min
max
mean
median
standard deviation
Gender
1
2
1.4988
0.5001
Age
11
14
11.2776
11
0.4696
Premium Pupil
0
0.2842
0
0.4512
Synthetic response generation. For each question, we generate synthetic profiles by sampling with replacement from the real profiles. We then generate synthetic answers by prompting LLMSs to pretend that they are student with the synthetic profile and answer the question. We adapted the prompt from (He- Yueya et al., 2024) with slight modifications to reduce computational cost. An example prompt featuring an l-yearroold boy who is not eligible for free school meals is as follows:
Pretend that you are an 1l I-year-old student. Your gender is male. You are not eligible for free school meals or pupil premium due to being relatively financially advantaged. Given your characteristics. is likely that you would be able to solve the following problem?
19

---

Uncertainty Quantification for LLM-Based Survey Simulations
Problem: [Insert question here]
If yes, pul the final answer choice (a single letter) in double square brackets. If you are likely to struggle with this problem, put a plausible incorrect answer choice (a single letter) in double square brackets.
An example answer from GTP-4o when given the second example question above is as follows:
As an 1l-year-old student, might have learned about square numbers and multiplication school. However, the problem may bit tricky if haven't thought about how multiplying square numbers by other numbers can also result in square numbers. I might immediately realize that is actually square number itself (3 squared), which makes this property more evident.
Considering this, I could find the reasoning challenging and decide based on a misconception. I might go with a plausible incorrect answer choice like [[B]] because might think that it's only sometimes possible without realizing the full mathematical principle involved.
C.3. Visualization of Simulation Sample Size Selection
In Figure 5, we visualize the process of selecting the simulation sample size Tc: it is the first & which (I(K)) up-crosses the threshold ya.
0.12
train miscoverage rate
threshold
0.10
train miscoverage rate
0.08
0.06
0.04
0.02
0.00
20
80
100
k
Figure 5. Visualization of simulation sample size selection for the OpinionQA dataset. Here a 0.1, = 0.5, K 100, c= // and
M 2. The red dotted horizontal line threshold "ya 0.05. The blue curve represents {L(k) ] [k=1- Our approach chooses & as the point at which the blue line first up-crosses the red threshold: 7: == 57.
D. A Addtioonal Experiment Results
In this section, we provide additional experiment results for three performance metrics: the miscoverage probability proxy L ((k) defined in (18), the selected simulation sample size k. and the half width of the synthetic confidence interval
Sayn (k)
20

---

Uncertainty Quantification for LLM-Based Survey Simulations
D.I. Miscoverage Probability Proxy
In Table and Table 4, we present the p-values for the hypothesis test E (k)) S a against [(k)) > a for various LLMs and a over the OpinionQA and EEDI datasets, respectively. In Table 5 and Table 6, we present the means and standard errors of (k) as defined by (18) over 100 random splits. They complement Figure 2.
Table 3. p-values for the hypothesis test E [[]]]]] S a against ] > a for various LLMs and over the OpinionQA dataset. Note that the a values specify the experiment configuration and are not the significance levels of the hypothesis test.
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
GPT-3.5-turbo
0.0618
0. 1293
0.1996
0.2245
0.6981
0.3460
0.3387
0.2591
0.7846
0.8430
GPT--4-mini
0.
0.
0.0658
0.2352
0.4741
0.4363
0.2000
0.1150
0.5717
0.3812
GPT-4o
0.3679
0.3218
0.4910
0.2243
0.5837
0.4379
0.2329
0.3388
0.9686
0.9226
Claude 3.5 Haiku
0.3857
0.5889
0.5945
0.4558
0.7030
0.6029
0.7370
0.8853
0.9922
0.9791
Llama-3-8B
0.7571
0.4376
0.2609
0.6072
0.5689
0.4178
0.7018
0.6132
0.7391
0.5169
Llama 3.3 70B
0.5505
0.6519
0.7687
0.5936
0.7055
0.7207
0.6050
0.3530
0.6973
0.5274
Mistral 7B
0.6249
0,0505
0.0517
0.0333
0.6473
0.5988
0.5394
0.2119
0.3380
0.3426
DeepSeek-Ve V3
0,0686
0.0969
0.0841
0.1513
0.5075
0.2257
0.1678
0.2638
0.3457
0.3024
Random
0.7779
0.6214
0.7527
0.6586
0.9570
0.9903
0.9321
0.7984
0.9259
0.9783
Table p-values for the hypothesis test IE [[(}) S against I8 [ī.(k) for various LLMs and over the EEDI dataset. Note that
the a values specify the experiment configuration and are not the significance levels of the hypothesis test.
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
GPT-3.5-turbe
0.9988
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
GPT-4o-mini
1,0000
1.0000
1.0000
1.0000
1.0000
1,0000
1.0000
1.0000
1.0000
1.0000
GPT-44
0.1832
0.5820
0.9998
0.9732
0.9938
0.9807
0.9292
0.8259
0.9757
0.9922
Claude 3.5 Haiku
0.8656
0.9999
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
Llama 3.3 70B
0.9575
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
Mistral 7B
1.0000
1,0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
DeepSeek-V3. V3
0.9444
0.9789
0.9789
0.9981 1
1.0000
0.9992
1.0000
1.0000
1.0000
1.0000
Random
0.9800
1,0000
1.0000
0.9997
0.9954
0.9895
0.9979
0.9997
0.9920
0.9990
Table 5. Means and standard errors L(k)
over 100 random splits for various LLMs and over the OpinionQA dataset.
0.05
0.10
0.1.5
0.20
0.25
0.30
0.35
0.40
0.45
0.50
GPT-3.5
0.0547
0.1051
0.1549
0.2051
0.2461
0.3032
0.3536
0.4056
0.4432
0.4909
-turbo
(0.003)
(0.0045)
(0.0059)
(0.0067)
(0.0075)
(0.0082)
(0.0087)
(0.0086)
(0.0086)
(0,009)
GPT-4o
0.0535
0.1052
0.1588
0.2052
0.2505
0.3014
0.3574
0.4104
0.4484
0.5027
-mini
(0.0036)
(0.0053)
(0.0059)
(0.0072)
(0.008)
(0.0089)
(0.0088)
(0.0087)
(0.0086)
(0.009)
0.051
0. 0.1022
0.1501
0.2049
0.2486
0.3012
0.3555
0.4031
0.4361
0.489
GPT-4o
(0.0031)
(0.0048)
(0.0058)
(0.0065)
(0.0068)
(0.0075)
(0.0075)
(0.0075)
(0.0075)
(0,0078)
Claude-3-5
0.051
0.0988
1.8484
0.2008
0.246
0.2979
0.3448
0.3899
0.4284
0.4813
-Haiku
(0.0036)
(0.0052)
(0.0065)
(0.007)
(0.0076)
(0.008)
(0.0082)
(0.0084)
(0.0089)
(0.0092)
0.0481
0.1006
0. 1.336
0. 0.19
0.2487
0,3017
0.3452
0,3974
0.4442
0.4996
Llama-3-8B
(0.0028)
(0.0041)
(0.0057)
(0.0067)
(0.0075)
(0.0081)
(0. (0.0091)
(0.009)
(0.0091)
(0.0092)
Llama-3.3
0.0496
0.0983
0. 14558
0.1984
0.2464
0.2955
0.3479
0.4032
0.4453
0.4994
-70B
(0.0031)
(0.0043)
(0.0057)
(0.0066)
(0.0067)
(0.0078)
(0.0078)
(0.0086)
(0.009)
(0.0094)
0.049
0.1075
0.1586
0.21
0.2478
0.2983
0.3492
0.4065
0.4536
0,5035
Mistral-7B
(0.0033)
(0.0046)
(0.0053)
(0.0055)
(0.0058)
(0.0067)
(0.0079)
(0.0081)
(0.0087)
(0.0087)
0.0549
0.106
0.1571
0.2062
0.2499
0.3051
0.3579
0.4053
0.4535
0.5049
DeepSeek-VV V3
(0.0033)
(0.0046)
(0.0052)
(0.006)
(0.0069)
(0.0067)
(0.0082)
(0.0084)
(0.0088)
(0.0095)
0.0478
0.0986
0.1462
0.1975
0.2403
0.2844
0.3412
0.3948
0.4401
0.4853
Random
(0.0029)
(0.0046)
(0.0055)
(0.006)
(0.0057)
(0.0067)
(0.0059)
(0.0062)
(0.0068)
(0.0073)
21

---

Uncertainty Quantification for LLM-Based Survey Simulations
Table Means and standard errors of L(k)
over 100 random splits for various LLMs and
the EEDI dataset.
a
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0,50
GPT-3.5
0.0401
0.0882
0.1122
0.1401
0.2067
0.26
0.308
0.2806
0.3655
0.3925
-turbo
(0.0033)
(0.0025)
(0.0057)
(0.0044)
(0.0038)
(0.0051)
(0.0088)
(0.0114)
(0.0116)
(0.0104)
GPT-4o
0.035
0.0773
0.1155
0.1463
0.2074
0.2425
0.2909
0.3395
0.3417
0.3747
-mini
(0.0033)
(0.0026)
(0.0066)
(0.006)
(0.0055)
(0.0037)
(0.0043)
(0.0108)
(0.0117)
(0.0109)
0.0528
0.099
0.1365
0.1891
0.2313
0.283
0.3383
0.3931
0.4342
0.4805
GPT-4o
(0.0032)
(0.0047)
(0.0039)
(0.0057)
(0.0075)
(0.0082)
(0.008)
(0.0074)
(0.008)
(0.0081)
Claude-3-5
0.0461
0.0833
0. 1234
0.1618
0.1993
0.2577
0.2772
0.3326
0.3948
0.4327
-Haiku
(0.0036)
(0.0044)
(0.0046)
(0.0055)
(0.0072)
(0.008)
(0.008)
(0.0081)
(0.0089)
(0.0081)
Llama-3.3
0.0444
0.0785
0.1261
0. 0.1355
0.2096
0.2564
0.2998
0.348
0.3988
0.4235
-70B
(0.0033)
(0.0031)
(0.0055)
(0.0059)
(0.0054)
(0.0051)
(0.0059)
(0.0061)
(0.0085)
(0.0133)
0.0366
0.0829
0.1621
0.2074
0.2447
0.2716
0.2858
0.3337
0.391
Mistral-7B
(0.0033)
(0.0043)
(0.003)
(0.0031)
(0.008)
(0.0098)
(0.0108)
(0.0055)
(0.0043)
(0.005)
0.045
0.0913
0. 1.99
0.1835
0.2153
0.2777
0.3223
0.3667
0.4166
0.46
DeepSeek-V3
(0.0032)
(0.0043)
(0.005)
(0.0057)
(0.0068)
(0.0071)
(0.0067)
(0.0069)
(0.0075)
(0.0073)
0.0446
0.0645
1.12
0.1812
0.2338
0.2851
0.3299
0.3702
0.4302
0.4731
Random
(0.0026)
(0.0038)
(0.0064)
(0.0055)
(0.0062)
(0.0065)
(0.007)
(0.0086)
(0.0082)
(0.0087)
D.2. Selected Simulation Sample Size
In Table and Table 8, we present the detailed results for the selected simulation sample size R. They complement Figure 3.
Table 7. Average 1o (with 95% the largest & on average.
margin of error in parentheses) for various LLMs and various over the OpinionQA dataset. GPT-4o has
0.05
0.10
0.1.5
0.20
0.25
0.30
0.35
0.40
0.45
0.50
GPT-3.5
36.65
44.03
44.62
44.22
43.59
43.21
43.44
44.75
45.22
45,91
-turbo
(1.064)
(0.9373)
(0.4879)
(0.4263)
(0.4794)
(0.5823)
(0.7403)
(0.7367)
(0.6492)
(0.5296)
GPT-4o
26.85
33.68
37.37
37.56
37.08
38.4
40.09
41.36
41.6
41.51
-mini
(0.7517)
(0.8189)
(0.5871)
(0.4058)
(0.5025)
(0.6345)
(0.6149)
(0.5745)
(0.4671)
(0.5328)
54.1
62.19
65.35
68.77
69.58
74.29
81.83
88.7
89.37
91.86
GPT-4o
(1.7862)
(1.2286)
(1.0516)
(0.9885)
(1.1095)
(1.5015)
(1.6431)
(1,5013)
(1.2831)
(1.2268)
Claude-3-5
35.75
40.7
42.46
44.74
45.07
46.27
47.79
49.84
50.37
51.34
-Haiku
(1.0428)
(0.7842)
(0.7336)
(0.6872)
(0.6561)
(0.6464)
(0.7881)
(0.8488)
(0.7415)
(0.6819)
28.81
31.93
34.35
34.83
35.13
37.3
38.1
38.9
40.5
44.25
Llama-3-8B
(0.6264)
(0.7149)
(0.4828)
(0.5382)
(0.6855)
(0.5248)
(0.5491)
(0.6747)
(0.8084)
(0.8016)
Llama-3.3
46.3
47.67
46.39
48.12
53.49
56.21
57.45
58.04
58,62
61.8
-70B
(0.6921)
(0.5339)
(0.5331)
(1.0276)
(1.2524)
(0.8797)
(0.7294)
(0.7947)
(1.0285)
(1.0627)
30.81
33.47
39.47
44.59
47.53
52.43
56.34
58.43
60.29
64.85
Mistral-7B
(0.2566)
(0.8928)
(0.8815)
(0.8935)
(0.8622)
(1.0825)
(1.0113)
(0.9535)
(1.1178)
(1.2481)
37.59
46.75
46.66
45.1
46.03
47.77
47.76
47.61
47.92
49,51
DeepSeek-V.
(1.4767)
(0.79)
(0.4771)
(0.6201)
(0.5657)
(0.6135)
(0.4606)
(0.48)
(0.6989)
(0.7031)
17.09
18.07
19.33
19.83
19.85
20.36
20.01
19.92
19.93
20.4
Random
(0.3761)
(0.4402)
(0.3361)
(0.4046)
(0.3261)
(0.3227)
(0.3191)
(0.2849)
(0.2662)
(0.2252)
22

---

Uncertainty Quantification for LLM-Based Survey Simulations
Table Average & (with 95% margin of in parentheses) for various LLMs and various the EEDI dataset. DeepScek-V V3 has
the largest & on average.
a
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
GPT-3.5
5.68
5.89
5.19
5.03
4.94
4.71
4.17
4.29
4.13
-turbo
(0.1033)
(0.0613)
(0.0769)
(0.0334)
(0.0195)
(0.0465)
(0.0889)
(0.0736)
(0.0889)
(0.0659)
GPT-4o
6.26
5.94
5.44
5.14
5.06
5.0
4.99
4.6
4.25
4.15
-mini
(0.086)
(0.0465)
(0.0973)
(0.068)
(0.0465)
(0)
(0.0195)
(0.096)
(0.0849)
(0.07)
8.04
7.82
8.31
8.1
7.81
7.83
7.84
7.93
7.9
8.17
GPT-4o
(0.1438)
(0.1672)
(0.1853)
(0.116)
(0.1609)
(0.152)
(0.1459)
(0.1308)
(0.1224)
(0.1178)
Claude-3-5
7.57
7.42
7.05
6.82
6.59
6.47
6.15
6.15
6.12
6.04
-Haiku
(0.1337)
(0.0967)
(0.0802)
(0.0849)
(0.0964)
(0.1017)
(0.07)
(0.07)
(0.0889)
(0.0674)
Llama-3.3
6.19
5,94
5.6
5.09
5.05
5.0
5.0
4.92
4.83
4.54
-70B
(0.0863)
(0.0542)
(0.096)
(0.0561)
(0.0427)
(0.0277)
(0.0392)
(0.0532)
(0.0736)
(0.0977)
5,48
5.2
5.01
4.97
4.68
4.52
4.29
4.02
4.0
4.0
Mistral-7B
(0.0979)
(0.0877)
(0.0195)
(0.0334)
(0.0914)
(0.0979)
(0.0889)
(0.0274)
(0)
I0.35
10.51
10.76
10.92
11.21
11.53
11.67
11.8
11.88
11.66
DeepSeek-V. V3
(0.1695)
(0.1628)
(0.1841)
(0.1954)
(0.1551)
(0.1504)
(0.1594)
(0.1518)
(0.1337)
(0.1364)
8.02
7.06
6.64
6.92
7.06
7.53
7.69
7.49
7.56
7.59
Random
(0.1239)
(0.0465)
(0.1127)
(0. (0.1198)
(0.1435)
(0.1223)
(0.1405)
(0.1828)
(0.1475)
(0.1361)
D.3. Half-Width of Confidence Interval
In Figure 6, Table 9, Figure 7 and Table 10, we present the detailed experiment results for the half-width of the synthetic confidence interval 8 (k))
gpt-3.5
gpt-4o-mini
gpt-4o
claude-3---haiku
Llama-3-88
Llama-3.3-708
synthetic CI width
Mistral-7B
0.7
DeepSeek-V3
random
0.5
0.3
0.1
0.2
0.3
0.4
0.5
Figure Confidence interval half-width for various LLMs and a over the OpinionQA datasct.
23

---

Uncertainty Quantification for LLM-Based Survey Simulations
Table Average half-width (with 95% margin of parentheses) of the synthetic confidence interval for various L.MM and over
the OpinionQA dataset. GPT--4o has the shortest confidence interval on average.
a
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
GPT-3.5
0.6401
0.5241
0.4825
0.4568
0,4373
0.4198
0.4017
0,3803
0.364
0.348
-turbo
(0.0099)
(0.0058)
(0.0027)
(0.0021)
(0.0024)
(0.0027)
(0.0033)
(0.0031)
(0.0026)
(0.002)
GPT-4o
0.7465
0.5999
0,5279
0.4958
0.4745
0.4457
0.418
0.3953
0.3792
0.3661
-mini
(0.0097)
(0.0072)
(0.0043)
(0.0027)
(0.0032)
(0.0037)
(0.0032)
(0.0028)
(0.0022)
(0.0024)
0.528
0.4405
0.3991
0.3667
0.3466
0.3208
0.293
0.2702
0.2589
0.2461
GPT-4o
(0.009)
(0.0041)
(0.0031)
(0.0025)
(0.0026)
(0.0031)
(0.0029)
(0.0024)
(0.0018)
(0.0017)
Claude-3-5
0.6478
0.5446
0,4954
0,4548
0,4305
0,4057
0.383
0.3604
0,3449
0.3292
-Haiku
(0.0095)
(0.0052)
(0.0041)
(0.0035)
(0.0031)
(0.0028)
(0.0031)
(0.0031)
(0.0026)
(0.0023)
0.7189
0.6156
0,5503
0.5154
0,4884
0,4519
0,4286
0,4079
0,3852
0.3552
Llama-3-8B
(0.0078)
(0.0069)
(0.0038)
(0.0038)
(0.0047)
(0.0032)
(0.0028)
(0.0032)
(0.0036)
(0.0034)
Llama-3.3
0.5658
0,502
0.4732
0.4393
0.3965
0.3683
0.3489
0.3336
0.32
0.3004
-70B
(0.0045)
(0.0029)
(0.0026)
(0.0044)
(0.0048)
(0.0029)
(0.0023)
(0.0022)
(0.0028)
(0.0025)
0,6925
0.6024
0.5149
0.4563
0.4196
0.3821
0.3529
0.3328
0.3156
0.2935
Mistral-7B
(0.0028)
(0.0079)
(0.0058)
(0.0047)
(0.0035)
(0.0041)
(0.0033)
(0.0026)
(0.0029)
(0.0028)
0,6376
0.5078
0.4717
0.4527
0.4257
0.3992
0.3824
0.3681
0.3536
0.3353
DeepSeek-Vi V3
(0.0143)
(0.0047)
(0.0025)
(0.0031)
(0.0026)
(0.0026)
(0.0019)
(0.0018)
(0.0025)
(0.0024)
0.9337
0.8194
0.7341
0.6842
0.649
0.612
0.5918
0.5697
0.5481
0.522
Random
(0.0105)
(0.0106)
(0.006)
(0.0067)
(0.0054)
(0.005)
(0.0048)
(0.0042)
(0.0037)
(0.0028)
gpt-3.5
gpt-4o-mini
0.8
gpl-4o
claude-3--hailku
Llama-3.3-708
Mistral-70
synthetic CI width
DeepSeck-V3
random
0.4
0.1
0.2
0.3
0.4
0.5
Figure 7, Confidence interval half-width for various LLMs and over the EEDI dataset.
24

---

Uncertainty Quantification for LLM-Based Survey Simulations
Table 10. Average half-width (with 95% margin of error in parentheses) the synthetic confidence interval for various M.LMs and a over
the EEDI dataset. DeepSeck- V3 has the shortest confidence interval average.
a
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
GPT-3.5 0.8085
0.714
0.7078
0.6768
0.6457
0.6203
0.6106
0.6229
0.592
0.5806
-turbo
(0.0075)
(0.0041)
(0.0048)
(0.002)
(0.0015)
(0.0034)
(0.0062)
(0.0049)
(0.0057)
(0.0041)
GPT-4o
0.769
0.7107
0.6922
0.6703
0.6415
0.616
0.5911
0.5941
0.5945
0.5794
-mini
(0.005)
(0.0031)
(0.0061)
(0.004)
(0.0026)
(0)
(0.0014)
(0.0064)
(0.0055)
(0.0043)
0.6794
0.6215
0,5613
0.5343
0.5181
0.494
0.4731
0.4517
0.4355
0.4128
GPT-4o
(0.0059)
(0.0062)
(0.0068)
(0.0039)
(0.0052)
(0.0047)
(0.0043)
(0.0036)
(0.0033)
(0.003)
Claude-3-5
0.7001
0.6364
0.6069
0.582
0,5629
0.5428
0.533
0.5122
0,4947
0,4797
-Haiku
(0.0059)
(0.0041)
(0.0035)
(0.0038)
(0.0042)
(0.0042)
(0.0028)
(0.0027)
(0.0036)
(0.0027)
Llama-3.3
0.7734
0.7108
0.6821
0.6733
0.6421
0.6162
0,5908
0.5727
0,5572
0.5551
-70B
(0.0052)
(0.0036)
(0.006)
(0.0033)
(0.0024)
(0.0018)
(0.0024)
(0.0036)
(0.0047)
(0.0061)
0.823
0.761
0.7191
0.681
0.6693
0.6509
0.6399
0.633
0.6107
0.5887
Mistral-7B
(0.0073)
(0.0062)
(0.0012)
(0.0027)
(0.007)
(0.0071)
(0.0062)
(0.0018)
(0)
(0)
0.5984
0.5351
0.4921
0.4607
0.4315
0.4063
0.3872
0.3699
0.3548
0.3453
DeepSeek- V3
(0.0045)
(0.004)
(0.0043)
(0.0042)
(0.0031)
(0.0027)
(0.0029)
(0.0025)
(0.002)
(0.0021)
0.6799
0.6517
0.6263
0.5786
0,545
0.5032
0.4779
0.4665
0.4459
0.4287
Random
(0.0056)
(0.002)
(0.0053)
(0.0052)
(0.0058)
(0.0041)
(0.0049)
(0.0061)
(0.0045)
(0.0039)
25


Reviews:
[{"review_id": "bZVNkVe2u4", "reviewer": "Reviewer_xDnM", "summary": "**Survey Simulation Problem**: This paper proposes using LLMs to estimate the statistics for each survey question without asking humans to fill out the survey.  For example, in an educational test scenario, if we aim to estimate the probability $\\mu$ of students correctly answering a test question, we can employ an LLM to sample multiple answers, and then we can use these answers to construct a confidence interval to estimate this probability. \n\n**Methodology**:\n- 1. Given a range of user profiles (e.g., gender, age), sample K user profiles. \n- 2. For each survey question, combine it with K=50 user profiles, and prompt LLM to get K corresponding answers.\n- 3. Calculate the confidence interval (the statistics we want) based on these K answers (e.g., Apply Hoeffding’s concentration inequality).\n- 4. Utilize survey questions in the training dataset, and their corresponding real human answers to identify the optimal hyperparameter \\( K \\).\n- 5. For a new survey question, follow steps (1)-(3) to compute the respective confidence interval.\n\n## update after rebuttal\nIn general, I think the problem is important and practically useful, and the solution of applying conformal-prediction-style calibration to obtain a better-calibrated statistics seems quite fit in this case. Given these points, I am leaning towards acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and the encouraging score. Below are our responses.\n\n**Experimental Designs Or Analyses**\n> *... most of the experimental results are put in the Appendix and some details are not shown...*\n\nThanks for raising this point. We will add more experiment details and results in the main text, especially those verifying our guarantees on the coverage probability. In particular, we will examine the miscoverage proxy $\\tilde{L}(\\hat{k})$ in Equation (18) of the paper. As we explain in the paper, $\\mathbb{E}[\\tilde{L}(\\hat{k})]\\le\\alpha$ implies a controlled miscoverage probability $\\mathbb{P}(\\theta(\\psi)\\not\\in\\mathcal{S}^{\\text{syn}}(\\hat{k}))\\le\\alpha$. Thus, to check the validity of our method, we will perform a statistical hypothesis test on $H_0:\\mathbb{E}[\\tilde{L}(\\hat{k})]\\le\\alpha$ against the alternative $H_1:\\mathbb{E}[\\tilde{L}(\\hat{k})]>\\alpha$. The following tables report the $p$-values computed from the one-sided $z$-test for different LLMs and $\\alpha$ over the 100 random splits. We see that all $p$-values are reasonably large, supporting the null hypothesis $\\mathbb{E}[\\tilde{L}(\\hat{k})]\\le\\alpha$ at, say, the 0.05 significance level. This shows that the miscoverage probability is effectively controlled by our method.\n\nOpinionQA:\n\n|min|25% quantile|median|75% quantile|max|\n|:---:|:---:|:---:|:---:|:---:|\n|0.03|0.25|0.51|0.70|0.99|\n\n|$\\alpha$|0.05|0.10|0.15|0.20|0.25|0.30|0.35|0.40|0.45|0.50|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| :---:|\n|GPT-3.5-turbo|0.06|0.13|0.20|0.22|0.70|0.35|0.34|0.26|0.78|0.84|\n|GPT-4o-mini|0.16|0.16|0.07|0.24|0.47|0.44|0.20|0.12|0.57|0.38|\n|GPT-4o|0.37|0.32|0.49|0.22|0.58|0.44|0.23|0.34|0.97|0.92|\n|Claude-3.5-Haiku|0.39|0.59|0.59|0.46|0.70|0.60|0.74|0.89|1.00|0.98|\n|Llama-3-8B|0.76|0.44|0.26|0.61|0.57|0.42|0.70|0.61|0.74|0.52|\n|Llama-3.3-70B|0.55|0.65|0.77|0.59|0.71|0.72|0.61|0.35|0.70|0.53|\n|Mistral 7B|0.62|0.05|0.05|0.03|0.65|0.60|0.54|0.21|0.34|0.34|\n|DeepSeek-V3|0.07|0.10|0.08|0.15|0.51|0.23|0.17|0.26|0.35|0.30|\n|Random|0.78|0.62|0.75|0.66|0.96|0.99|0.93|0.80|0.93|0.98|\n\nOnly one $p$-value (Mistral with $\\alpha=0.2$) is below 0.05. This is not surprising due to the multiple comparison problem. Indeed, even when the null hypothesis is true, one would expect to see one out of 20 $p$-values to be below 0.05.\n\nEEDI:\n|min|25% quantile|median|75% quantile|max|\n|:---:|:---:|:---:|:---:|:---:| \n|0.18|1.00|1.00|1.00|1.00|\n\n|$\\alpha$|0.05|0.10|0.15|0.20|0.25|0.30|0.35|0.40|0.45|0.50|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|GPT-3.5-turbo|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|\n|GPT-4o-mini|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|\n|GPT-4o|0.18|0.58|1.00|0.97|1.00|0.98|0.93|0.83|0.98|0.99|\n|Claude-3.5-Haiku|0.87|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|\n|Llama-3.3-70B|0.96|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|\n|Mistral 7B|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|\n|DeepSeek-V3|0.94|0.98|0.98|1.00|1.00|1.00|1.00|1.00|1.00|1.00|\n|Random|0.98|1.00|1.00|1.00|1.00|0.99|1.00|1.00|0.99|1.00|\n\n**Weaknesses**\n> *1. ...the abstract is difficult to parse initially; including a brief explanation or definition of the survey simulation task at the beginning would help ...*\n\nThanks for the suggestion. We will change the first sentence of the abstract to \"We investigate the use of large language models (LLMs) to simulate human responses to survey questions, and perform uncertainty quantification to gain reliable insights.\"\n\n> *2. While the test miscoverage rate is quite important for practical validation, it is relegated to the appendix. Similarly, the discussion about selecting the simulation sample size ... might benefit from its own dedicated section ...*\n\nThanks for the suggestions. We will include results on the miscoverage rate in the main text. Please refer to our response to your comment in \"Experimental Designs Or Analyses\" for more details. We will also move the results on the chosen simulation sample size $\\hat{k}$ to a separate part.\n\n**Questions**\n\n> *When creating (K) user profiles based on a given range of features (e.g., gender, age), how exactly are these profiles sampled? ...*\n\nThe reviewer is correct that in our experiments, the profiles are randomly drawn from the dataset, where we have access to the profile information. When real-data profiles are not available, practitioners may rely on their knowledge of the target population to create fictional profiles. We point out our method does not require the profile distribution $\\mathcal{P}^{\\mathsf{syn}}$ used for simulation to be the same as the target population $\\mathcal{P}$, and automatically addresses any distribution shift between $\\mathcal{P}^{\\mathsf{syn}}$ and $\\mathcal{P}$. Our Theorems 2.4 and 3.3 on the coverage guarantees do not assume any relation between them. Of course, we expect the confidence sets to be tighter when $\\mathcal{P}^{\\mathsf{syn}}$ and $\\mathcal{P}$ are close.", "claims_and_evidence": "See the following \"Methods And Evaluation Criteria\" and \"Theoretical Claims\" for details.\nIn general, the claims in this paper are supported by convincing evidence.", "methods_and_evaluation_criteria": "**Method**\n\nThis paper's goal is to estimate a statistic for a given survey question—one that would require data from the real population—by instead simulating responses from an LLM. The challenge is that if we sample extensively from an LLM, the resulting estimate reflects the LLM’s internal distribution rather than the true population’s. \n\nThis paper handles this misalignment by using actual responses to similar questions for calibration. Then it compares each LLM-based confidence set against these real outcomes. If the LLM-based confidence sets repeatedly fail to include the real responses, which indicates a sizable mismatch between the synthetic and real distributions, then the algorithm will stop increasing the synthetic sample size (and hence interval narrowing) early, maintaining adequate coverage. In this way, it ensures, on average, the final confidence interval achieves near $1-\\alpha$ coverage of the real statistic.\n\nA limitation is that the coverage guarantees hold on average over random draws of survey questions and data, not necessarily for any particular question. In many practical applications, one does care about accuracy for each specific question. However, in the calibration literature, per-question calibration (i.e., sample-wise) is typically infeasible.\n\nOverall, the proposed calibration procedure is still effective with a theoretical guarantee.\n\n**Evaluation Criteria**\n- The 2 datasets are representative: 1) use EEDI Dataset to estimate the probability a student answers a math question correctly and produce a confidence interval around that probability; 2) use OpinionQA to estimate the mean sentiment score for a given question and produce a confidence interval.\n-  The evaluation metrics test miscoverage rate is reasonable.", "theoretical_claims": "I briefly reviewed Theorems 2.6 and 3.3; they seem correct, though I might have overlooked some details.", "experimental_designs_or_analyses": "Overall, the experimental design (train/test approach, empirical miscoverage checks) is reasonable and aligns with the proposed calibration method. \nThe issue is that most of the experimental results are put in the Appendix and some details are not shown, which quite disrupts the flow of the reasoning and understanding.", "relation_to_broader_scientific_literature": "The authors’ work falls into the category of using LLMs as “synthetic subjects” in survey or experimental contexts. While several of those prior studies demonstrate that LLMs can be cost-effective in reproducing or approximating human‐subject responses, most do not provide formal guarantees on how well those simulations match real distributions. This paper’s core contribution lies in formalizing and ensuring coverage—i.e., constructing confidence sets around population estimates that remain valid despite misalignment between real and simulated data.\n\nThe authors extend ideas from uncertainty quantification (notably in distribution‐free or conformal settings) by adapting them to an LLM scenario, where the synthetic distribution is not guaranteed to reflect the true data‐generating process. Their approach incorporates “calibration data” (a set of questions with real responses) to select the simulation sample size, addressing a gap in earlier LLM-based studies that did not systematically gauge or correct for misalignment. In that sense, their approach parallels conformal inference’s reliance on real data for coverage calibration—except they repurpose it for a new “simulation from an imperfect distribution” challenge.", "essential_references_not_discussed": "To the best of my knowledge, it appears to include the necessary references.", "other_strengths_and_weaknesses": "**Strengths**  \nThe paper addresses a relatively new and interesting task—survey response simulation—where large language models are used to estimate the real population's responses for a given questionnaire. More importantly, it provides a framework for estimating the survey’s statistical results using a confidence interval, which gives a quantifiable range with a $1-\\alpha$ coverage guarantee.\n\n**Weaknesses**  \n1. **Writing and Structure**: The paper’s presentation could be more accessible. Specifically, the abstract is difficult to parse initially; including a brief explanation or definition of the survey simulation task at the beginning would help readers understand the purpose immediately.  \n2. **Experimental Analyses**: While the test miscoverage rate is quite important for practical validation, it is relegated to the appendix. Similarly, the discussion about selecting the simulation sample size \\(k\\) (which indicates misalignment between the LLM and real data) might benefit from its own dedicated section or case study, rather than being nested within the \"Experiment Results\" section.", "other_comments_or_suggestions": "See the section \"Other Strengths And Weaknesses\" above.", "questions_for_authors": "1. When creating \\(K\\) user profiles based on a given range of features (e.g., gender, age), how exactly are these profiles sampled? From what I understand, it seems they are randomly drawn from the dataset where such profile information already exists. But if we have a survey question without corresponding real-data profiles, do you then sample feature values uniformly from the space of possible profiles? Or use some other method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lZf4Dyo3N8", "reviewer": "Reviewer_RRnK", "summary": "The paper proposes a novel framework for converting synthetic survey responses generated by large language models (LLMs) into statistically valid confidence intervals for population parameters. By focusing on uncertainty quantification, the authors develop a principled method to adaptively select the simulation sample size (denoted as k̂) such that the confidence intervals maintain a prescribed coverage probability. The paper supports its contributions through rigorous theoretical guarantees—including coverage and sharpness results—and extensive experiments on two real-world datasets (EEDI and OpinionQA) using several state-of-the-art LLMs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review. Below are our responses. We hope they address your questions and concerns.\n\n**Claims And Evidence**\n\n>*... the discussions focus on multiple-choice survey responses... which raises questions about its generality.*\n\nOur motivating example in Section 2 focuses on binary responses for the purpose of illustration and ease of understanding. However, our general method and theory in Section 3 apply to a much broader range of survey questions. Specifically, they are applicable as long as the raw responses can be converted into a quantifiable metric. This includes the possibilities of text responses as well as a survey consisting of multiple related questions. Moreover, most existing works on assessing the fidelity of LLM simulations (e.g., those cited in \"Related Works\") focus on multiple-choice questions.\n\n**Methods And Evaluation Criteria**\n\n>*The proposed method ... is also not tailored for LLMs*\n\nThe reviewer is correct that our method treats the LLM as a black box. An advantage is that the method works with LLMs with arbitrary fidelities and architectures. This ensures its wide applicability, as many LLMs are not open-source. In addition, our framework also applies to any black-box generative models beyond LLMs, such as image or video generators. \n\n>*LLMs can introduce significant bias...*\n\nWe agree with the reviewer that LLMs can introduce significant bias against certain populations. However, as long as the target population is specified and we have human responses from that population, our method will construct a reliable confidence set for that population regardless of any bias the LLM may have. The intuition is that by comparing synthetic responses with human responses, we choose a suitable effective sample size $\\hat{k}$ so that the bias of LLM is dominated by the random error associated with $\\hat{k}$ synthetic samples. Therefore, the confidence set does take care of the LLM's bias for the given population.\n\nIn addition, recent works have investigated mitigating LLMs' bias towards certain populations, through fine-tuning, prompting, etc. One may use these techniques to simulate more truthful responses from the LLM, and apply our method over these simulations to obtain tighter and more informative confidence sets.\n\n>*A considerable portion of the theoretical and experimental work is devoted to the binary response setting...*\n\nPlease refer to our response to your comment in \"Claims And Evidence\".\n\n**Experimental Designs Or Analyses**\n\n>*It also relies on some assumptions such as the independence of survey questions and the applicability of CLT approximations...*\n\nIn our experiments, there are at least 100 human responses per question. As a standard practice, we expect the CLT-based confidence intervals $\\mathcal{S}_j$ to be accurate.\n\nThe independence of survey questions is used for theoretical analysis only. It is similar to the assumption of i.i.d. data points in the statistical machine learning literature. We totally agree that it may not hold in practice, including the real-world datasets in our paper. However, our experiments show that even when this assumption may be violated, our method still achieves good coverage empirically.\n\n**Weaknesses**\n\n>*My main concerns are the limited scope of the paper... neglecting the bias on underlying LLMs.*\n\nPlease refer to our response to your second comment in \"Methods And Evaluation Criteria\".\n\n**Other Comments Or Suggestions**\n\n>*The paper writing is difficult to follow in general. It would be better if the authors could clarify the scope of the paper earlier (in the introduction, for example).*\n\nThanks for the suggestion. To clarify the main goal of our paper, we will change the first sentence of the abstract to \"We investigate the use of large language models (LLMs) to simulate human responses to survey questions, and perform uncertainty quantification to gain reliable insights.\" \n\nTo clarify the survey simulation procedure that our paper is concerned with, we will add the following sentence to the first paragraph of Section 1: \"The typical simulation procedure consists in prompting an LLM with a real or fictional persona as well as a survey question, and collecting the LLM's responses.\"\n\n**Questions**\n\n>*Since we're having access to human responses, can we use it to calibrate LLM responses to reduce bias?*\n\nWe believe that this is an important consideration in LLM simulation. In Section 5, we have mentioned the possibility of combing our approach with debiasing methods. As an example, one may first use part of the human data for LLM alignment, and then apply our method with the rest of the human data. However, this is beyond the scope of our current work, as our goal is to convert any black-box LLM simulations to reliable confidence intervals, regardless of the bias of the LLM. Thus, we will leave it as future work to use human data to debias LLM simulations.", "claims_and_evidence": "The paper makes several claims:\n\n- It introduces a data-driven method to determine an effective sample size that produces confidence intervals with prescribed coverage probabilities.\n- The method provides theoretical guarantees on both coverage and sharpness, implying that the selected sample size yields neither overly conservative nor excessively loose intervals.\n\nWhile the theoretical analysis seems sound, the discussions focus on multiple-choice survey responses (with binary or limited choice scenarios), which raises questions about its generality. Given that LLMs excel at generating diverse and open-ended responses, this narrow focus does not fully leverage their potential in simulating human opinions.", "methods_and_evaluation_criteria": "- The scope of the paper is not clearly stated in the introduction or problem statement. The paper seems to focus only on multiple-choice survey, which is quite limited given the advantage of LLMs in human simulation is to produce opinions on open-ended questions. Therefore, the taste of LLMs in this paper is not very clear. The proposed method to estimate the sample size is also not tailored for LLMs.\n    \n- The method relies entirely on the base LLM to produce synthetic responses. It is well-known that LLMs can introduce significant bias—especially against minority groups—which is not adequately addressed in the current framework.\n\n- A considerable portion of the theoretical and experimental work is devoted to the binary response setting, which may limit the practical applicability of the method to more complex survey scenarios.", "theoretical_claims": "I haven't verified proofs in the paper.", "experimental_designs_or_analyses": "The experimental designs for the EEDI and OpinionQA datasets are methodologically sound within the limited scope of multiple-choice surveys. \n\nIt also relies on some assumptions such as the independence of survey questions and the applicability of CLT approximations, which would be beneficial if the authors could investigate the effect if those assumptions do not hold. Especially the independence of survey questions.", "relation_to_broader_scientific_literature": "The paper focuses on the problem of applying LLM for human simulation. However, the scope is limited since they only focus on determining the number of sample sizes needed, which has limited scope.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "My main concerns are the limited scope of the paper with the main emphasis on determining the number of samples needed, neglecting the bias on underlying LLMs.", "other_comments_or_suggestions": "The paper writing is difficult to follow in general. It would be better if the authors could clarify the scope of the paper earlier (in the introduction, for example).", "questions_for_authors": "Since we’re having access to human responses, can we use it to calibrate LLM responses to reduce bias?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qmbLwRlXn8", "reviewer": "Reviewer_6eYT", "summary": "This study proposed a method to convert LLM simulated results into confidence sets for population parameters of human responses. It provides an estimate for the optimal number of simulations with theoretical proofs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review. Below are our responses. We hope they address your questions and concerns.\n\n**Claims And Evidence**\n\n> *the current study does not provide results for other confidence set construction.*\n\nWe clarify that the coverage guarantee (Theorem 3.3) of our general method in Section 3 holds for any confidence set construction procedures, such as inverting hypothesis tests, the bootstrap, and the empirical likelihood ratio function. In our experiments, we constructed confidence intervals based on Hoeffding's concentration inequality, because it is arguably the simplest construction procedure that has a valid coverage guarantee for any finite sample size.\n\n**Methods And Evaluation Criteria**\n> *I do not understand why simply showing this value [$\\hat{k}$] across different alphas demonstrates the validity of the proposed method.*\n\nThe reviewer is correct that the value of $\\hat{k}$ does not demonstrate the validity of our method. Instead, we should verify the coverage guarantee of the induced confidence sets as claimed in our theorems. We examine this through the miscoverage proxy $\\tilde{L}(\\hat{k})$ in Equation (18). According to the explanations there, $\\mathbb{E}[\\tilde{L}(\\hat{k})]\\le\\alpha$ implies a controlled miscoverage probability $\\mathbb{P}(\\theta(\\psi)\\not\\in\\mathcal{S}^{\\mathsf{syn}}(\\hat{k}))\\le\\alpha$. Thus, to check the validity of our method, we will perform a statistical hypothesis test on $H_0:\\mathbb{E}[\\tilde{L}(\\hat{k})]\\le\\alpha$ against the alternative $H_1:\\mathbb{E}[\\tilde{L}(\\hat{k})]>\\alpha$. Due to space constraint, we kindly ask the reviewer to refer to the tables of $p$-values in our response to reviewer xDnM. In short, all $p$-values are reasonably large, supporting the null hypothesis $\\mathbb{E}[\\tilde{L}(\\hat{k})]\\le\\alpha$ at, say, the 0.05 significance level. This shows that the miscoverage probability is effectively controlled by our method. We will add these results and discussions to the paper.\n\n>*... since this study aims to estimate the optimal number of simulations that can best represent the human response distributions, why not evaluate the method by actually simulating the responses $\\hat{k}$ times on the two datasets and compute the distribution differences between the simulated responses and real human responses?*\n\nWe thank the reviewer for raising this potential connection. Our method aims to select a simulation sample size $\\hat{k}$ such that the confidence set $\\mathcal{S}^{\\text{syn}}(\\hat{k})$ constructed from $\\hat{k}$ simulated samples can cover the human population parameter with a prescribed probability while being as tight as possible. Therefore, a natural way to evaluate our method is to examine its coverage probability as well as the size of the confidence set. The results are summarized in Appendix C.4.2 and Table 3 in the original manuscript. Please also refer to our response to your comment in \"Methods And Evaluation Criteria\". Those results indeed reflect the distributional difference between the simulated and real responses: for instance, high coverage probability plus small size would certify small difference. We did not directly simulate $\\hat{k}$ samples and directly compute the distributional differences, as it would require defining (additional) discrepancy metrics.\n\n**Weaknesses**\n\n> *(1) The main weakness lies in the clarity of the evaluation and lack of a more direct evaluation method ...*\n\nPlease refer to our responses above.\n\n> *(2) It is not intuitively straightforward why the domains of the survey may not be a big factor when designing the approach.*\n\nIt is true that our procedure for selecting the simulation sample size $\\hat{k}$ as well as the validity of our approach does not rely on the specific survey domain. However, the survey domain does implicitly play a role in the outcomes of our approach. Different domains may have different target populations and formats of survey questions (e.g., the survey responses could be multiple-choice, numerical, or in a Likert scale). Moreover, the simulation power of the LLM can also be very different across different domains. All these factors can cause our method to choose different simulation sample sizes $\\hat{k}$ and confidence sets $\\mathcal{S}^{\\text{syn}}(\\hat{k})$, but the coverage guarantee will always hold.\n\n**Questions**\n\n> *(1) Would different hyperparameter settings of the LLMs influence the final results...?*\n\nThe coverage guarantee of our method always holds regardless of the choice of LLM. On the other hand, the hyperparameter setting of the LLM does affect the quality of the LLM simulations. This will in turn affect the chosen simulation sample size $\\hat{k}$ and the size of confidence set $\\mathcal{S}^{\\text{syn}}(\\hat{k})$. For example, if a certain hyperparameter setting improves the quality of the LLM simulations, then we expect our method to select a larger simulation sample size $\\hat{k}$, which gives a tighter confidence set.", "claims_and_evidence": "Claims: \n\n(1) A general mathematical formulation of uncertainty quantification for LLM-based survey simulations was provided.\n\n(2) The proposed method converts simulated responses into confidence sets for population parameters of human responses. It adaptively selects simulation sample size. This method is applicable for any LLMs, irrespective of its fidelity, and any methods for confidence set construction.\n\nThe first claim is true. I am not sure about the second claim as I do not fully understand the evaluation method. Additionally, the current study does not provide results for other confidence set construction.", "methods_and_evaluation_criteria": "I am not fully following the evaluation of the selected sample size. Figures 2 and 3 show the average \\hat{k} against different alphas. I understand that the number \\hat{k} is the measure that the proposed study aims to predict. I do not understand why simply showing this value across different alphas demonstrates the validity of the proposed method. \n\nMoreover, since this study aims to estimate the optimal number of simulations that can best represent the human response distributions, why not evaluate the method by actually simulating the responses \\hat{k} times on the two datasets and compute the distribution differences between the simulated responses and real human responses?", "theoretical_claims": "Yes, I checked the proof. Based on the assumptions, it seems correct. However, I have doubts in the assumption. I wonder if the domain of the surveys would influence the simulation results.", "experimental_designs_or_analyses": "Yes. However, I think this lacks a direct comparison of the simulated responses with \\hat{k} and the real human responses.", "relation_to_broader_scientific_literature": "The potential contribution would be impactful as this method would provide an effective way to select the number of simulations, saving the inference cost.", "essential_references_not_discussed": "References are satisfactory.", "other_strengths_and_weaknesses": "Strengths\n\n(1) The research question is important. The potential contribution could be impactful as it may provide a principle in using LLMs for survey response simulation.\n\n(2) This study provides theoretical support for selecting the number of simulations.\n\nWeaknesses\n\n(1) The main weakness lies in the clarity of the evaluation and lack of a more direct evaluation method. As aforementioned, it would be better if the authors could further elaborate on the evaluation of the selected simulation size. Additionally, it would strengthen the study if the simulated results using \\hat{k} can be compared directly with the real human responses.\n\n(2) It is not intuitively straightforward why the domains of the survey may not be a big factor when designing the approach. It would be better if this can be further discussed.", "other_comments_or_suggestions": "I did not find typos.", "questions_for_authors": "Please see my main questions (concerns) for the evaluation method.\n\nOther questions:\n\n(1) Would different hyperparameter settings of the LLMs influence the final results (e.g., decoding strategy)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LApHi3fZSo", "reviewer": "Reviewer_XMzS", "summary": "This study investigates the reliable use of simulated survey responses generated by large language models  from the perspective of uncertainty quantification. The proposed approach transforms synthetic data into confidence intervals for human response group parameters, addressing distributional shifts between simulated and real populations. A key innovation is determining the optimal number of simulated responses: too many lead to overly narrow confidence intervals with poor coverage, while too few result in overly loose estimates. To address this, the method adaptively selects the sample size to ensure effective coverage on average. It is broadly applicable to any LLM, regardless of fidelity, and any confidence interval construction process. Additionally, the selected sample size quantifies the discrepancy between the LLM and the target population. The study demonstrates the effectiveness of this approach through experiments on real datasets and LLM-generated responses.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and the encouraging score. Below are our responses.\n\n**Weaknesses**\n\n> *1. I extremely expected to see some discussions on the nature and characteristics of language models for simulation, leading to methodological design and theoretical analysis based on these aspects. However, the paper does not seem to be structured in this way; instead, it treats the model more as a black box.*\n\nThe reviewer is correct that our method treats the LLM as a black box. An advantage is that the method works with LLMs with arbitrary fidelities and architectures. This ensures its wide applicability, as many LLMs are not open-source.\n\n> *2. The paper starts by directly positioning \"the number of synthetic samples\" as the core research direction, which feels somewhat unnatural. This claim assumes that the distribution simulated by the language model changes with the number of samples and that there exists an optimal quantity range that aligns better with the real-world distribution. This conclusion is non-trivial and requires further justification.*\n\nThanks for raising the confusion. To perform simulation, one must specify a sample size $k$. Thus, it is an important quantity to determine.\n\nWe would like to clarify that the distribution of the simulated responses does not change with the simulation sample size $k$. Instead, the sample size $k$ involves a trade-off between the width and the coverage validity of the confidence interval. Simulating too few samples leads to an overly wide confidence interval, while simulating too many samples causes the confidence interval to concentrate around the population parameter of simulated responses rather than the human responses. Our method is designed to find a simulation sample size $\\widehat{k}$ that balances this trade-off.\n\n**Other Comments Or Suggestions**\n\n> *1. I think the figure 1 does not provide much useful information and may not be suitable for inclusion in the main body of the paper.*\n\nThanks for the suggestion. We will move the figure to the appendix.\n\n> *2. Adding a conclusion section would help readers quickly grasp the key findings of the study.*\n\nThanks for the suggestion. In the first two sentences of Section 5, we have summarized the main goal and methodology of our study. We will add the following sentence to summarize the key findings: \"Numerical experiments on real datasets verified the coverage guarantees of our approach, and revealed that LLMs exhibited higher fidelity in simulating opinions to social problems than in simulating student responses to mathematics questions.\"", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths \n1. The paper addresses a highly valuable problem. Given the large number of simulation experiments based on large language models, ensuring their reliability is a fundamental issue.\n2. The paper provides a detailed formal discussion, offering a strong reference for future research.\n3. The study includes comprehensive experiments.\n\n### Weaknesses\n1. I extremly expected to see some discussions on the nature and characteristics of language models for simulation, leading to methodological design and theoretical analysis based on these aspects. However, the paper does not seem to be structured in this way; instead, it treats the model more as a black box.\n2. The paper starts by directly positioning \"the number of synthetic samples\" as the core research direction, which feels somewhat unnatural. This claim assumes that the distribution simulated by the language model changes with the number of samples and that there exists an optimal quantity range that aligns better with the real-world distribution. This conclusion is non-trivial and requires further justification.", "other_comments_or_suggestions": "1. I think the figure 1 does not provide much useful information and may not be suitable for inclusion in the main body of the paper.\n2. Adding a conclusion section would help readers quickly grasp the key findings of the study.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers were divided on this paper, so I have also carefully reviewed it. Having published multiple times in the area of silicon sampling, I consider myself well qualified to assess it, and I was quite happy with the paper and its contributions, so I recommend acceptance.

The paper is solid. It addresses an important problem; it suggests a clean and workable solution; it nicely blends theory and experiment; and is well-written.

My only hesitation is that the the calibration questions must bear some similarity to the target question; assessing that similarity in any sort of formal way is a hard problem. However, I believe the strengths of the paper outweigh that weakness.
Decision: Accept (poster)