SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
Authors: Tiancheng Hu
Joachim Baumann
Lorenzo Lupo
Nigel Collier
Dirk Hovy
Paul Röttger
Venue: ICLR 2026 Poster (2026)
Keywords: human behavior simulation
large language models
benchmarking
computational social science
human-AI alignment
calibration
human-centered AI
Primary area: datasets and benchmarks

Abstract: Large language model (LLM) simulations of human behavior have the potential to revolutionize the social and behavioral sciences, if and only if they faithfully reflect real human behaviors. Current evaluations of simulation fidelity are fragmented, based on bespoke tasks and metrics, creating a patchwork of incomparable results. To address this, we introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation. By unifying 20 diverse datasets covering tasks from moral decision-making to economic choice across a large global participant pool, SimBench provides the necessary foundation to ask fundamental questions about when, how, and why LLM simulations succeed or fail. We show that the best LLMs today achieve meaningful but modest simulation fidelity (score: 40.80/100), with performance scaling log-linearly with model size but not with increased inference-time compute. We discover an alignment-simulation tradeoff: instruction tuning improves performance on low-entropy (consensus) questions but degrades it on high-entropy (diverse) ones. Models particularly struggle when simulating specific demographic groups. Finally, we demonstrate that simulation ability correlates most strongly with knowledge-intensive reasoning (MMLU-Pro, $r=0.939$). By making progress measurable, we aim to accelerate the development of more faithful LLM simulators.

Published as a conference paper at ICLR 2026
SIMBENCH: BENCHMARKING THE ABILITY OF LARGE LANGUAGE MODELS TO SIMULATE HUMAN BEHAV- IORS
Tiancheng Hu', Joachim Baumann", Lorenzo Lupo", Nigel Collier!, Dirk Hovy and Paul Röttger
"University Cambridge, Stanford University, Bocconi University, 4 UUniversity of Oxford hh65 60 cam. ac.uk .uk
Data
Code
Website
ABSTRACT
Large language model (LLM) simulations of human behavior have the potential to revolutionize the social and behavioral sciences, if and only they faithfully reflect real human behaviors. Current evaluations of simulation fidelity are fragmented, based on bespoke tasks and metrics, creating a patchwork of incomparable results. To address this, we introduce SIMBENCII, the first large-scale, standardized bench- mark for robust, reproducible science LLM simulation. By unifying 20 diverse datasets covering tasks from moral decision decision-making to economic choice across large global participant pool, SIMBENCH provides the necessary foundation to ask fundamental questions about when, how, and why LLM simulations succeed fail. We show that the best L.MSs today achieve meaningful but modest simulation fi- delity (score: 40.80/100), with performance scaling log-linearly with model size but not with increased inference-time time compute. We discover an atignment-simulation tradeoff:: instruction tuning improves performance on low-entropy (consensus) questions but degrades it on high-entropy (diverse) ones. Models particularly struggle when simulating specific demographic groups Finally, we demonstrate that simulation ability correlates most strongly with (MMI.U-Pro r 0.939). By making progress measurable, we aim accelerate the development of more faithful LLM simulators.
reasoning
We combine 20 datasets a unified format.
We test the ability of LLMs simulate group-level responses.
A crain will kill 5 people the track. You can flip switch divert the tain
side taak where it will kill just people.
ChaosNLI
MoralMachineC
Choices13k
AfroBarometer
What do you do?
OpinionQA
Bigs
A: Flip the switch
B:
Do nothing
NumberGame
DICES990
WisdomO@CCrowds
Jester
Each dataset contains multiple-choice questions.
A
B
A
A
A
B
LatinoBarometto
ISSP
Figure 1: SIMBENCH is the first large-scale benchmark to evaluate how well LLMs can simulate group-level human behavior across diverse simulation settings and tasks.
INTRODUCTION
Large-scale human experiments and surveys have long been essential tools for informing public policy, commercial decisions, and academic research. Running experiments and surveys, however, is costly

---

Published as a conference paper at ICLR 2026
and time-consuming. Large language models (LLMs) may help address this challenge by simulating human behaviors quickly and at low cost, to complement or even substitute human studies. This prospect, alongside encouraging early evidence the efficacy of LLMs simulators (Aher et al.)
2023
Argyle et al. 2023: Horton 2023), has motivated a large body of recent work across many disci-
plines investigating the ability of LLMS to simulate human behaviors (Bishee et al. [002] Dominguez
Olmedo et al. 2024 Manning et al., 2024 Binz et al. 2025 Hu & Collier 2025 inter alia).
However, this rapid exploration has produced a fragmented body of evidence. Most studies evaluate a narrow set of LLMs on a specific task, yielding varied and sometimes contradictory results that make it difficult to draw broader conclusions ($5). The field lacks a unified framework determine when, how, and why LLM simulations succeed, or how to train better simulators.
To confront this challenge, we introduce SIMBENCH: the first large-scale, standardized benchmark for group-level human behavior simulation. By harmonizing 20 diverse datasets, which span moral dilemmas, economic games, and psychological assessments across a vast global participant pool, SIMBENCII enables rigorous measurement and comparison of simulation fidelity across models, tasks, and populations.
Using SIMBENCH, we move beyond isolated experiments to build a comprehensive picture of LLM simulation. We structure our investigation around six research questions. We first establish a baseline, asking how well current LLMs perform at social simulation (RQ1) and how characteristics like model size and inference-time compute affect their simulation ability (RQ2). We find that top models achieve meaningful but modest simulation fidelity (top score: 40.8 40.80/100), though performance scales lig-lilearly with size and, surprisingly. is not improved by scaling inference-time compute. Next, we explore the sources of this variance, asking how task selection (RQ3) and human response plurality (RQ4) affect simulation fidelity. We find that fidelity varies substantially by task, and uncover an sesggmenn-simmllateee improves performance on low-entropy (consensus) questions but harms performance on high-entropy (pluralistic) questions. A causal analysis confirms this tradeoff stems from two opposing forces: beneficial instruction-follene following effect and a harmful entropy-reuuctie LLMs are better at simulating some demographic groups than others (RQ5) and what extent simulation ability correlates with other model capabilities (RQ6). We show models struggle most with religiousideologicaal groups and that simulation ability correlates most strongly with deep, wwwledge-intensiveeec intensive reasoning (MMLU-Pro, r = 0.939).
tradeoff: instruction tuning's 's mode-seeking objective systematically
effect. Finally, we ask if
knowledge
Progress in Al is only possible through rigorous evaluation, and large-scale benchmarks such as MMLU (Hendrycks et al. (2021) have significantly contributed to improvements in LLM capabili- ties. In the same spirit, SIMBENCH provides the foundational infrastructure needed to move LLM simulation from collection of ad-hoc studies measurable and systematic science. We note that while predicting group-level response distributions is the instrument used here for benchmarking, we view this as a foundational proxy for the broader capability of human behavior simulation. We acknowledge that full human simulation inherently includes interactive and complex dynamics not fully captured by static response distributions (see Appendix (A). We have Project Website, and all of SIMBENCH is available on GitHub and HuggingFacce
CREATING SIMBENCH
2.1 DATA CURATION
To create SIMBENCH, we combine a repository-drif approach, where we query major social and behavioral science repositories (e.g., Harvard Dataverse, ICPSR, OSF), and a literature-driven approach, where we identify key papers relevant fields and trace back to their underlying data sources. We then apply strict set of selection criteria to all candidate datasets: large participant counts, to ensure meaningful group-level distributions; permissive licensing to allow for redistri- bution; single-turn, self-contained questions, to establish standardized evaluation paradigm free from multi-turn or contingent interactions: multiple-choice or ordinal response formats, to enable quantitative evaluation; and English-language questions validated translations for consistency.
We complement these criteria with a curation strategy that balances competing objectives. We prioritize novelty, favoring datasets not previously used in LLM simulation evaluation, but also
2

---

Published as a conference paper at ICLR 2026
ensure backward comparability by including well-established benchmarks (e.g., OpinionQA, ChaosNLI). Furthermore, we prioritize datasets with rich sociodemographic data to enable fine- grained analysis of specific subpopulations ($2.4). However, we make targeted exceptions for datasets like Jester and Choices 3k, which, despite lacking demographic data, provide unique and essential task diversity.
This principled curation process yields the 20 datasets that comprise SIMBENCH, which we list in Appendix L providing details on participants and example questions. To demonstrate the rigor of our process, and as a resource for the community, we list datasets that were considered but ultimately excluded in Appendix B, Crucially, SIMBENCH is fully modular by design, so that future work can easily add more datasets using the processing pipeline described in belbo..ww.
2.2 BENCHMARK PROPERTIES
As whole, SIMBENCH is defined by two key properties: task diversity and participant diversity.
1) Task Diversity: SIMBENCII tasks are highly diverse in terms of the human behavior they cap- ture. SIMBENCII includes decision-making questions (e.g., in Choices13k, MoralMachine), where participants are presented with a set of actions that concern themselves, and they have to select the action they would hypothetically take. SIMBENCH also includes self-assessment questions (e.g.. in OpinionQA, OSPsychBig5), where participants are presented with a set of descriptions or attributes, and they have select the one that best describes themselves. Further, SINBENCH includes judgment questions (e.g., in ChaosNLI and Jester) where participants are presented with some external object and a choice of labels, and they have select the label they think fits best. Lastly, SIMBENCI includes problem-solving questions (c.g., in WisdomOfCrowde and OSPsychMGKT), where participants are presented with set of answers a factual question, and they have to select the answer they think is correct. Consequently, LLMs have to accurately simulate several distinct types of human behavior order to perform well on SIMBENCH.
2) Participant Diversity: SIMBENCH captures rich demographic landscape spanning more than 130 different countries across six continents (see Appendix [ for full country-level breakdown). SIMBENCH prioritizes international representation: samples from the Anglosphere West constitute only 27.9% of the datal] This substantial global scope is driven by a diverse collection of sources: datasets (e.g., Latinobarometro, Afrobarometer) exclusively feature participants from regions outside the US, datasets (e.g.. GlobalOpinionQA, TISP) draw from multi-country samples across different continents, and 2 datasets collect responses from global pool of internet users. Importantly, 8 out of the 20 datasets employ representative sampling techniques, enhancing the ecological validity of these constituent components. To perform well on SIMBENCH, LLMs must therefore demonstrate the ability to accurately simulate the behavior of human participants across diverse cultural, linguistic, and socioeconomic backgrounds
2.3 UNIFYING SIMBENCH DATASET FORMATS
core contribution of SIMBENCII is the harmonization of 20 heterogeneous datasets into one standardized format. This process ensures that LLM simulation ability can be compared rigorously across diverse tasks and populations.
Question Normalization:
We standardize all items into multiple-choice format, making minimal edits that primarily consist of mapping existing discrete options to standardized letter keys, each corresponding single token, enable clean extraction of per-option probabilities from base models while preserving the original experimental structure. For the few datasets with continuous scales (Jester), we map responses discrete bins. We further ensure consistency by collapsing answer options where appropriate, limiting the maximum to 26 choices (though typically fewer than
We define the Anglosphere West the U.S., Canada, U.K., Australia, and New Zealand. Even using broader definition of "the West" that includes Western Europe, these nations account for less than half (45.9%)
the benchmark.
Note that, while some constituent datasets recruit representative samples, SIMBENCH whole is not fully representative of any single population. We discuss this limitation in Appendix A
3

---

Published as a conference paper at ICLR 2026
10), and using the official English-language versions of all questions This is deliberate choice to standardize the evaluation and avoid confounding simulation ability with multilingual performance capabilities, ensuring that differences in scores reflect simulation fidelity rather than translation quality, even for datasets originally collected in local languages.
Response Aggregation: To evaluate group-level simulation, standardize all data into group- level probability distributions, For the majority of our datasets, which provide raw individual-leve responses, we create these distributions by aggregating the data ourselves. sostfftratiiiiation weights are applied whenever applicable (e.g., ESS). For the few datasets that are already provided in an aggregated format (e.g., GlobalOpinionQA), we process and normalize their existing statistics to conform to the SIMBENCH schema.
We create the simulation targets - the ground-truth response distributions that models must predict in two ways: 1) Default Grouping. For every question in a dataset, we create baseline target by aggregating responses from all participants. This represents the "default" population for that dataset (e.g.. "US-based Amazon Mechanical Turk workers") and is used to measure general simulation ability. 2) Specific Grouping. For datasets with rich sociodemographic data, we create more fine- grained targets by aggregating responses from participants sharing a specific attribute (e.g., age or gender). This allows us to evaluate LLM ability to simulate narrower, more specific demographic groups. We detail the available grouping variables for each dataset in Appendix
Each simulation target is paired with a prompt that describes the corresponding group. Overall, this harmonization process yields 10,930,271 unique question-gree grrsup simulation targets. From this set, curate our final benchmark splits ($2.4) to enable robust evaluation of LLM simulation capabilities. We note that while training data contamination remains an inherent risk, our zero-shot aggregation task largely mitigates this by testing distributional prediction (see Appendix A for a full discussion).
2.4 SIMBENCH SPLITS
The full set of over 10 million simulation targets is too vast for practical evaluation. We therefore curate two benchmark splits, each designed probe different facet of LLM simulation ability.
1) The SimBenchPop split covers all questions in all 20 datasets after processing as in $2.3 We combine each question with the dataset-specific default grouping prompt to create one unique test case, resulting in 7.167 test cases. We obtain the response distribution for each test case by aggregating all individual responses to that test case over all participants in that dataset. Conceptually, SimBenchPop measures the ability of LLMs to simulate responses of broad and diverse human populations.
2) The SimBenchGrouped split draws from the five large-scale survey datasets in SIMBENCH (Afrobarometer, ESS, ISSP, Latinobarómetro, and OpinionQA). which are the only datasets with sufficient participant counts yield reliable response distributions even when conditioning on a specific demographic attribute (e.g.. age 30-49). For each dataset, we select questions that exhibit significant variation across demographic groups, ensuring that the benchmark captures meaningful demographic differences in responses. This results in 6.343 test cases overall. For more details on the sampling process, see Appendix C Conceptually, SimBenchGrouped measures the ability of LLMS to simulate responses from narrower participant groups based specified group characteristics 5
3 EXPERIMENTAL SETUP
Tested Models: To demonstrate the utility of SIMBENCH and answer our six research questions (.)) we evaluate 45 recent LLMs. This includes both commercial and open-weight, base and instruction -unnee models, with sizes ranging from 0.5B to 405B parameters. Table lists all models.
'We note that simulation ability may plausibly be correlated with prompt language, and encourage future work in this direction.
'Ideally, we would also like to measure LLM simulation ability for intersectional groups that combine multiple characteristics (e.g., female + age 30-49). However. selecting on multiple characteristics substantially decreases group size, thus increasing sampling noise in the response distributions. Reliable evaluation of intersectional group simulation ability would require datasets with more participants than have access
4

---

Published as a conference paper at ICLR 2026
Model Elicitation: For each model, we collect predictions for the two main splits of SIMBENCH ($2.4). To obtain model response distributions, we use one two methods, depending on model type: 1) For base models, we directly extract token probabilities for each response option based on first-token logits. This a natural way of eliciting a distribution out of an L.L,, especially a base L.M. 2) For instruction-tune ieruue models, distribution prediction (Tian et al. 2023; Wang et al. 2024 Meister et al. 2025 and use verbalized distributions (e.g., "Option A: 30%, Option 70% elicited through prompting. We empirically validate this methodological choice in Appendix E] which provides strong evidence that verbalized distributions substantially and consistently outperform direct token probabilities for instruction tunnd models. This ensures each model class is evaluated using its most suitable answer elicitation method. For implementation details and prompt formats, see Appendix D
follow recent literature on LLM calibration and
To measure L.M simulation ability, we derive the SIMBENCH score S from the Total Variation Distance (TVD). Conceptually. quantifies the improvement of a model's prediction Q over a uniform baseline U, relative to the human ground truth P:
Evaluation Metric:
S(P,Q) = 100 (1 TVD(P, TVD(P,
where score of 100 indicates perfect alignment and 0 indicates performance equivalent to random guessing. Since in practice a direct point-wise calculation undefined when the question-level human distribution is uniform (P U), to ensure numerical stability, we compute the score S; for each specific test case i by normalizing against the dataset-level mean baseline:
TVD(P; Q.) (1 (1- (1- [Dt EjED TVD(P,.
S. = 100
=1000
(1-
where the denominator represents the average TVD between human responses and the uni- form distribution across all test cases j in the dataset D. This ensures the metric remains ro- bust across datasets with varying entropy. The final reported score for model is the average of S; across all evaluated test cases. As described in $2.4. cap each dataset at 500 questions to limit over-representation of larger datasets; datasets with fewer total questions naturally con- tribute fewer test cases and thus carry propor- tionally less weight in the overall score.
Table 1: Overall simulation ability of represen- tative LLMs measured by SIMBENCH score S averaged across the two main splits of SIMBENCH. Reasoning models are highlighted in italics. A full table with all 45 models is in Appendix Table
Model
Type
Release
S (1)
Top-Performing Models
Claude-3.7-Soman
Instr. Closed
40.80
Claude- 3.r.e..n---SS------
Instr.
Closed
39.46
GPT-4.1
Instr.
Closed 34.55
DeepSeek-RI
Instr. Open
34.52
04-mini-high
Instr. Closed
28.99
Llama-3. 1-tma- --05-5--
Instr. Open
28.40
RESULTS
Dwen2.5-728-Insstee
Instr. Open
27.61
Dwen2.5-32BB-n. IInstrcct
Instr.
Open
23.76
OLMo-2-32B-DD DPO
Instr.
Open
19,80
4.1 RQI:
Top-Performing Base Models
GENERAL SIMULATION ABILITY OF LLMS
OLMo-2-32B
Base
Open
15.90
OLMo-2-13B
Base
Open
13.83
To evaluate general simulation ability, we mea- sure overall SIMBENCU score S averaged across the two main splits of SIMBENCH in Table Tand Appendix Table
Owen2.5-72B 5-7BB
Base
Open
13.34
Qwen2.5-32B
Base
Open
12.27
Models Performing Below Uniform Baseline
Gemma-3-4BB Bwen2.5-3B-Innstee LL.Mo-2-7B-nnsruue
Base
Open
-0.65
Instr.
Open
-12004
We find that the best LLMs achieve meaning- ful but modest simulation fidelity, as measured across the 20 datasets SIMBENCH. Claude-3.7-Sonnet the berf-performing model overall, achiev-
Instr.
Open
-21.36
ing a score of 40.80 out of 100 on SIMBENCH. While this score means the best model's predictions remain closer to uniform distribution than to the human ground truth, nonetheless closes roughly 40% of the gap between the two, demonstrating a genuine and non-trivial simulation signal. The top open-wette weenht model, DeepSeek-R1. scores 34.52. The majority of the 45 models we test perform substantially worse, scoring less than 20. Notably, ten models test score below 0, indicating that their predicted response distributions are, on average, even further away from the true human re- sponse distribution than uniform response distribution. Statistical analysis, detailed in Appendix
5

---

Published as a conference paper at ICLR 2026
confirms that the performance differences between most top-ranked models as well as within each model family are statistically significant. Overall, these results consolidate the mixed findings from prior work into clearer, if somewhat sobering, picture. When evaluated across diverse range of tasks and populations, today's L.MMs do possess genuine, non-trivial simulation abilities but are still far from being consistently reliable, general-purpose simulators. The stark performance differences between models also caution strongly against the use of smaller, less capable models for simulation, many of which perform worse than simple uniform baseline.
4.2 RQ2: IMPACT OF LLM CHARACTERISTICS ON SIMULATION ABILITY
While even the best models struggle to perform well on SIMBENCH, Table also shows clear differences across models. Therefore, we investigate how performance varies depending on model characteristics, specifically 1) model size, and 2) inference-time compute.
1) Model Size
To evaluate the impact of model size simulation ability, we plot SIMBENCH Score against model parameter count for the four LLM families that we can test across multiple model sizes (Figure?) 2 and Appendix Figure Our results suggest that there is log-lineat scaling trend for LLM simulation ability. Across examined model families, for both base and instruction models, an increase in parameter count generally corresponds
increase in SIMBENCH score S, indicating better alignment between predicted and human response distributions. This relationship is robustly supported by model families with comprehensive size variants (e.g.. Qwen2.5, Llama-3.1) though additional data points would be needed to fully characterize the trajectory for families with fewer models (e.g., OLMo). There is also an interaction between model size and the effect of instruction tuning. While instruction-tune models consistently outperform their base counterparts at larger scales (>10B parameters), this relationship appears to invert for smaller models. For example, the OLMo-2 base models outperform their instruction-tuned variants at the 7B and 13B scale. Furthermore, the plot shows that instruction-tuie appear to scale more effectively. The steeper slope of the dashed lines (e.g., for Sive2..5-Instuuee compared to the solid lines suggests that instruction tuning may improve model's 's ability to capitalize increases in parameter count for the simulation task. We present a more comprehensive plot including all evaluated model families in Appendix G which confirms this trend holds across models.
models not only reach a higher peak performance but also
Overall, the clear positive scaling trends across model families suggest that simulation ability is a capability that improves with scale. However, the log-linear nature of this relationship implies diminishing returns from scale alone, reinforcing the need for targeted approaches such as distributionnpreserviin alignment ($44)) to achieve substantial further gains.
30
SimiBench Siroee
10
10
-10
Instruct
1.0
10.0
Parameter Count (Billions)
Figure 2: Model parameter count vs. simulation ability. We measure model size by parameter count and simulation ability by SIMBENCHI score averaged across the two main splits of SIMBENCH. Shaded bands denote 95% confidence intervals around the mean score.
2) Inference- Time Compute To analyze the effects of increasing inference- diference-time compute on LLM simulation ability, conduct two sets of experiments. First, we compare the performance of two distinct o4-mini checkpoints ("low" vs. "high", which vary in the amount of reasoning effort), as well as Claude-3. Staude-3.7-Sonnee 7-Sonnet without reasoning and with 4000-token reasoning budget. Second,

---

Published as a conference paper at ICLR 2026
apply zero-shot Chain-of- Thoughh (CoT) prompting strategy (Wei et al. 2022) to GPT-4.1 and DeepSeek- V3-0324 (see Appendix D for prompt details).
Our results suggest that, actoss the models tested, increasing inference-time compute provides no meaningful benefit for LLM simulation ability. The o4-mini model shows minor improvement (SIMBENCH S score: 27.77 28.99), while Claude-3.--Sonnet's performance slightly decreases (40.80 - 39.46). Similarly, applying CoT prompting leads to a small performance drop for GPT-4.1 (34.55 + 13 and a negligible change for DeepSeek-V3-0333 covers limited number of models, this result aligns with a growing body of recent work showing that inference time compute benefits are highly rask-dependent (Liu al. 2025b, Sprague al. 2025; Gema et al.) 2025) and do not necessarily improve role-playing ability (Feng et al. (202). hypothesize this is because CoT forces an overly rational deliberation that mismatches the often heuristic-driven nature of human responses in SimBench.
(32.89 - 33.16). While analysis
4.3
RQ3: IMPACT OF TASK SELECTION ON SIMULATION FIDELITY
The 20 datasets in SIMBENCH correspond to tasks that are highly diverse in terms of which aspects of human behavior they measure (see 1. Breaking down simulation fidelity by dataset (Figure 3) reveals that simulation fidelity varies substantially across tasks, Models are most successful simulating responses to standard survey questions regarding stated opinions, attitudes, and self- assessments (e.g., OpinionQA, Afrobarometer). However, performance degrades on tasks requiring the simulation of a behavioral choice, whether in risky choice problems (Choices13K or moral dilemmas (MoralMachine).. This finding provides large-scale evidence for "value-action gap" LLMs, echoing recent work (Shen al. 2025) which suggests a misalignment exists between LLM-generated value statements and their actions.
Finally, models struggle severely to capture human response distributions on datasets involving traits or belicfs that conflict with standard alignment objectives. For even the best LLMs, observe extremely poor performance, often worse than a uniform baseline, on datasets measuring Machiavellianismm (OSHsychMACH), conspiratorial beliefs (ConspiracyCorr) or humor rating (Jester). This aligns with a growing body of work (Liu et al., 2024; Kumar et al.] [025 Yi et al. (2025) showing that alignment filters may inhibit the simulation of ""atypical" or counter-normative human perspectives. While most LLMs exhibit these performance patterns, GPT-4.1 I a notable outlier, scoring exceptionally high (61.9) on OSPsychRWAS.
100
Cland-87-8ams-2000
-10
DespSok-R1
713
276
43.7
364
123
26.
-155
DegjBod Vae
363
311
2.2
31.
484
45.3
155
01.0
26.4
36.8
315
25.3
25.0
170
173
43
210
-258
-144
CPT-41
063
45.
913
Mosaltt idber Clacks
Uphionng4
Wind FeddhaeiiFaaal Fmande
rehardMaak MORT
Cheeeh!
OaFerbWWWS
Crapiren
TTP
Lotinn Rnteeeaanmni
Chebre
OSPadsAACH
Aotee
Mar Marbbre
Figure 3: Simulation fidelity by dataset measured by SIMBENCH score S for each of the 20 datasets in SimBenchPop. We show results for the top five models based on results in Table [
4.4 RQ4: THE ALIGNMENT- SIMULATIOO TRADEOFF
Faithful simulation requires models to capture the full spectrum of human opinion, from strong con- sensus widespread disagreement. We operationalize this "response plurality" using the normalized entropy of the human response distribution.
An Empirical Tradeoff' between Alignment and Plurality. Prior work suggests that standard alignment via instruction tuning encourages confident, low-entropy outputs (Brown et al. 2020 Tian jet al.) 2023. Cruz et al. 2024 Meister et al.) 2025). which creates a potential conflict with simulating diverse human perspectives. Our analysis reveals exactly this tradeoff. As detailed in Appendix base models consistently outperform their instruction-tumm counterparts on high-entropy questions, while the inverse is true for low-entropy questions. To precisely quantify this effect, we compute the
7

---

Published as a conference paper at ICLR 2026
Liuces
AShrnBench Stare (FP- PT)
No Inprovenzent
-20
0.0
0.2
0.4
0.8
0.8
1.0
Normalizal Human Response Estropy
Figure 4: The Alignment-Simulation Tradeoff: Instruction Tuning Helps Consensus Ques- tions but Hurts on Diverse ones. We bin questions by the normalized entropy of their human response distribution (X-axis) and plot the mean improvement in SIMBENCH Score when switching from a base model to its instruction-tuned counterpart (y-axis).
change in SIMBENCH score (AS Sintruct Sasse) for 13 model pairs and plot this performance gain from post-training against human response entropy.
The result, shown in Figure is near-perfect negative linear relationship reveals two distinct regimes. On low-entropy questions where humans agree, post-training provides substantial benefit, improving the S-score by up to 40 points. However, as human disagreement (entropy) increases, the benefit of instruction tuning systematically erodes, crossing point of no- improvement around an entropy 0.8. For questions with the highest plurality, post-training becomes actively detrimental, making the aligned model worse simulator than its base counterpart.
00.942). This plot
This empirical finding well-explained by the theoretical framework of reinforcement learning (RL) as Bayesian inference (Levine 2018 Korbak et al. 2022). The pre-training objective of base model typically minimizes a mass-covering KL divergence (DxL (pllg)), which encourages the model (ạ) to place probability mass wherever the true data distribution (p) has mass. This process naturally leads to models that represent the full, multi- modal diversity of human language and opinion seen their training data. In contrast, alignment via Kl.rregularized RI. (e.g., RLHF) minimizes mode-seeking KI. divergence (DxL(gl (gloo) This objective incentivizes the model (ạ) to find and concentrate its probability mass on single, high-reward mode of the target preference distribution (ar), even the cost of ignoring other valid modes. Our results provide strong empirical validation of this theoretical distinction: alignment optimizes for single "best" response, effectively encouraging the model to discard the pluralistic, high- distributions characteristic of genuine human populations.
To formally test this mechanism. causal mediation analysis (Appendix L.3) decomposes the effect of instruction tuning into two larger, opposing forces: large, positive direct effect on performance (+6.46), likely from improved instruction-following and a significant, negative indirect effect (-1.74) mediated by the model's s reduced output entropy.
Case Study: General-Poee General-Purpose vs. Specialist Cognitive Tuning. This formally-decomposed tradeoff is perfectly illustrated by comparing general-purposs purpose instruction tuning with the specialist cognitive tuning of the Centaur models (Binz et al. 2025). Centaur models are Llama models fine-tuned Psych-101, a large dataset of lab experiments, making our diverse SIMBENCH powerful out-of-distribution test of their generalization. Both approaches improve simulation over the 1lama-3.11708 base model, but they do so via opposing mechanisms. General-pur Gurreos--puposee instruction tuning (S = 16.56) leverages the helpful direct effect of alignment, excelling on low-entropy consensus tasks. In contrast, specialist cognitive tuning = 8.54) improves performance by avoiding the harmful indirect effect, preserving the base model's intrinsic ability to capture high- entropy pluralistic responses. The existence of these two distinct-and currently spparate-path to improving simulation underscores key challenge and opportunity: the most faithful simulators of the future will likely need to synthesize the benefits of both general-purpose alignment and distribution- preserving cognitive modeling. Other promising directions include post-hoe weight interpolation (Hu) et al. 2025) and inference- cimeeee-emmeeeee methods to amplify system prompt adherence beyond the default assistant persona (Dong et al. 2026).
8

---

Published as a conference paper at ICLR 2026
4.5 RQ5: SIMULATION ABILITY ACROSS PARTICIPANT GROUPS
Many applications require simulating responses from spe- cific demographic groups rather than general populations. Using SimBenchGrouped, we evaluate how LLM simulation ability changes when conditioned specific demographic attributes.
Table 2: Ungrouped vs. grouped sim- ulation performance (A.5). + de- notes 95% CI.
Category
AS
We measure this change AS = Syrouped Sungrouped- where Sungrouped is the SIMBENCH score for simulating the general population and S grouped is the score when sim- ulating a specific demographic group on the same question. negative AS indicates that the model's simulation ability decreases relative to the uniform baseline when it ís asked simulate specific demographic groups.
By Models
Claude-3.7-Soman
40.63
Claude-3.7- B.aude-37-Sommet--e 7-Sonnet-4400
4.6114..6
DeepSeek-RI DeepSeek- V3-0324
-0.79
-1.27
GPT-4.1
-3.9410.76
By Demographics Religiosity/Practinnn Political Addeoo//yooogy
-9.91
4..77
Importantly, for SimBenchGrouped. we specifically selected questions where human response distributions showed the highest variance across demographic groups (see $2.4). The observed degradation simulation performance therefore likely represents upper bound on the challenges LLMs face when simulating specific demographic groups. Our results in Table 2 show that LLMs struggle more with simulating specific demographic groups compared to general populations. All evaluated models show negative mean AS values, with degradation ranging from -1,27 for DeepSeek- V3-0324 to -4.61 for Claude-3 Claude-3.7-Somme 7--Sonnet-4000. All degradations statistically significant, with 95% confidence intervals excluding zero in every case.
Religion (Affiliation)
4.83
Income/Social Standing
-1.51
Comicile/Urbanice
-3.1710.82
Employment Status
-3.03+0.82
Education
-..5540.99
Marital Status Age Gender
-1880+0.87
-1 -1.50
1.24
The performance degradation varies substantially by demographic category. Models struggle most when simulating groups defined by religious attributes, with conditioning on "Religiosity/Practie" causing the largest decrease in simulation accuracy (AS -..91), followed by "Political Affili ation/Ideology" (AS -4.97) and "Religion (Affiliation)" (AS -4833). In contrast, models maintain relatively better performance when simulating groups defined by "Gender" (AS -1.24) and "Age" (AS 1.50).
While these findings may not fully generalize to cases where demographic differences are less pronounced, they highlight potential limitations in how current LLMs capture the nuanced response patterns of specific demographic groups We argue that such challenging benchmarks are crucial for identifying areas where improvements are most needed, partieularly for applications that aim to model the behaviors of specific subpopulations.
4.6 RQ6: SIMULATION ABILITY VS. GENERAL CAPABILITIES
Finally, we analyze the relationship between LLM simulation ability and more general model capabilities by correlating performance on SIMBENCI with popular LLM capability benchmarks. We collect performance data for eight models on five benchmarks representing distinct capabilities and calculate the Pearson correlation with their SIMBENCH scores (see Appendix ! for implementation details and scatter plots).
We find that SIMBENCH performance correlates most strongly with benchmarks requiring knowledge- intensive reasoning, such as MML.U-Pro = 0.94) and GPQA Diamond (r 0.86). The correlation is weaker for general helpfulness (Chatbot Arena ELO, == 0.71) and instruction following (IF-Eval, 0.79). Crucially, the correlation is substantially weaker for narrow, specialized skills like advanced mathematics (OTIS AIME, = 0.48). We posit that accurately simulating human behavior is complex capability rooted in broad, mnowledge-intense
reasoning, which aligns with the diverse social and behavioral topics in SIMBENCH. The weaker correlations with Chatbot Arena and OTIS AIME suggest that neither general conversational ability nor narrow mooblem-solving skills are sufficient proxies for strong simulation performance.

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Human Behavior Simulation with LLMs
LLMs as human behavior simulators have attracted
significant interdisciplinary attention. Researchers have evaluated their efficacy across political
science (Argyle et al., 2023 Bisbee et al. 2024 Dominguez-Olmedo et al. 2024), psychology (Aher
al. 2023 Manning et al. 2024 Hewitt et al.
2023 Hu & Collier 22024] Dong 2024] Binz et al.
2025), economics (Horton 2023
Aher et al. 2023), and computer science applications Parr et al.
ct al, 2024] Hu & Collier 2025). Evidence regarding
L.MS' simulation fidelity remains mixed, with
some studies reporting promising results (Argyle et al. 2023) while others identify critical limitations,
including homogenized group representations (Cheng et al.
2023 Wang et al., 2025), a tendency
toward hyper-rationality rather than human-like error (Liu et al. 2025a). and deterministic rather than
distributional predictions (Park et al. 2024b).
Existing work has predominantly focused on individual-level simulation with minimal demographic conditioning, typically evaluating only one or two models in narrowly defined contexts. SIMBENCH addresses these limitations by providing a comprehensive benchmark for group-level simulation simuuaaatoo across diverse domains with systematic demographic conditioning and standardized metrics. SIMBENCH's distributional evaluation framework, based on Total Variation Distance, captures how accurately models represent the full spectrum of human response variation. This is an approach advocated for by researchers both simulation (Anthis et al., 2025) and general LLM evaluation studies (Ying) et al. 2023). For broader context on this emerging field, we refer readers to recent comprehensive
surveys (Ma et al.
2024 Kozlowski & Evans, 2025 Olteanu et al., 2025 Anthis et al. 2025
Appendix M continues our discussion of related work.
6 CONCLUSION AND FUTURE WORK
For LLM simulations to become reliable tools for the social and behavioral sciences, their fidelity to real human behavior must be measurable. However, prior evaluations have been fragmented, hindering systematic progress. To address this, we introduce SIMBENCH, the first large-scale, standardized benchmark for group-level human behavior simulation. By unifying 20 diverse datasets, SIMBENCH provides the necessary infrastructure to robustly evaluate and compare LLM simulators.
Using SIMBENCH, we provide the first systematic analysis of simulation ability across 45 LLMs. We show that even the best LLMs have limited simulation ability, that performance scales log- linearly with model size, and that there exists a fundamental tradcoff between standard alignment and simulating diverse human opinions. We further show that models struggle more when simulating specific demographic groups, and that strong simulation ability correlates with deep, knowledge- intensive reasoning. While significant progress is needed, SIMBENCH makes this progress measurable, providing an open foundation to accelerate the development of more faithful LLM simulators.
We hope that future research can build this foundation by expanding beyond standardized formats to evaluate interactive and open-ended behavioral simulations. Finally, addressing the observed tradeoff between alignment and simulation fidelity requires developing distribution-preservve alignment techniques, alongside further investigation into the causal mechanisms linking other model capabilities to simulation performance.
ACKNOWLEDGMENTS
This work was partially conducted while TH, JB, and PR were at the Data and Marketing Insights research unit of the Bocconi Institute for Data Science and Analysis. TH was supported by the Gates Cambridge Trust (grant OPP1 144 from the Bill & Melinda Gates Foundation). JB acknowledges
DH was supported by the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation program (grant agreement No. 949944, INTEGR ANTER PR was supported by a MUR FARE 2020 initiative under grant agreement Prot. R20YSMBZSS (INDOMITA).
funding from the Swiss National Science Foundation (SNSF) through project 55000222353222
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
SIMBENCH'S primary purpose is to benchmark LLM ability to simulate human behavior. While advancements in LLM simulation capabilities can support helpful applications such as pre-testing policies, these do not come without risks of misrepresentation and dual use.
RESPONSIBLE USE AND ACKNOWLEDGMENT OF LIMITATIONS
First and foremost, due to the observed limited simulation ability of state-of-the-arl LLMs, we caution against relying LLM-powered simulations of human behavior for tasks where downstream harm is possible. Even as models improve, substituting algorithmic approximations for authentic human participation carries the risk of disadvantaging under-represented marginalized communities by removing their opportunities to directly shape decisions that affect them. Furthermore, while benchmarks like SIMBENCH help measure simulation capabilities, must be careful not mistake increasing benchmark performance for genuine understanding of complex human behavior.
DATA PROVENANCE AND TRANSFORMATIVE USE
The creation of SIMBENCH from 20 diverse sources was guided by a commitment to responsible data handling. Our curation process prioritized datasets with clear and permissive terms. As result, 17 out of the 20 datasets are governed by explicit permissive licenses (c.g.. Creative Commons, MIT). For the few remaining datasets that are publicly available for research without explicit license, apply a consistent framework built on the principle of transformative use.
1. Transformative Use. SIMBENCH does not contain redistribute any raw, individual-level participant data. It is a new, derivative work consisting of aggregated, non-reversible group-level distributions. This process protects the privacy of the original human subjects.
Multi-Level Licensing, Our public release includes a detailed LCENSE file. The SIMBENCH framework (our code and pipeline) is permissively licensed (e.g., CC-BY-NC-SA 4.0). For each of the 20 constituent datasets, the documentation explicitly lists the original source, its specific license or terms of use, and clear statement clarifying its status as an aggregated, derivative work whose original terms should still be consulted.
SCOPE OF REPRESENTATION AND INTERSECTIONAL ANALYSIS
While SIMBENCH includes diverse demographic groups. cannot adequately support simulations of intersectional identities due sample size limitations. By conditioning on one demographic variable at time, we cannot systematically assess how well models handle the rich overlap of identities (e.g., "older Latinx women," "young Black men"). This to maintain the statistical integrity of the ground-truth distributions, as small intersectional group sizes make it difficult combine multiple characteristics simultaneously due increasing sampling noise in response distributions. Yet intersectional simulation is precisely where societal biases and model limitations often emerge, making this an important direction for future work. Additionally, the conditional prompting approach we use conceptualizes simplistic human populations and may thus fail to appropriately account for nuances of individual behavior.
deliberate methodological choice
CONCLUSION
Nevertheless, believe SIMBENCH is important step toward making LLM simulation progress measurable and raising awareness state-of-the model blind spots. Together, we hope this will ultimately create accountability for models deployed in socially sensitive contexts.
REPRODUCIBILITY STATEMENT
SIMBENCH is released on HuggingFace, Our codebase is available GitHub We provide de- tailed descriptions of our experimental setup, including the exact prompts used for both base and instruction-tuned models, in Appendix D and an empirical validation of our elicitation methodology
in Appendix E
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Afrobarometer. Afrobarometer data, all countries (39), round 9, 2023.
http:/ /ttp://www www
afrobarometer org 2023. Accessed: March 2025.
Gati V Aher, Rosa Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 337-371. PMLR, 23-29 Jul 2023. URL https::// http:///ooeeeedng.s. mlr. press/ w202 2ttp02/ahheeee aheraaarraaa html
Georg Ahnert, Max Pellert, David Garcia, and Markus Strohmaier. Extracting affect aggregates from longitudinal social media data with temporal adapters for large language models. Proceedings of the International AAAI Conference on Web and Social Media, 19:15 36, June 2025. ISSN 2162-3442 doi: 10. 1111669//cw..11990 13501. URLhttp:/ /xx
ww666//1cssst v v19i1 http::///xxddoo.o////... www.
35801
Jacy Reese Anthis, Ryan Liu, Sean M Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, and Michael S. Bernstein. Position: LLM social simulations promising research method. Forty-second International Conference on Machine Learning Position Paper Track, 2025.
Anthropic. Claude 3.7 sonnet and claude code, February 2025. hwws:claade-33---eeee
Accessed: 2025-05-14.
Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, doi: 10. 20.11077ppa.2023.1
31(3):337-35,, 2023.
Lora Aroyo, Alex S. Taylor, Mark Díaz, Christopher M. Homan, Alicia Parrish, Greg Serapio-García, Vinodkumar Prabhakaran, and Ding Wang. Dices dataset: diversity in conversational ai evaluation for safety. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc.
Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean- François Bonnefon, and lyad Rahwan. The moral machine experiment. Nature, 563(7729):59-644 2018.
Edmond Awad, Sohan Dsouza, Azim Shariff, Tyad Rahwan, and Jean- Gean-François Bonnefon. Universals and variations in moral decisions made in 42 countries by 70,000 participants. Proceedings of the National Academy of Sciences, 117(5):2332-2333 2020.
Eric Bigelow and Steven Piantadosi. large dataset of generalization patterns in the number game. Journal of Open Psychology Data, 4(1))e4-e4, 2016.
Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K Eckstein, Noémi Éltetō, et al. A foundation model to predict and capture human cognition. Nature, pp. 1-8, 2025.
James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, 32 (4):401-416, 2024. doi: 101 22.0017/an.2024.55
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020
Myra Cheng, Tiziano Piccardi, and Diyi Yang. CoMPosT:
Characterizing and evaluating caricature in LLM simulations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 10853-10875. Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v 1/t02/////222///// 120.:333/1112222mmmmmmmm. mamnlp-main... 669. URL https:ttpt..mm. //acl anthologye www.lllo...oogoo//o 20033 emn am- 6697
12

---

Published as a conference paper at ICLR 2026
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang. Michael I. Jordan, Joseph E. Gonzalez. and lon Stoica. Chatbot arena: an open platform for evaluating Ilms by human preference. In Proceedings of the 41st International Conference on Machine Learning, ICML'224. JMIR.org 2024.
André F. Cruz, Moritz Hardt, and Celestine Mendler-Dinner. Evaluating language models as risk scores. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang Advances in Neural Information Processing Systems, volume 37, pp. 97378-97407 Curran Associates, Inc., 2024. URL https: http://ss//pooceeeggg...eeeeuuee cappapr files/
Bnd__enchmarrs__reen
DeepSeek-AL. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19437,,
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int80: 8-bit matrix multiplication for transformers at scale. Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387108
Ricardo Dominguez-Olmede Moritz Hardt, and Celestine Mendler-Dünner. Questioning the survey responses of large language models. In A. Globerson, L. Mackey, D. Bel- grave, Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems, volume 37, pp. 45850-45878. Curran Associates, Inc., 2024. URL httttp:://ppooeeeeettc http::/pwweeceonn....eeeee..eeap http://ppperr.lileeee//eee//222 File
pdf
Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can LLM be personalized judge? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for
Computational Linguisties: EMNLP 2024, pp. 10126-1000 10141, Miami, Florida, USA, November http:////2/....iinnnnnmmml////// findings-emmll.592. htttpss:/cclaatthoooooooooo org/ 2024 findings emnlp. 592/
2024, Association for Computational Linguistics. doi: 10. 10.18653/1120244
Yijiang River Dong, Tiancheng Hu, Zheng Hui, and Nigel Collier. Steer model beyond assistant: Controlling system prompt strength via contrastive decoding. arXiv preprint arXiv:2601.06400 2026.
Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Sitfield-Dodds. Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCan- dlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models. In First Conference on Language Modeling, 2024. htt.hhtps://// /ttttp::///penreeieee net. forum?id
2 b116i
Adam Enders, Casey Klofstad, Amanda Diekman, Hugo Drochon, Joel Rogers de Waal, Shane Littrell, Kamal Premaratne, Daniel Verdear, Stefan Wuchty, and Joseph Uscinski. The sociodemographic correlates of conspiracism. Scientific reports, 14(1):14184, 2024.
European Social Survey European Research Infrastructure (ESS ERIC). ESS1 Integrated File, Edition 2.0 [Data set], 2024. URI
1e02_0
Xiachong Feng, Longxu Dou, and Lingpeng Kong. Reasoning does not necessarily improve role- playing ability. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguisties: ACL 2025, pp. 10301- 10314, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176. 256-5. doi: 10. Firdings-acl.35 5377
Linauttaioonlllnnuuutsiiisi URL /tt.:tssss//////atttoooooooogggg
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. httpp:: httpaa///hqqqinge htwhhhgggggggceeee ctleceeccoooppeeeeee//eeeeeeeeee ctwppaces/cpep/e/ memeenllmmt leaderboard/
open 1 mm Leaderboaaaa 2024.
13

---

Published as a conference paper at ICLR 2026
Aryo Pradipta Gema, Alexander Hägele, Runjin Chen, Andy Arditi, Jacob Goldman- Wetzler, Kit Fraser- Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, and Ethan Perez. Inverse scaling in test-time compute. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. htt.tttts:///pneee
ew. net/
d= c77M Featured Certification, J2C Certification.
Ken Goldberg, Theresa Roeder, Dhruv Gupta, and Chris Perkins. Eigentaste: A constant time collaborative filtering algorithm. information retrieval, 1133--11,,,001
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Aditya K, Alex Chohlas-Wood Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay. Jonathan Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag. Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun lyer, and Zehua Li.. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neu- ral Information Processing Systems 36 (NeurlPS 2023), Datasets and Benchmarks Track, 2023. URL lttts::/ wewpiiinnnnnnuuuuureee neurips wwtpaper_piiiaeeppcppe www.sfiappeeeeeeeeeeeeeee pttee//ppprre22111// hash/ 83e 3eeeeeeeeeeecccceeeeeeeeeeeeeeea
Abstract Datasets Abstraatt-aaasetts and
Benchmarks. html
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao. Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding. Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen. Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou. Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang. Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin L., Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and 7hen Zhang. Deepseek-rl incentivizes reasoning in llms through reinforcement learning. Nature, 645(8088))633--633 2025. ISSN 1476-4687. doi: 10. 110011//41156-055--000400 025 09422-z
URL
hittps://doi.oo attps://doi.org/10 0tt///1001118//00
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL htttps://openee ew. net fortt//rumm??
d7KB 3GmQ.
Luke Hewitt Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer. Predicting re- sults of social science experiments using large language models, August 2024.
URL hrg.r
20sciences
pdd
14

---

Published as a conference paper at ICLR 2026
John J Horton. Large language models simulated economic agents: What can learn from homo silicus? Technical report, National Bureau of Economic Research, 2023.
Tiancheng Hu and Nigel Collier. Quantifying the persona effect in LLM simulations. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10289-10307, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 18 2024.act-long.554 5544. URL BtLhttts:///a ant logy wttp:rrr/02222a.111oo......55
Tiancheng Hu and Nigel Collier. iNews: A multimodal dataset for modeling personalized affective responses to news. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 25000-25040, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 977-8-89176-2511111 doi: 10. 10.188653/1//2225..oo aal---llllnngg long.llonn..n 1217.
URL UtRLhttps:/// lttt:ss://aclanthoolooooo org 2025 acl wwwaaalllloo.ooo..ne 1on.lloon...111 1217/]
Tiancheng Hu, Benjamin Minixhofer, and Nigel Collier. Navigating the alignment-calibration
trade-off: A pareto-superior frontier via model merging. arXiv preprint arXiv:2510.174: 17426, 2025.
Tiancheng Hu, Caiqi Zhang, Dirk Hovy, and Nigel Collier. Position: Large language model failures from hallucination to homogenization are different facets of miscalibration. Authorea Preprints, 2026.
ISSP Research Group. International social survey programme: Social networks and social resources issp 2017. GESIS Data Archive, Cologne. ZA6980 Data file Version 2.0.0, https///doi.org/1.42222/1/
2019.
ISSP Research Group. International social survey programme: Religion iv - issp 2018. GESIS Data Archive, Cologne. ZA7570 Data file Version 2. 2.0.0 http://d6ooorr/11.2232//..1..
2020.
ISSP Research Group. International social survey programme: Social inequality issp 2019. GESIS, Cologne. ZA7600 Data file Version 3.0.0, https://doi.org wrtp//..////////// 14009, 2022.
ISSP Research Group. International social survey programme: Environment iv issp 2020. GESIS, Cologne. ZA7650 Data file Version 2.0.0, http:///bb.org11142222///1
2023.
ISSP Research Group. Za8000 international social survey programme: Health and health care ii issp 2021. GESIS, Cologne. ZA8000 Data file Version 2.0.0, http:://bbi.rrg//442222//// 22804422222557700000000000 2024.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language
models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 2021. doi: 10.1162/tacl_ a a_00407. URL
https: https://aclaatthooe
Adam Tauman Kalai and Santosh S. Vempala. Calibrated language models must hallucinate. In Pro-
ceedings of the 56th Annual ACM Symposium on Theory of Computing. STOC 2024, pp. 160-171, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703836. doi:
01144/3618260.364999
URL /ttpt:tt///ddtt http///iooor/g/.o/oo/oo/oo/ooo wrg/10../0..0. 1411 3611//36660 826.1.667.7.7777
Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. Calibration-tuningg Teaching large language models know what they don't 't know. Raúl Vázquez, Hande Celikkanat, Dennis Ulmer, Jörg Tiedemann, Swabha Swayamdipta, Wilker Aziz, Barbara Plank, Joris Baan, and the 1st Workshop on Uncertainty- NLP (UncertaiNLP 2024), pp- 1-14, St Julians, Malta, March 2024. Association for Computational Linguistics. URL https: a/c wathhhnnnhooooooo logy or org/ www.ertaiimpll.......
de Marneffe (eds.), Proceedings of
Tomasz Korbak, Ethan Perez, and Christopher Buckley, RL with KL penalties is better viewed as Bayesian inference. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083-1091. Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/ https:: http://aaaaantol...ooooo//////// htt.:/222.....nniin.eeiiiiiiia.e findings- emm
2022. mandings-emnlp.77 h 777
15

---

Published as a conference paper at ICLR 2026
Austin C. Kozlowski and James Evans. Simulating subjects: The promise and peril of artificial intelligence stand-ins for social agents and interactions. Sociological Methods & Research, 54(3): 1017-1073, 2025. doi:
00491241251337322
Sai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, and Ziyu Yao. Can LLMs simulate personas with reversed performance? systematic investigation for counterfactual instruction following in math reasoning context. In First Workshop on Social Simulation with L.MMs, 2025.
URL htt.tttppss//peeeveeeteeee http:///eaaaaaeeeeeeeeeeoooaaaaa
net/ (oroformmmm/2d332299793
Latinobarómetro. Latinobarómetro 2023. http://wwww.atinnnbbrretttteeeee cessed: March 2025.
2023. Ac-
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young. and Phil Blunsom. Mind the gap: Assessing temporal generalization in neural language models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Advances in Neural Information Processing Systems, volume 34, pp. -9348-29363. Vaughan Advances
Curran Associates, Inc., 2021. URL ffbffff5ffff/ http://////.//oooo.ooooooooo./6/
fi
pdf
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:/805.0 /005. 2018
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Mcasuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18663/1122022aalll 2297
URL htttps::/aclaattlllgggoooooo///e http://attttooloooorg/2/22ee wwwg///g2200. acl- long
Andy Liu, Mona Diab, and Daniel Fried. Evaluating large language model biases in persona- steered generation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 9832-9850, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 100.18653//1220 1/2024. findings-acl.586. URL wttp://22...iiiinn.anao... htt.://.2....iinnna.....5..
Ryan Liu, Jiayi Geng, Joshua Peterson, Ilia Sucholutsky, and Thomas L. Griffiths. Large language models assume people are more rational than we really are. In The Thirteenth International Conference on Learning Representations, 2025a. URL /ttttp:://openreviee net forum?
Id=cAeeTBçxxq]
Ryan Liu, Jiayi Geng, Addison J. Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L. Griffiths. Mind your step (by step): Chain-of-though can reduce performance tasks where thinking makes humans worse. In Forty- International Conference on Machine Learning, 2025b. URL
https:///ppenrevie net forum wtt.://oooommmdd-ooooooooo (irrrmmm1dd-J337zbbyxxx
Bolei Ma, Xinpeng Wang, Tiancheng Hu, Inna-Carolina Haensch, Michael A. Hedderich, Bar- bara Plank, and Frauke Kreuter. The potential and challenges of evaluating attitudes, opin- ions, and values in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8783-8805, Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10.18653/v 1/2024. 1/2024.innin.ss--eeeeeeo Findings-ent w.w.eaammpp.11.1 513/
URL
Benjamin S Manning, Kehang Zhu, and John J Horton, Automated social science: Language models as scientist and subjects. Technical report, National Bureau of Economic Research. 2024.
Niels Mede, Viktoria Cologna, Sebastian Berger, John Besley, Cameron Brick, Marina Joubert, Edward W Maibach, Sabina Mihelj. Naomi Oreskes, Mike S Schäfer, al. Perceptions of science, science communication, and climate change attitudes in 68 countries-th the tisp dataset. Scientific data, 12(1): 114, 2025.
16

---

Published as a conference paper at ICLR 2026
Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto. Benchmarking distributional alignment of large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pp. 24-9,, Albuquerque, New Mexico, April 2025. Association for Computational Linguisties. ISBN 979-8-89176- 189-6.
URL
.rag:..rgg/ee htap//2222.aaaaa.. naacl 1ong.27
Meta Al. Introducing Llama 3.1: Our most capable models to date, 2024. URL https://ai m lawp:///mmmeetalaaaaaaaaaaaaaa-a
Accessed: 2025-05-14.
Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), P) pp. 9131-9,43, Online, November 2020. Association for Computational URL httpps://aclantthhooo...//
Linguistics. doi: 10. 11188//////2220mmmmpppaa 2020.eemnlp-aall 7347
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint 1rX1v:2303.13375 2023.
Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, and Myra Cheng. Ai automatons: Ai systems intended imitate humans. arXiv preprint arXiv:2503.022550, 2025.
OpenAL. Introducing GPT-4.1 in the API, 2025a. gpt 4-1/ Accessed: 2025-05-14.
URL
OpenAl. Introducing openai o3 and 04-mini, April 2025b. URL and- 04 mini/ Accessed: 2025-05-14.
com/
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the
36th Annual ACM Symposium on User Interface Software and Technology. UIST '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701322 320. doi: 00.1145/3556183336666 33.3666663. URL
000.0....455550 30555/5868833 360
Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10119, 2024a.
Peter S. Park, Philipp Schoenegger, and Chongyang Zhu. Diminished diversity standard large language model. Behavior Research Methods, (6.5775-5770, 2024b. doi: 10. 3758/s 3428 023-02307 URL httth::ss http:s///..aa...prrggg.....////d wttp://erw..oo/taceeeeeeeeeeee/e
s13428
Joshua C Peterson, David Bourgin, Mayank Agrawal, Daniel Reichman, and Thomas Griffiths. Using large-scale experiments and machine learning to discover theories of human decision-making. Science, 1209 1214, 2021.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pp. 29971-30004, PMLR, 2023.
Skipper Seabold and Josef Perktold. Statsmodels: Econometric and statistical modeling with python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 92 92-96, 2010. doi: 10. 10.25080/Majorz. weterr--bffff20000
Hua Shen, Nicholas Clark, and Tanu Mitra. Mind the value-action gap: Do LLMs act in alignment with their values? In Christos Christodoulopoulls, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 1099--118,, 18, Suzhou, China, November 2025. Association for Computational doi: 10/665//v//222..ooo/ main.154 154. URL https 7/ ho ogy www.ooo.ooooaoooooooo emn 1p-maaiie n. 154/
Linguistics. ISBN 8-99-88-9976-332-4
17

---

Published as a conference paper at ICLR 2026
Camelia Simoiu, Chiraag Sumanth, Alok Mysore, and Sharad Goel. Studying the "wisdom of crowds" at scale. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pp. 171-179, 2019.
Zayne Rea Sprague, Fangcong Yin. Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of- thought helps mainly on math and symbolic reasoning. In The Thirteenth International Confer- ence on Learning Representations, 2025. |RLhttts: https: /tpp//ppaeeiiieeeeeeeeeeeeeeeeee / W661cc88Knn
net. www.nnee/ffourree forum?id
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rívière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433 5442, Singapore, December 2023, Association for Computational Linguisties. doi: 10. 118866/1//222222 1/05:///102....mmmmmmmmmma 2023. emn 1 p ma i 3307
ac URL httpt: /atl//aaaaanin www.nnhologgyy....oooooooooooole org/
Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. Do Ilms exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguisticss, 12:1011 09 2024. ISSN 2307-387X. doi: 10.1 162/tacl_a 00685. URL https: http://o....1111111////////
00685
Vals AI, Inc. Mmlu pro benchmark. https://wwe /www vals. a wtt///bncmmrkk/////e
100-004-5-2000 April 2025. Last updated 15 April 2025.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 1118866/11118844/// URL https::/acclantholoo orgpt/hhoooooooooooooooooo
org/ W18 54467
Angelina Wang, Jamie Morgenstern, and John Dickerson. Large language models that replace hu- man participants can harmfully misportray and flatten identity groups Nature Machine Intelligence, pp. 1-12, 2025.
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. "my answer is C": First-token probabilities do not match text answers in instruction-tuned language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024. pp. 7407-7416, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/1//024. andings-acl.441 BtRtstttss///acaaaee
erg//224 findiin ngs-ncllll..... .449
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou, Chain-of th-tuuggtt prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,, pp. 38 48-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.18653//120200c.mmp...
URL https: /cc/ antho ogy. 2020. emn 1 67
18

---

Published as a conference paper at ICLR 2026
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Zihao Yi, Qingxuan Jiang, Ruotian Ma. Xingyu Chen, Qu Yang, Mengru Wang, Fanghua Ye, Ying Shen, Zhaopeng Tu, Xiaolong Li, and Linus. Too good to be bad: On the failure of Ilms to role-play villains, 2025. URL htt.t:tts/////. /ttepps://rxi..../// wrwiv..xivvv///iio// org/ wwwp///.g..aeeeeeee.eeeeeeeeeeee
04962
Lance Ying, Katherine Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L Griffiths, and Joshua B Tenenbaum. On benchmarking human-like intelligence in machines. arXiv preprint arXiv:2502. 2025.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 18-24 Jul 2021. URL
mlr. www.pppwsssvvv/.e www.zla. html
Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. On the calibration of large language models and alignment. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Compulational Linguistics: EMNLP 2023, pp. 9778-9795, Sin- gapore, December 2023, Association for Computational Linguistics. doi: 10.18653/v1/2033. finding--emnl.654. URL https: https::/acct http:e////olaanttoooooooorr//// ftnd://......................... findeeiininiinnnnnnnnn
654/
19

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
The Scope of Benchmarking Human Behavior The challenge of benchmarking human behavior lies in its vastness. No single study, nor dozens, could ever capture its full complexity. We operational- ize this challenge by focusing on a core set of fundamental cogmitive and social tasks widely studied in the behavioral sciences: decision-making, self-assessment, judgment, and problem-solving. This focus necessarily means that SIMBENCH does not capture the full complexity of human interaction, such embodied or multi-turn social dynamics. As the first large-scale benchmark for group-level simulation, SIMBENCII provides the essential infrastructure to establish robust baselines, uncover fundamental properties like scaling trends, and map the frontier for future work on more complex, interactive simulations.
Scope of Representativeness Although SIMBENCH spans 20 diverse datasets, the combined sample does (and can) not fully represent any single population in its full complexity. Many geographic regions are still underrepresented or entirely absent, potentially limiting generalizability to populations with different cultural backgrounds and preferences. Even within countries, demographic representativeness may vary, only a subset of our 20 datasets are based on nationally representative sampling techniques. Each dataset carries its own statistical uncertainty. Opt-in samples and crowdsourced data (e.g., from Amazon Mechanical Turk) may have larger margins of error than nationally representative surveys, potentially affecting the benchmark's 's precision for certain questions. We view these limitations as opportunities for collaborative extension of SIMBENCH improve global coverage and representativeness over time.
Temporal Dimensions The current version of SIMBENCII utilizes static datasets that capture human behavior at specific points in time. This approach allows for systematic evaluation across domains but cannot yet assess how well LLMs simulate evolving preferences, opinion shifts, or behavioral adaptation all fundamental aspects human behavior. Future iterations of SIMBENCH could incorporate longitudinal data to address these dynamic aspects of human behavior and expand the henchmark's s evaluative capacity.
Task Format Considerations SIMBENCII currently focuses on multiple-choice, single-answer, single-turn questions and interactions. This standardized format enables systematic comparison across diverse domains but necessarily excludes more complex behavioral simulations including multi-step decision processes and interactive social dynamics. We see this as a pragmatic starting point that establishes foundational evaluation capabilities while inviting future extensions to capture more nuanced aspects of human behavior.
Training Data Overlap A fundamental challenge for all LLM evaluation is the potential for training data contamination. While this cannot be definitively ruled out without full access to model training corpora, several aspects of our methodology and empirical findings suggest this risk is substantially mitigated for SIMBENCH:
1) Much of our source data not easily ingested by standard web scrapes. Virtually all of our source data are primarily distributed as structured data files in specialized formats like R or SAS in academic archives. This makes data contamination much less likely than for generic web text, as these files cannot be meaningfully read or interpreted as plain text by standard scraping tools.
2) SimBench's 's core task is not to recall fact but to predict a response distribution for a specific demographic subgroup (e.g., "women in Slovakia"). Even if model's training data included thousands of individual survey responses, it would still need to learn, without supervision, how to aggregate these individual points into coherent distribution for an arbitrary, specified subgroup. This is sophisticated, zero-shot social reasoning skill that is unlikely to emerge from simply seeing the raw data.
3) On datasets that are most likely to appear in training data (e.g., US-centric OpinionQA), even the best- tess-performin models achieve an S-score of only 60, far from the 100-point maximum. If models had memorized this benchmark, we would expect scores far closer to perfect. This clear performance ceiling demonstrates that our benchmark is testing genuine capability rather than memorization.
20

---

Published as a conference paper at ICLR 2026
4) The consistent scaling patterns observe across diverse datasets suggest genuine simulation capabilities rather than artifacts of training data overlap.
Nevertheless, acknowledge that data contamination remains fundamental challenge in LLM ovaluation, and future work should develop more robust methods to detect and quantify its impact. We include this consideration for completeness while believing it unlikely significantly impact current findings.
B DATASET CURATION AND EXCLUSION RATIONALE
As described 2. our dataset curation process involved a systematic review numerous prominent datasets in the social and behavioral sciences. While our search was extensive, many promising candidates were ultimately excluded for failing to meet our strict inclusion criteria for redistributable, multiple-choice benchmark.
Table provides an illustrative list well-known datasets that were considered during this review but not included in the final SIMBENCH collection. This list is not exhaustive but serves to highlight the common methodological and logistical challenges that arise when creating a benchmark from existing scientific data, such restrictive licensing, complex experimental designs, and non-standard response formats. By documenting these exclusion rationales, we aim to provide transparency into curation process and offer resource for future benchmark creators.
Table 3: Examples of Datasets Considered and Excluded from SIMBENCH
Dataset
Exclusion Reason
Understanding America Study
Licensing Restrictions
ManyLabs & ManyLahs
Complex treatment condition Licensing Restrictions Licensing Restrictions
General Social Survey
Demographic and Health Surveys
Global Preferences Survey
Licensing Restrictions No raw question wording available
World Risk Poll
Yourmorals.org
Licensing Restrictions Licensing Restrictions
Asian Barometer
The Glasgow Norms
Licensing Restrictions Licensing Restrictions
MovicLens
Health Information National Trends Survey
Licensing Restrictions
BBC Big Personality Test
Licensing Restrictions
Time-sharing Experiments for the Social Sciences
Often complex treatment conditions
Project Implicit
Hard to model reaction time in L.LMs Licensing Restrictions
Children's Worlds Survey
Monitoring the Future Survey
Licensing Restrictions Individual test items not accessible Free text response
TIMSS
UMDDOurDataHelly
MobLab datasct
few questions; lack detailed demographics data
C
SIMBENCHPOP AND SIMBENCHGROUPED SAMPLING DETAILS
We curated data at two levels of grouping granularity, corresponding to our two main benchmark splits: SimBenchPop and SimBenchGrouped.
SimBenchPop measures LLMs' ability to simulate responses of broad, diverse human populations. We include all questions from all 20 datasets in SimBench, combining each question with its dataset- specific default grouping prompt (c.g., "You are an Amazon Mechanical Turk worker based in the United States"). We sample up to 500 questions per dataset to ensure representativeness while
21

---

Published as a conference paper at ICLR 2026
keeping the benchmark manageable. For each test case, aggregate individual responses across all participants the dataset to create population-leve response distributions. This approach creates a benchmark that represents population-Ieve responses across diverse domains while maintaining reasonable size of 7,167 test cases.
focus only on five large-scale survey datasets with rich demographic information and sufficient sample sizes: OpinionQA, ESS, Afrobarometer, ISSP, and Latinobarómetro. Our sampling approach prioritizes questions showing meaningful demographic variation. For each dataset, we identify available grouping variables (e.g., age, gender, country) with sufficient group sizes form meaningful response distributions. We calculate the variance of responses across demographic groups for each question and rank questions by their variance scores, prioritizing those showing the strongest demographic differences. We select questions that exhibit significant variation across demographic groups to ensure the benchmark captures meaningful differences in responses. For each selected question, we create multiple test cases by pairing it with different values of the grouping variables (c.g., age "18-29", age "30-49"). This process results in 6,343 test cases that specifically measure LLMs' ability to simulate responses from narrower participant groups based on specified demographic characteristics. Table 4 summary of the sampling process across all datasets, showing the minimum group size thresholds and the number of test cases in each benchmark split.
For SimBenchGrouped.
Table 4: Dataset Sampling Summary; NaN refers to dataset that is only available in aggregated form and no grouping size is known.
Dataset
Min. Group
SimBench
SimBenchPop
SimBenchGrouped
WisdomOfCrowds
100
1,604
114
Jester
100
136
136
Choices 13k
NaN
14,568
500
OpinionQA
300
1,074,392
500
984
MoralMMachineClasiie
100
3,441
15
MoralMachine
100
20.771
500
ChaosNLI
100
4,645
500
ESS
300
7,783,780
500
1,643
Afrobarometer
300
517,
500
1,531
OSPsychBig5
300
1,950
250
OSPsychMACH
300
3,682,700 700
100
OSPsychMGKT
300
20,610
500
OSPsychRWAS
300
975,585
22
ISSP
300
594,336
500
940
Latinobarómetro
300
80,684
500
1,245
GlobalOpinionQA
NaN
46,329
500
DICES
10
918,064
500
NumberGame
10
15,984
500
ConspiracyCorr
300
968
45
TISP
300
172,271
485
Total
10,930,271
7,167
6,343
D IMPLEMENTATION DETAILS
For base models, use HuggingFace Transformers (Wolf ct al. 2020) to inference single NVIDIA RTX A6000 Ada GPU, We structure prompts so that the next token corresponds to the model's answer choices. For models smaller than 70B parameters, implemented in bitsandbytes (Dettmers et al. 2022), while 70B models use 4-bit quantization.
8-bit quantization
For instruction-tuuae models, API, while other models are accessed via OpenRouter. We request verbalized probability outputs in ISON format with temperature initially 0. If parsing fails, we increase temperature to I and
API calls. OpenAl models accessed directly through their
22

---

Published as a conference paper at ICLR 2026
retry up to 5 times. All models successfully produced valid JSON under these conditions. When probability outputs do not sum to I, we apply normalization.
Our evaluation includes diverse set of models: Qwen 2.5 Yang et al. 2024 (0.5B-72B), Gemma (Team ct al. 2025) PT and IT (4B-27B), 04-mini (OpenAl 2025b), Claude 3.7 Sonnet (Anthropic
2025), DeepSeek RI (Guo al. 2025). DeepSeek- DeepSeek-V3-0322 V3-0324 DeepSeek-A1 (2024), GPT-4.1 ApenAAl
(2025a), and Llama-3. Llama-3.I-Instruue InIstrucct (8B (8B-4055 405B) (Meta A1, 2024).
To ensure the validity of our results, we perform two checks: 1) We verify that base models assign the vast majority of probability mass to the provided answer options. Even for small models like Qwen2.5-0.5B, the of probabilities across answer tokens is high as 0.98, confirming that models rarely predict tokens outside the designated answer space. 2) We also evaluate the effect of quantization on model performance using subset of SimBench. As shown in Table [S] performance remains consistent across quantization levels, with minimal variation in total variation scores even for quantization-sense
models like Llama-3.1.
We detail below the prompts used in our experimental conditions for token probability and verbalized distribution prediction.
The following system prompt was consistent across all experimental conditions:
You are group of individuals with these shared characteristics:
[default system prompt] grouping system prompt (if any)
For token probability prediction, we adapted the prompt structure from Nori et al. (2033)
** **Quesion***; {question] }
Do not provide any explanation, only answer with one of the following options: (answer options). Answer**;
Prompt for eliciting verbalized probability prediction:
* *Question**** **; {question] }
Estimate what percentage of your group would choose each option. Follow these rules:
1. Use whole numbers from 100
2. Ensure the percentages sum to exactly 100
3. Only include the numbers (no % symbols)
4. Use this exact valid JSON format: (answer options) and do NOT include anything else.
5. Only output your final answer and nothing else. No explanations intermediate steps needed.
Replace X with your estimated percentages for each option.
** **
Prompt for zero-shot CoT:
* ** {question]
Estimate what percentage your group would choose each option.
Think step by step about how people with your shared characteristies would reason about this question.
Consider different perspectives within your group and what factors would influence their choices.
Please provide your reasoning first, then give your final answer in ISON format. Follow these rules for your final answer:
1. Use whole numbers from to 100
2. Ensure the percentages
to exactly 100
3. Only include the numbers (no % symbols)
4. Use this exact valid JSON format: iisso_formmtt_te
5. Replace with your estimated percentages for each option. Answer
23

---

Published as a conference paper at ICLR 2026
E VALIDATION OF ELICITATION METHOD
key methodological choice in SIMBENCH is how to elicit probability distributions from LLMs. For base models, we direct token probabilities from the first token of the response. For instruction- tuned models, however, two primary methods exist: direct token probabilities and requesting a "verbalized" distribution (e.g., JSON object with percentages). To validate our choice of using verbalized distributions for instruction-tuned models, as recommended by recent work (TTan et al.
2023
Meister al. 2025), conducted direct comparison.
Figure 5 compares the SIMBENCH scores for several instruction-tune models using both methods. The results unequivocal: using verbalized distributions (teal dots) dramatically and consistently outperforms direct token probabilities (orange dots) for every instruction-tuned model tested. In many cases, using token probabilities results in scores far below zero, indicating that the model's raw logits are poorly calibrated for this task after instruction tuning. In contrast, base models (black bars) perform reasonably well with token probabilities, as they are not subject to the same post-training shifts.
This analysis provides strong empirical support for our methodological decision use token proba- bilities for base models and verbalized distributions for instruction-tuned models, ensuring that are evaluating each model class using the most effective and well-calibrated elicitation technique.
20
SimDench Score
20
40
60
Base Token Prob
Iustruct Token Frob
Iustruct- Ver balbalddd
413
12B
27B
8B
70B
7B
13B
32B
3B
7B
14B
32B
72B
Gemma-3
Llama-3.1
OLMo-2
Qwen2.5
Figure 5: Verbalized distributions are superior for instruction-tuned models. This plot compares SIMBENCH scores for base models (using token probabilities) against their instruction ceatucttion-uued tuned counter- parts using either token probabilities (orange) or verbalized distributions (teal). The large vertical gap for all instruction-tuned models demonstrates the significant performance gain from using verbalized distributions, validating our choice elicitation method.
Table 5: Total Variation for different models various guantization levels. Lower values indicate better performance.
Model
4-bit
8-bit
16-bit
32-bit
Llama-3. 1-8B-Instruct
0.272
0.266
0.262
0.262
Qwen2.5-7B
0.307
0.307
0.306
0.307
24

---

Published as a conference paper at ICLR 2026
F METRIC ROBUSTNESS CHECK
TVD ranges from 0 (perfect match) to 1 (complete disagreement), with lower values indicating better simulation fidelity. TVD provides an interpretable measure of how closely model predictions align with actual human response distributions. TVD is particularly well-suited for simulation evaluation compared to alternatives like KI. divergence or Lensen-Shannon divergence (JSD). Unlike KL divergence, TVD remains well-defined even when the model assigns zero probability to responses that humans give, avoiding the infinite penalties that KL would impose in such cases. Additionally, TVD is symmetric and bounded, making it more interpretable across different datasets and response distributions than KL divergence. While JSD offers similar advantages in terms symmetry and boundedness, TVD provides a more direct and intuitive interpretation of the maximum possible error in probability estimates. This property is especially valuable when evaluating how accurately models simulate the distribution of human responses rather than just matching the most likely response. For further discussion on TVD as an evaluation metric, see also Meister et al. (025). We show the results of Table D in terms of raw TVD values in Table [0]
To ensure our findings are robust across different metrics, we complement TVD with two alternative metrics: Censen-Shannon Divergence (JSD) and Spearman's 's Rank Correlation (RC). Table presents these metrics for a subset of evaluated models. The strong Pearson correlation between TVD and JSD (r 0.92) indicates these metries provide consistent model rankings. The moderate negative correlation (r = -0.5 00.77) between TVD and RC is expected, as lower distances correspond to higher correlations. This multi-metric evaluation confirms that our model comparisons remain consistent across different statistical measures.
We chose the uniform distribution as the primary baseline U because represents state of maximum uncertainty, or the "zero-knowledse guess. This provides the most conservative and universally applicable baseline across questions with varying numbers of choices. While other baselines, such as majority baseline, could be considered, they would incorporate some knowledge about the distribution, making the S=0 point less interpretable as a true "no-skill" score.
Table 6: Comparison of models three metrics: Total Variation Distance (TVD), Jensen-Shannon Divergence (JSD), and Spearman Rank Correlation (RC). Lower values are better for TVD and JSD; higher better for RC.
Model
Total Variation
JS Divergence
Rank Correlation
Claude-3.7-Sone
0.191
0.057
0.673
0.195
0,060
0.648
DeepSeek-RI
0.211
0.069
0.623
DeepSeek-V3-0322
0.216
0,069
0,620
GPT-4.1
0,209
0.070
0,646
0.231
0.085
0.593
04-mini-high
0.225
0.079
0.621
o4-mini-low
0.230
0.082
0.609
G FULL SIMBENCH RESULTS (RQ1)
We show the SIMBENCH scores for all 45 models we evaluate in Table 7 We show the scaling law plots for all models in Figure 6
H STATISTICAL ANALYSIS OF MODEL PERFORMANCE
To ensure the robustness of our findings, conducted comprehensive statistical analysis of model performance on SIMBENCH. This includes confidence intervals for overall scores, pairwise significance tests between models, and within-family analyses to validate scaling law observations.
25

---

Published as a conference paper at ICLR 2026
Table 7: Overall simulation ability as measured by SIMBENCII score S averaged across the two main splits of SIMBENCH. Reasoning models are highlighted in italics. Models are sorted by score. Models below the dotted line perform worse than uniform bascline.
Model
Type
Release
S (t)
Claude-3.1 3.aee-..7-Sneee
Instr.
Closed
40.80
3.7oul--37-om
Instr.
Closed
39.46
GPT-4.1
Instr.
Closed
34.55
DeepSeek-RI
Instr.
Open
34.52
DeepSeek- DeepSeek-V3-0324
V3-0324
Instr.
Open
32.89
04-mini-high
Instr.
Closed
28.99
Lama-3.1-40%3 405B-Instruct
Instr.
Open
28.40
04 04-mini-low
Instr.
Closed
27.77
Wwen2.5-72B-Intteee
Instr.
Open
27.61
Owen2.5-32B-Inae Instruct
Instr.
Open
23.76
Qwen2.5- 5-144--Insttuee
Instr.
Open
20.75
OLMo OLMo-2-0325- 2-0325-3255
Instr.
Open
19.80
1-mma-33112B-11
Instr.
Open
18.62
Gemma-3-27B-IT
Instr:
Open
18.33
OLMo OLMo-2-0325- --0325-33B-Insttee
Instr.
Open
18.32
Llama-3, 1.taa-3.1-700
Instr.
Open
16.56
OLMo-2- 2-0325-322
Base
Open
15.90
Llama-3. 3.1-Minitaur-888
Base
Open
14.50
OLM4-2-1124-13B
Base
Open
13.83
Qwen2.5-72B
Base
Open
13.34
Qwen2.5-32B
Base
Open
12.27
Qwen2.5-14B
Base
Open
11.92
DL.Mo-2-0325-32-SST
Instr.
Open
11.28
OLMo-2211124-13B-nnnii
Instr.
Open
9.59
01.Mo-211124-33--DPP
Instr.
Open
9.42
Qwen2.5-3B
Base
Open
8.84
Qwen2.5-7B
Base
Open
8.75
Llama-3 1onna-3.1-Caae 1.11--ntauur-08B
Base
Open
8.54
Gemma-3-1221 3-12PPPT
Base
Open
7.66
Owen2.5-7BB-Ime
Instr.
Open
6.36
Semmm--327BBPP"
Base
Open
5.53
Qwen2.5-1.55
Base
Open
5.34
Llama-3.1 3.1-BB
Base
Open
4.12
OI-M--2-112-77
Base
Open
2.56
Llama-3.11700
Base
Open
1.21
Llama-3 3.1-8BBInnttruct
Instr.
Open
-0.15
Gemma-3-44 3-4B-PT
Base
Open
-0.65
Demmma-3-4B-Be
Instr:
Open
-1.11
Qwen2.5 5-0.5B
Base
Open
3.00
OLMo 2-1122-7B-SFT
Instr.
Open
11.36
Qwen2.5-3B. --3-Inttettt
Instr.
Open
12.04
Gemma-3-1B-B
Base
Open
-16.17
Ol 2-112--7B-D
Instr.
Open
19.62
OLMo-2-1124 2-1124-13B-SS
Instr.
Open
20.54
OLMo 11-14-B-IIsstruu
Instr.
Open
-1.36
26

---

Published as a conference paper at ICLR 2026
Quear
10
-10
Bax
310
Parsmnler (Billion))
Figure 6: Model parameter count vs. simulation ability. We measure model size by parameter count and simulation ability by SIMBENCH score S averaged across the two main splits of SIMBENCH.
H.1 OVERALL MODEL RANKING AND SIGNIHICANCE
Table & expands on the results from Table in the main paper. We report the 95% confidence interval (CI) for each model's mean SIMBENCH score. We also perform independent two-sample t-tests to determine if the performance difference between each model and the one ranked immediately below it is statistically significant. The results confirm a clear and statistically robust performance hierarchy among the evaluated models.
Table 8: Detailed statistical analysis of SIMBENCII scores for the top models. Pairwise significance tests compare each model the one ranked immediately below it. Symbols indicate significance: "p < 0.05, "'p p < 0.01, *"" < 0.001.
Rank
Model
Score (S) + SE
95% CI
1
Claude-3.7-SS 3.7--onnet
40.80(43) (33)*
[39.96, 41.64]
2
Claude-3.7-Sonne-4400
39.46 42) ...
[38.63, 40.29]
3
GPT-4.1
34.55 48)
[33.62, 35.50]
4
DeepSeek-R1
34.52 43 "*
[33.68, 35.37]
5
DeepSeek- V3-0324
32.89 42 ***
[32.07,33.72]
6
04-mini-high
28.99
[27.97, 30.02]
7
Llama-3. 1-011B--
28.40 28.40(49)
[27.46, 29.36]
8
04nmini-low
27.77
[26.77,28 28.78]
H.2 WITHIN-FAMILY PERFORMANCE ANALYSIS
To investigate the impact of model size (RQ2), we conducted one-way ANOVA followed by post-hoc Tukey HSD tests for pairwise comparisons within model families. The results, presented in Table 9 show that for most families, increases in model size lead to statistically significant improvements in simulation ability, supporting the scaling trends observed in Figure 2.
27

---

Published as a conference paper at ICLR 2026
Table 9: Within-family pairwise performance comparisons. All differences are statistically significant at 0.001 unless otherwise noted (ns).
Model Family
Comparison
Result
Hlama-3.1-Instreee
405B 70B
AS 11.84 (p 0.001)
(ANOVA: F=518.8, p<0.001)
405B vs 8B
AS 28.55 (p 0.001)
70B 8B
AS 166.71 (p 0.001)
Gemma-3-IT
27B vs 12B
AS 0.29 (ns, p = 0.673)
(ANOVA: F=710.1, p<0.000)
12B vs 4B
AS 20.54 (p 0.001)
27B vs 4B
AS 20.24 (p 0.001)
Qwen2.5-Instruc
72B vs 32B
A$ = 3.85 < 0.001)
(ANOVA: F=518.8, p<0.001)
72B 14B
AS 6.86 (p < 0.00T)
72B VS 7B
AS = 21.24 (p< 0.001)
32B VS 14B
AS 3.01 (p 0.001)
32B VS 7B
AS = 17.39 (p < 0.001)
14B vs 7B
AS 14.38 (p < 0.001)
I ANALYSIS OF THE ALIGNMENT-SIMULATUE TRADEOFF
1.1 OBSERVATIONAL ANALYSIS
As a complement to the analysis in $4.4 this section provides a direct observational comparison of how base and instruction-tuned models perform as a function of human response plurality. We operationalize response plurality as the normalized entropy of the human response distribution, where high entropy indicates widespread disagreement while low entropy indicates strong consensus. Simulation fidelity is measured by Total Variation Distance (TVD), where lower values indicate better performance. Figure 7
visualizes this relationship for all question-model pairs in SimBenchPop, separated by model type. The plots reveal clear and divergent behavioral pattern:
Base models exhibit negative correlation between entropy and TVD. This demonstrates observable property of this model class: they are generally more accurate (lower TVD) on high- entropy questions where human opinions are diverse.
Instruction-tuned models exhibit a positive correlation. This demonstrates the opposite property: they are generally more accurate on low-entropy questions where humans have reached a consensus.
This direct visualization establishes a core empirical finding: the two model classes have fundamen- tally different strengths when simulating human responses across the spectrum of opinion plurality. The analysis in the main paper ($4.4) builds upon this observation to more formally quantify the effect of post-training that drives this divergence.
1.2 REGRESSION ANALYSIS
To formally test the relationship between human response entropy and simulation performance across different model types, we an Ordinary Least Squares (OLS) regression model predicting Total Variation (TV) distance at the individual question-model level. The model specification was as follows:
Total Variation [iidatasetnnmpe + + (instrucct flag) Human Normalized Entropy
Here, Total_ Variation is the dependent variable. C'dataset name) and C(model) represent fixed effects for each dataset and model, respectively, controlling for baseline differences in difficulty and capability. The crucial term is the interaction Ciintrruct flag) Human Normalized, Bormalized_EEttopyy.. where instruci_fag is a binary indicator for instruction-tuned models (0 for base, for instraction-tunee))
The key results from Table 11 the coefficients for the interaction terms:
28

---

Published as a conference paper at ICLR 2026
Table 10: TVD for each model in SimpenchPop and SimBenchGrouped. Lower values indicate better performance. PT and IT refer to pretrained and instruction- tuned versions, respectively.
Model
SimBenchPop
SimBenchGrouped
Average
Baselines
Uniform baseline
0.335
0.362
0.348
Models
Claude-3.7 3anude-3.--Sonet
0.197
0.183
0.191
Cland--37-Somne-40000
0.201
0.189
0.195
GPT-4.1
0.212
0.206
0.209
DeepSeek-RI
0.211
0.212
0.211
DeepSeck-V3-0324
0.215
0.218
0.216
04-mini-high
0.235
0.214
0.225
04-mini-low
0.234
0.226
0.230
Llama-3.1-405BB
0.237
0.225
0.231
Owen2.5-72B-Immg I-sttructtt
0.229
0.246
0.237
Qwen2.5-32B Instruutttttt
0.242
0.258
0,250
OLMo-2-0325--2BB-D
0.258
0.258
0.258
Owen2.55-44BInstrueet
0.247
0.270
0.258
OLMo-2-0325-322--mmme Instruct
0.263
0.260
0.261
Manna-3.--7B-Imssiii
0.277
0.247
0,263
Cemmma-3-12B-.
0.262
0.274
0.267
Gemma 1ommma---27BB-o
3-27B-IT
0.270
0.273
0.272
OLMo-2-0325-32B
0.271
0.297
0,283
OLMo-2-0325-32B-55
0.298
0.265
0.283
Qwen2.5-72B
0.268
0.300
0.283
Qwen2.5-32B
0,273
0.308
0,290
Llama-3 1omma-3.1-Miie 3.1-Minitaur-B1 8B
0.288
0.296
0.292
OLMo-2-1124-11 124-13B
0.284
0.302
0.293
Qwen2.5-14B
0.285
0.314
0.298
OLMo-2-1124-13BB-
0.293
0,306
0,299
OLMo-2-1124-153 Instruct
0.295
0.304
0.299
Qwen2.5-7B
0.290
0.326
0.307
Llama-3. 3.10a-3.1-44 1-CCntturr70B
0.309
0.313
0.31
Swen2.5-7B-Instruse
0.292
0.332
0.311
Qwen2.5-3B
0.300
0.327
0.313
Gemma-3 3-12B-PT PT
0.310
0.317
0.314
Gemma-3-27B-t 3-27B-PT
0.309
0.325
0,317
Llama-3.1 -nItruucctt
0.321
0.318
0.320
Qwen2.5-1.58
0.321
0.324
0.322
Llama-33 -lama-3.1-8B
0.326
0.323
0.324
Llama- 3.1-70B
0.331
0.324
0.328
OLMo-22 1-Mo-2-11 2-1124-BB
0.324
0.349
0.336
Gemma-3-4B-PP 3-4B-PT
0.334
0.341
0,337
Gemma-3-A4 3-4-IT
0.337
0.341
0.339
Qwen2.5-0.5B
0.337
0.364
0.349
OLMo-2-1124-7B-SPT
0.393
0.355
0.375
Dwen2.5-3B-Innttun
0.397
0,363
0.381
Gemma-3-1B-PT
0.382
0.414
0.397
OL.Mo-2-1124-7BBB
0.413
0.382
0.399
OLMoo--111247BB-nnnmmuu
0.420
0.386
0.404
OLMo-2-1124-13B--0
0.416
0.414
0.415
29

---

Published as a conference paper at ICLR 2026
Table 11: Results: Ordinary least squares
Mideel: OLS
Adj. R-squared:
0.168
Dependent Variable:
Tosal, Variation
AIC:
1163587.0704
2025-09-23
-163085
Observation:
207837
Log- ikelihood:
81843,
Df Model:
48
F F-satistic:
875.3
Df Resiuall::
207788
Prob (F (F-saaissic)::
0.00
squared:
Coef.
Sia.Emr
P> [I
(0.023
0.975
Intereept
0.1811
0.0011
73,5586
00000
0.1761
0.185
Cidatsset [.
400334
4.092
0.0019
-29.9249
0.000
001010
0.0954
0.0240
0.0017
-5.0722
0.000
0,033
0.0147
40031
-15,8345
-0.0522
400220
nadmi [[SS]]
000299
Cidanset CidmtseL [
0,0406
0.0019
-2.1621
0.0000
0.0444
0.0359
[acintse__aamcffinn (SSSSP
4U0099
0.00119
-15.1813
0,0000
-0,030
.00225
Cidanset_namell
0.1046
Cidataset, aataaaae_so@@TTltiaaaaaaaeaa [Taithhhhhhhhre
-0.005
+00330
Cidottset MMccMMMiiMcc
410438
0.0019
-22.988
0.000
.00051
0.000
Chamiseeaammel Marmmm///MMMaaMaiiieeeeeaaaa
40.6666
0,0079
-110836
0.0000
.0.1331
0.1520
Cidatsset Caatileee_ammeTTTTnaaaaaaaa
4242
-000890
-0.0814
Chdataaet_nammlll ptt.:aaammmaaaa/ae (.SPPyyy
-0.125
400200
0.0033
-5,9006
0.0000
1200000
000144
Chdatsset, [ame:][00PPyyeeMEE..
40.1221
0,0019
-57.9955
0.0000
40003
10.1199
0.0235 SSSS
Chdstaset same:[T.Opinion(M
56,3479
0.0000
Cidatiset same[[T:TSP]
<00429
0.0020
-21.900
0.0000
Chdatsse_nammily [aamee//EENasaoootttttoo
<000224
0,0022
-T0874
0.0000
1000286
400162
CiModda][TDDeegaskRRR]]
0.114
CiMoodi/TTDerpSees.....aaa
6,687
0,000
0,204
CIModel][T.GPPi4.11
0,0122
0.0024
5.1048
0.000
0.0168
0.0622
0.074
26.3002
0.0000
000555
0.0668
115.064
0.3581
20-1055
0.0000
00000
0,075
CiMooda][TTemmm-42778111
0.3509
00011
114,6053
0.0000
0,3449
0.3560
(TMedelj[TL.aamm33.111
0.0573
0.0024
15.7766
0.0000
0:0336
0.0419
32.7116
0.0819
CIModel]j11Lamm-31...8]
0.3676
0.0011
120.1756
0.0000
013010
0.3736
0.1312
0.004
51,3589
0.0000
11116
0.1258
CiMedel][T.O1
M0-2415523
0.306
0.3185
26-9244
00000
0.0682
15-2-1144333
0.3361
0.001
106. 9964
0.0000
0.3201
0.3321
0.0953
0.0044
4013576
0.0000
0.007
0.0999
CiMooel][T.Q@m225...5...
0.3561
CIMoodaljT.Qqwe22.5.1110
106.6913
0.3204
0.3524
0.4811
(10024
20.3827
0.0000
0.0435
0.0828
C(Model)
0.3152
0.001
103.0509
0.0000
0.3091
0.3212
18,032
0.3362
CiMoskl)[T.Quabb
0.3103
0.0011
101 1261
0.0000
0.3043
0.3163
(MModal(TQQQwn2.5-200
0.0092
0.004
12.3490
0.0000
0.0346
0.0358
[MMoeel(T.Qween35..
CMModel[[To-mmii--iig
15,0086
0.0000
0.0401
0.944
0.0074
14,5682
0.0000
0.0298
0.0390
Clinstrut [ogp Nomallzed Entropy
002451
0.0024
-00.0554
0.0000
0.2499
0.2603
fag]iinstnc]HHamam Normalized Enerpy
Cuunbbu::
32553.497
Durbas Watson:
1.732
Prol-Ommibus):
0.000
Janque Beuue-Berii (JR)
60320.484
0.995
Proh(B):
Kunosis:
4.733
Coadition
30

---

Published as a conference paper at ICLR 2026
Base Models (N=12)
Instruction- Tuned Models (N=15) 15)
1,00
Stope
Slope:
Variation Distanet
0.35
500
Depsity of Data Points
000
Dansity of Data
0.46
200
0.00
Human Response Entropy (N (Normalized)
Human Response Entropy (Normaliaed)
Figure 7: Response plurality simulation fidelity for base and instruction- models all questions in SimBenchPop. We measure response plurality by normalized entropy of the human response distribution and simulation fidelity by total variation distance at the question level.
For base models: The coefficient on the interaction between base models and Human Normalized Entropy is 00.2451 (p < 0.001), indicating that for every one-unit increase normalized entropy, the TVD decreases by approximately 0.25 units. This means that base models perform better (lower TVD) when simulating human populations with more diverse opinions.
For instruction-ture models: The coefficient on the interaction between instruction-tuned models and Human Normalized Entropy is +0.0997 (p < 0.001), indicating that for every onc-unit increase in normalized entropy, the TVD increases by approximately 0.11 units. This means that instruction-tuna models perform worse (higher TVD) when simulating human populations with more diverse opinions.
These coefficients are both highly statistically significant (p < 0.001) and represent substantial effect sizes given that TVD ranges from 0 1, The model as whole explains approximately 17% of the variance in TVD (R2 0.168), which is substantial for a dataset of this size and complexity. The opposite signs of these coefficients provide strong evidence for our hypothesis that base models and instruction-tune tuned models respond differently to the challenge of simulating populations with diverse opinions. This pattern holds even after controlling for the specific datasets and models involved, suggesting it represents general property of the two model classes rather than an artifact of particular model or evaluation datasets.
1.3 CAUSAL MEDIATION ANALYSIS: DECOMPOSING THE DUAL EFFECTS OF INSTRUCTION TUNING
To formally test the causal mechanisms behind the alignment- simmulatiiin tradeoff ($4.4) ducted causal mediation analysis. This analysis aims to decompose the effect of instruction tuning, separating its impact the model's output entropy from other effects like improved instruction following.
Causal Model Our analysis based on linear mediation model designed to decompose the total effect of instruction tuning into direct and indirect pathways. The hypothesized causal graph is as follows:
Instruction Tuning (Treatment, X) ++ Model Prediction Entropy (Mediator, M) SimBench Score
(Outcome, Y)
The central hypothesis that instruction tuning (X) affects simulation performance (Y) at least partially through its systematic effect the entropy of the model's output distribution (M).
31

---

Published as a conference paper at ICLR 2026
Methodology and Model Specification The analysis was implemented in Python using the stat smodels library (Seabold & Perktold, 2010). We fit two Ordinary Least Squares (OLS) regression models to estimate the relevant causal paths, following the standard mediation framework:
Mediator Model (Path a): We first model the effect of instruction tuning on the mediator (model prediction entropy). The model is specified
M, OM ax; T + eM,
where M; is the normalized entropy of the model's prediction for given question, X; is binary indicator for whether the model is instruction-tunet (1 if true, 0 if base model), and Z₃ is vector of control variables. The coefficient a captures the average effect of instruction tuning model prediction entropy.
2. Outcome Model (Paths b and c'): We then model the outcome (SimBench Score) as a function of both the treatment and the mediator:
Y; ay + 6M, + + EY.
where Y; is the SIMBENCH Score. The coefficient e' represents the direct effect of instruction tuning on performance, holding model entropy constant. The coefficient b represents the effect of model entropy performance, holding instruction tuning constant.
Control Variables In both models, the vector Z; includes a set of control variables to account for potential confounders:
We control for the normalized entropy of the ground-truth human answer distribution to isolate the model's behavior from the inherent plurality of the question. Fixed Effects: We include fixed effects for both the model family (e.g.. Llama-3.1, Qwen2.5) and the dataset to absorb any baseline differences in performance or entropy across these groups.
Human Response Entropy:
Effect Calculation The key effects are calculated from the estimated coefficients:
Direct Effect (c'): Directly estimated from the outcome model.
Indirect Effect (a x b): The effect of instruction tuning that is mediated through model entropy. calculated as the product of the coefficients from the two models. The statistical significance of this indirect effect was assessed using the Sobel test approximation for the standard error.
Total Effect: The sum the direct and indirect effects (c' a x b).
This decomposition allows us to quantify how much of instruction tuning's overall impact on simulation fidelity attributable to its catropy-suppressigg nature versus other factors like improved instruction following.
Results and Interpretation Our analysis reveals that instruction tuning has two distinct and opposing effects simulation ability. The overall total effect modest but significant improvement of +4.72 points on the SIMBENCH score (p < .001). However, this net effect masks two powerful underlying mechanisms:
1. A Harmful Indirect Effect (-1.74 points): Instruction tuning significantly reduces model prediction entropy (Path A: 3 -0.11, < .001). In our models, higher entropy is generally associated with better performance (Path B = 15.60, p < .001). The indirect effect (A x B) is therefore negative (-1.74), quantifying the performance penalty that instruction tuning imposes by forcing the model into low-entropy, mode-seeking behavior.
2. A Strong, Helpful Direct Effect (+6.46 points): After accounting for the change entropy. a large positive direct effect remains (3 = +6.46, < .001). This reflects the benefits of instruction tuning that are independent of its impact on output diversity, such as improved instruction following and a better ability to reason about the specified persona.
Conclusion These results provide strong evidence for inconsistent mediation and resolve a key paradox in our findings. While analysis in $4.44shows that instruction tuning harms simulation fidelity on high-entropy questions, our main leaderboard (Table []] shows that the best overall simulators instruction-tuned. This mediation analysis explains why: the total effect of instruction
32

---

Published as a conference paper at ICLR 2026
tuning is the net outcome of two larger, opposing forces. First, a direct positive effect (+6.46 points) on capability, likely from improved instruction- and persona-following. Second, a smaller but significant indirect negative effect (-1.74 points) caused by entropy suppression. The net positive effect (+4.72 points) demonstrates that, on average, the direct benefits of alignment currently outweigh the harm from reduced distributional diversity. Future work on creating SOTA simulators should therefore focus on developing hybrid or ceteribution-rrseervii alignment methods that retain the direct benefits of instruction tuning while mitigating its harmful, entropy-reducing side effects.
J DETAILED CORRELATION ANALYSIS (RQ6)
To support our analysis in Section 4.6 this appendix provides the detailed data sources and scatter plots illustrating the correlation between SIMBENCH scores and five externa capability benchmarks. This analysis includes the subset of our evaluated models for which performance data on these external benchmarks could be reliably sourced. The benchmark performance data was collected from model developers' technical reports, the Open LLM Leaderboard FForrier et al. (2024), and Vals AI, Inc. (2025). Table 12 summarizes the Pearson correlation coefficients, and Figure presents the individual scatter plots for each benchmark.
Table 12: Summary of Pearson Correlation Coefficients (r) between SIMBENCH scores and external capability benchmarks for the models evaluated in our study.
Capability Benchmark
Pearson's r
MMLU-Pro
0.939
GPQA Diamond
0,862
IF-Eval
0.786
Chatbot Arena ELO
0.708
OTIS AIME
0.479
K CASE STUDY: DETAILED ANALYSIS OF CENTAUR
We present detailed visualization of model performance across the spectrum of human response entropy. Figure / breaks down the SIMBENCH Score for the Llama-3.1 8B and 70B models: base, instruction-tuned, and specialist cognitive-tuned (Minitaur/Centaur)) binned by the normalized entropy of the human ground truth. The figure reveals that these two fine-tuning paradigms improve simulation ability in fundamentally different and complementary ways. General-purpose instruction tuning excels low-entropy regimes where there is a clear human consensus. The orange, dashed lines for both 8B and 70B Instruct models show the highest performance (SIMBENCH Score) when entropy is low, but this advantage systematically decays as human opinions become more diverse. This aligns with its mode-seeking objective: trains the model identify and produce a single "correct" or preferred response. Specialist cognitive tuning, in contrast, mirrors the behavior of base models. The green and blue dash-dotted lines for Minitaur and Centaur show distinct pattern: performance is weaker on low-entropy tasks but progressively improves as human response entropy increases. This suggests that fine-tuning on behavioral data preserves or even enhances the base model's 's mass-covering ability to represent a diverse distribution of outcomes, rather than forcing it into a single mode.
This qualitative divergence is key. The two methods are not just different in degree, but in kind. Instruction tuning boosts performance by sharpening a model's ability to follow prompt instructions and converge on a consensus answer. Specialist tuning boosts performance by aligning the model's internal representations more closely with the patterns of human choice. Because they target different mechanisms, their benefits not mutually exclusive. This suggests that perhaps future gains in LLM simulation will come from bridid approaches that synthesize both paradigms, creating models that are both generally capable and foundationally aligned with the nuances of human behavior.
33

---

Published as a conference paper at ICLR 2026
50
40
30
30
40
51
60
70
04
0.5
0.7
0.5
MMLU-Pro Score
GPQA Diamond Score
(a) SimBench vs. MMLU-Pro (r = 0.939)
(b) SimBench vs. GPQA Diamond (r 0.862)
50
glaco
10
0
1876
Ev-ll Score
Chatbot Arena Score
(c) SimBench vs. 1v-Eval (r 0.786)
(d) SimBench vs. Chatbot Arena ELO =r 0.708)
40
1
30
Shockeethh
02
0.0
0.1
0.2
0.8
0.4
0.5
0.6
0.7
0.8
OTIS
AIME Score
(e) SimBench vs. OTIS AIME (r= 0.479)
Figure 8: Scatter plots showing the correlation between average SIMBENCH scores and performance on five external benchmarks. Each point represents LLM. The strong positive correlation is most pronounced for inotteege-intensive reasoning tasks (a, b).
34

---

Published as a conference paper at ICLR 2026
70B Models
8B Models
Model
Centair
StmiBenes Soore
Normalized
Respousc Entropy
Normalized
Resgonse Encropy
Figure 9: Effect of Centaur Fine-Tuning, The plots show the binned SIMBENCH Score against normalized human response entropy for Llama-3. models at the 70B (left) and 8B (right) scales. Shaded areas represent the 95% confidence interval of the mean score in each bin.
L DATASET DETAILS
We provide details on each of the 20 datasets in SIMBENCH. Note that for many datasets we use only subset of questions and participants for SIMBENCH, as a result of our preprocessing steps ($2.3)
L.1 WISDOMÖFCROWDS
This dataset contains factual questions that were administered to a large number of US-based Amazon Mechanical Turk workers. The data was originally collected to study wisdom of the crowd effects.
Description:
Questions: 114, with average of 518 responses per question.
Example question:
An analogy compares the relationship between two things ideas to highlight some point of similarity. You will be given pairs of words bearing a relationship, and asked to select another pair of words that illustrate similar relationship.
Which pair of words has the relationship 'Letter Word'?
(A): Page: Book
(B): Product : Factory
(C): Club: People
(D): Home work School
Participants: 722 US-based Amazon Mechanical Turk workers.
Participant grouping variables (n=4): age_grge gge_group: age bracket, gender: self-reported gender, education: education level, industry: the industry of the participant's job.
Default System Prompt:
You are an Amazon Mechanical Turk worker from the United States.
License: MIT
Publication: Simoiu al. 2019
35

---

Published as a conference paper at ICLR 2026
L.2 JESTER
Description:
This dataset contains jokes for which participants provided subjective judgments of how funny they found them. The data was originally collected to enable recommender systems and collaborative filtering research.
Questions: 136, with an average of 779 responses per question.
Example question:
How funny the following joke, on scale of -10 10? (-10: not funny, 10: very funny)
How many feminists does it take screw in light bulb? That's not funny.
Options:
(A): 7 to 10
(B): 3 to
(C): -2 to
(D): -5 to -3
(E) -6
Participants: 7,669 volunteer participants (sociodemographics unknown) who chose to use the Jester joke recommender website.
Participant grouping variables: None.
Default System Prompt:
Jester is joke recommender system developed UC Berkeley study social information filtering. You are user of Jester.
License: "Freely available for research use when cited appropriately."
Publication: Goldberg et al. (2001)
L.3 CHOICES 13K
This dataset contains large number of automatically generated decision-making scenarios that present participants with two lotteries to choose from. The data was originally collected to discover theories of human decision-making
Description:
Questions: 14,568, with an average of 17 responses per question.
Example question:
There are two gambling machines, and B. You need to make choice between the machines with the goal of maximizing the amount of dollars received. You will get one reward from the machine that you choose. A fixed proportion of 10% of this value will be paid to you as performance bonus. If the reward is negative, your bonus is set to $0.
Machine A: $-1.0 with 5.0% chance, $26.0 with 95.0% chance. Machine B: $21.0 with 95.0% chance, $23.0 with 5.0% chance.
Which machine do you choose?
Participants: 14,711 1 US-based Amazon Mechanical Turk workers.
Participant grouping variables: None.
Default System Prompt:
You Amazon Mechanical Turk worker based in the United States.
36

---

Published as a conference paper at ICLR 2026
License: "All data
available
the
public
without
registration
gthab..cmm/pptersoo/hooceel11
Publication:
Peterson et al. (202
L.4 OPINIONQA
Description:
This dataset contains survey questions that ask participants to provide self-assessments and sub- jective judgments. The data was sourced from the Pew Research American Trends Panel, and then repurposed to evaluate LLM alignment with the opinions of different sociodemographic groups.
Questions: 736, with average of 5,339 responses per question. Example question:
How would you describe your household's financial situation?
(A): Live comfortably
(B): Meet your basic expenses with a little left over for extras
(C): Just meet your basic expenses
(D): Don't even have enough to meet basic expenses
(E): Refused
Participants: roughly 10,000 paid participants from representative sample of the US populace.
Participant grouping variables (n=13): CREGION: U.S. region of residence, AGE: age bracket of the respondent, SEX: male or female, EDUCATION: highest level of education completed, CITTZEN: the respondent is (not) citizen of the US, MARITAL: current marital status, RELIG: religious affiliation, RELIGATTEND: frequency of religious service attendance, POLPARTY: political party affiliation, INCOME: income bracket, POLIDEOLOGY: political ideology (e.g., liberaa/connseraative), RACE: racial identity.
Default System Prompt:
You are from the United States.
License:
No licensing information
provided;
Data is
freely available with-
out
ht https: www.wweet.swsseeoeeoooo codalatc hwtpbb/owwwwwooseeeeeeeeee..eeee 0x61b693719977477666 33c37dd333333
registration at
Publication: Santurkar et al. (2023)
L.5 MORALMACHINECLASSII
This dataset contains three moral decision-making scenarios, for which participants provided subjective choices. The data originally collected study universals and variations in moral decision-makinn across the world.
Description:
Questions: 3, with an average of 17,720 responses per question.
Example question:
A man in blue is standing by the railroad tracks when he notices empty boxcar rolling out of control. It is moving so fast that anyone hits will die. Ahead on the main track are five people. There is person standing does nothing, the boxcar will hit the five people on the main track, but not the one person on the side track. If the man in blue flips switch next to him, it will divert the boxcar the side track where will hit the one person, and not hit the five people on the main track. What should the man in blue do?
side track that doesn't 't rejoin the main track. If the man in blue
37

---

Published as a conference paper at ICLR 2026
Participants: 19,720 volunteer participants (sociodemographics recorded) who chose to share their choices on the Moral Machine Classic web interface.
Participant grouping variables (n=6); country: respondent's country of residence, gender: gender of the respondent, education: level of education, age group: age bracket, political group: self-identified political orientation, religious group: self-identified religious affiliation.
politi:
Default System Prompt:
The Moral Machine website (moralmachine.mit.ddu) was designed to collect large-scale data on the moral acceptability of moral dilemmas. You are a user of the Moral Machine wehsite.
License: No licensing information provided.
Publication:
Awad al. (2020)
I.6 CHAOSNLI
This dataset contains natural language inference scenarios which participants were asked provide subjective judgments on. The data was originally collected study human disagreement on natural language inference scenarios.
Description:
Questions: 4,645, with exactly 100 responses per question.
Example question:
Given premise and hypothesis, determine if the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) based on the premise.
Premise: Two young children in blue jerseys, one with the number and one with the number 2 are standing on wooden steps in bathroom and washing their hands sink.
Hypothesis: Two kids at a ballgame wash their hands.
Choose the most appropriate relationship between the premise and hypothesis: (A): Entailment (the hypothesis must be true if the premise true)
(B): Contradiction (the hypothesis cannot be true if the premise is true)
(C): Neutral (the hypothesis may or may not true given the premise)
Participants: 5,268 Amazon Mechanical Turk workers.
Participant grouping variables: None.
Default System Prompt:
You are Amazon Mechanical Turk worker.
License: CC BY-NC 4.0
Publication: Nie al. (2020)
L.7 EUROPEAN SOCIAL SURVEY (ESS)
This dataset contains three waves of survey questions that ask participants to provide self-assessments and subjective judgments. The data was originally collected study attitudes and behaviors across the European populace. We use ESS wave 8-10.
Description:
Questions: 237, with average of 41,540 responses per task.
Example question:
Sometimes the government disagrees with what most people think is best for the country. Which one of the statements on this card describes what you think is best for democracy general?
38

---

Published as a conference paper at ICLR 2026
Options:
(A): Government should change its policies
(B): Government should stick its policies
(C): It depends on the circumstances
Participants: Around 40,000 participants in total from European countries.
Participant grouping variables (n=14); entry: respondent's country of residence, age_ group: age bracket, gndr: gender of the respondent, eisced: level of education (ISCED classification). household_size_greee _grup:: size of the household, mnactic: main activity status, rlgdgr: degree of religiosity, Irscale: self- eef-placement on left-right political scale, brnentr: born in the country or abroad, etzentr: citizenship status, domicil: urban rural living environment, dscrgrp: member of group discriminated against, uemp3m: unemployed in the last months, maritalb: marital status (married, single, separated, etc.).
Default System Prompt:
The year is (survey year).
License: BY-NC-S 4.0
Publication: European Social Survey European Research Infrastructure (ESS ERIC) 20244
L.8 AFROBAROMETER
Afrobarometer is an annual public opinion survey conducted across more than 35 African countries. It collects data on citizens' perceptions of democracy, governance, the economy, and civil society, asking respondents for self-assesmments and subjective judgments. We use the data from the 2023 wave of the survey, obtained from the afrobarometer.org website. We use Afrobarometer Round 9.
Description:
Questions: 213, with average of 52,900 responses per question. Example question:
Do think that in five years' time this country will be more democratic than it is now, less democratic, or about the same?
Options:
(A): Much less democratic
(B): Somewhat less democratic
(C): About the same
(D): Somewhat more democratic
(E): Much more democratic
(F): Refused
(G): Don't know
Participants: 1, 2,200-2,440 per country, 39 countries
Participant grouping variables (n=l1): country: respondent's s country, gender: male or female, edu- cation: education level, ggopg: age bracket, religion: stated religion, urban_rural: area living, employment: job situation, bank_accoume whether respondent has bank account, ethnic ethnic_group: group: respondent's 's ethnicity, subjective income: iincemeeveeiiomeee how often to go without cash income, discuss pooltiiss how often to discuss politics.
Default System Prompt:
The year is (survey year).
License: No explicit language forbidding redistribution.
39

---

Published as a conference paper at ICLR 2026
Publication:
Afrobarometer (2023)
1.9 OSPSYCHBIGS
Description:
This dataset contains collection of anonymized self-assessments from the Big Five Personality Test, designed to evaluate individuals across five core personality dimensions.
Questions: 50, with an average of 19,632 responses per question.
Example question:
Indicate your level of agreement with the following statement: am always prepared.
Options:
(A): Disagree
(B): Slightly Disagree
(C): Neutral
(D): Slightly Agree
(E): Agree
19,719 volunteer participants from all over the world, who chose to share their assessments the dedicated Open-Source Psychometrics web interface
Participants:
Participant grouping variables (n=3): country_name: country of residence, gender_cat male, female, or other, age_ age_group: age bracket.
Default System Prompt:
openpsychometrres.oom org is website that provides a collection of interactive personality tests with detailed results that can be taken for personal entertainment or learn more about personality assessment. You are user of openpsychometriesorg.
License: Creative Commons.
Publication: None.
L.10
OSPSYCHMGKT
Description: This dataset contains anonymized test results from the Multifactor General Knowledge Test (MGKT), psychometric instrument designed to assess general knowledge across multiple domains. Each of the original 32 questions presents 10 answer options, of which 5 are correct, For consistency with other datasets our study, we expand each question into separate binary-choice items, each asking whether a given option is correct.
Questions: 320, with an average of 18,644 responses per question.
Example question:
Is "Emily Dickinson" an example of famous poets?
Choose one:
(A) Yes
(B) No
Participants: 19,218 volunteer participants from all over the world, who chose to share their assessments on the dedicated Open-Source Psychometrics web interface
Participant grouping variables (n=4): muniry_namee country of residence, gender_car male, female, or other, age group: age bracket, engnat_catt (not) native English speaker.
Default System Prompt:
40

---

Published as a conference paper at ICLR 2026
openpsychomeerricsoogg is website that provides collection of interactive personality tests with detailed results that can be taken for personal entertainment or learn more about personality assessment. You are user of openpsychomeetris.org
License: Creative Commons.
Publication: None.
L.11 OSPSYCHMACH
This dataset contains anonymized self-assessments from the MACH-IV test, a psy- chometric instrument assessing the extent to which individuals endorse the view that effectiveness and manipulation outweigh morality in social and political contexts, i.e. their endorsement of Machiavellianism.
Description:
Questions: 20, with an average of 54,974 responses per question.
Example question:
Indicate your level of agreement with the following statement: Never tell anyone the real reason you did something unless it is useful to do so.
Options:
(A): Disagree
(B): Slightly disagree
(C): Neutral
(D): Slightly agree
(E): Agree
Participants:
73,489 volunteer participants from all over the world, who chose to share their assessments on the dedicated Open-Source Psychometrics web interface
Participant grouping variables (n=18): country_namee country of residence, gender_cat male, female, or other, age_group: age bracket, race_cat: respondent's race, engnat_cat: is (not) a native English speaker, hand_catt right-, left-, or both-handed, education _cttitooonattttt level of education, urban type of urban area, religion_catt _ctt stated religion, orientation_cem _cat: sexual orientation, voted_cot did (not) vote at last elections, married cat: never, currently, or previously married, familysize: number of people belonging to the family, TIPI E 5_Group: extraversion level based on TIPI score, TIPI A Group: agreeableness level based on TIPI score, TIPI C Group: conscientiousnes: level based on TIPI score, T7PI_ES _S__ropp: emotional stability level based on TIPI score, O_Group: openness-to-expeed
level based on TIPI score.
Default System Prompt:
openpsychometrics.oom org is website that provides collection of interactive personality tests with detailed results that can be taken for personal entertainment or to learn more about personality assessment. You are user of openpsychometries.rrg
License: Creative Commons.
Publication: None.
L.12 OSPSYCHRWAS
Description:
This dataset contains anonymized seff-assessments from the Right-Wing Authoritarian- ism Scale (RWAS), psychometric instrument assessing authoritarian tendencies such as submission to authority, aggression toward outgroups, and adherence to conventional norms.
Questions: 22, with an average of 6,918 responses per question.
Example question:
41

---

Published as a conference paper at ICLR 2026
Please rate your agreement with the following statement a scale from (A) Very Strongly Disagree (I) Very Strongly Agree.
Statement: The established authorities generally turn out to right about things, while the radicals and protestors are usually just "loud mouths" showing off their ignorance.
Options:
(A): Very Strongly Disagree
(B): Strongly Disagree
(C): Moderately Disagree
(D): Slightly Disagree
(E): Neutral
(F): Slightly Agree
(G): Moderately Agree
(H): Strongly Agree
(I): Very Strongly Agree
Participants: 9,881 volunteer participants from all over the world, who chose share their assess- ments the dedicated Open-Source Psychometries web interface.
Participant grouping variables (n=22): age age_group: age bracket, gender_cate male or female or other, ruce_cat respondent's s race, engnai_cuts is (not) English native, hand_cat: right/left/both- handed, education_cal catt level of education, urban_cca catt type of urban area, religion_cae _att: stated religion, orientation_cam sexual orientation, voted: did (not) vote at last elections, married: wwwer//urrenlly/rrvvvousl, familyysse number of people belonging to the family, TIPI E_ E_Group: extraversion level based TIPI score, TIPI A AUPI_A_Grom 2_Group: agreeableness level based TIPI score, TIPI_ C_ C_Group: conscientiousness level based on TIPI score, TIPI_ ES Group: emotional stability level based on TIPI score, TIPI_O O_Goppp: openness-to-experienee level based on TIPI score, house- hold_ine ind_mincmm:: income sufficiency, work_stalus: job situation, or_of_persons peersons_in_ttee householh: 1-7+, marital status: respondent's 's legal relationship status, domicil: type of urban area.
Default System Prompt:
openpsychometrics.om org is website that provides collection of interactive personality tests with detailed results that can be taken for personal entertainment or to learn more about personality assessment. You are user of openpsychomeeties.org.
License: Creative Commons.
Publication: None.
L.13 INTERNATIONAL SOCIAL SURVEY PROGRAMME (ISSP)
Description: The International Social Survey Programme (ISSP) is cross-national collaborative programme conducting annual surveys on diverse topics relevant to social sciences since 1984. Of all 37 surveys, here we include only the five most recent surveys, which were collected in the years 2017 to 2021.
Questions: 1,688, with an average of 7.074 responses per question.
Participants: 1,000 1,500 per country per wave
Participant grouping variables (n=ll): country: respondent's country, age: age bracket, gender: male or female, years_earrr of educcatiinn: 1-10+, household income: income sufficiency, work status: job situation, religion: stated religion, nr persons in household: 1-7+, marital status: respondent's legal relationship status, domicil: type of urban area, topbot: self-assessed social class.
Default System Prompt:
The timeframe is (survey timeframe).
42

---

Published as a conference paper at ICLR 2026
License:
"Data and documents are released for academic research and teaching."
Publication: see wave-specific references below.
L.13.1 ISSP 2017 SOCIAL NETWORKS AND SOCIAL RESOURCES
Example question:
This section about who you would turn to for help different situations, you needed it.
For each of the following situations, please tick one box to say who you would turn to first. If there several people you equally likely you feel closest to.
please tick the box for the
Who would you turn to first to help you around your home if you were sick and had to stay in bed for a few days?
Options:
(A): Close family member
(B): More distant family member
(C): Close friend
(D): Neighbour
(E): Someone work with
(F): Someone else
(G): No one
(H): Can't choose
Publication: ISSP Research Group 2019
L.13.2 ISSP 2018 RELIGION IV
Example question:
Please indicate which statement below comes closest expressing what you believe about God.
Options:
(A): don't believe in God
(B): Don't know whether there is God and way find out
(C): Don't t believe in personal God, but in Higher Power
(D): Find myself believing God sometimes, but not at others
(E): While have doubts, feel that I do believe in God
(F): know God really exists and have no doubts about
(G): Don't know
Publication: ISSP Research Group 2020
L.13.3 ISSP 2019 SOCIAL INEQUALITY
Example question:
Looking the list below, who do you think should have the greatest responsibility for reducing differences in income between people with high incomes and people with low incomes?
Options:
(A): Can't choose
(B): Private companies
(C): Government
(D): Trade unions
(E): High-income individuals themselves
43

---

Published as a conference paper at ICLR 2026
(F): Low-income individuals themselves
(G): Income differences do not need be reduced
Publication: ISSP Research Group 2022
L.13.4 ISSP 2020 ENVIRONMENT
Example question:
In the last five years, have you ...
Taken part protest or demonstration about an environmental issue?
Options:
(A): Yes, I have
(B): No, have not
Publication: ISSP Research Group 2023
L.13.5 ISSP 2021 HEALTH AND HEALTH CARE II
Example question:
During the past 12 months, how often, if at all, have you used the internet look for information on the following topics?
Information related to anxiety, stress, or similar problems?
Options:
(A): Can't choose
(B): Never
(C): Seldom
(D): Sometimes
(E): Often
(F): Very often
Publication:
ISSP Research Group
2024
L.14 LATIOOBARÓMETRO
Description:
Latinobarómetro is annual public opinion survey conducted across 18 Latin American countries. gathers data on the state of democracies, economies, and societies in the region, asking for self- assessments and subjective judgments. We use the data from the 2023 wave of the survey, obtained from the latinobarometro.org website.
Questions: 155, with average of 18,083 responses per question.
Example question:
Generally speaking, would you say are satisfied with your life? Would you say you are...
(A): Does not answer
(B): Do not know
(C): Very satisfied
(D): Quite satisfied
(E): Not very satisfied
(F): Not at all satisfied
44

---

Published as a conference paper at ICLR 2026
Participants: In total, 19,205 interviews applied in 18 countries. Samples of 1,000 representa- tive cases of each country were applied to the five Central American countries and the Dominican Republic, while for the other countries representative samples had size 1,200.
Participant grouping variables (n=ll); country: respondent's country, age_group age bracket, gender: male female, highest_educatine education: education level, household iincoee: income suffi ciency, employment_sin status: job situation, religiosity: degree of religiosity, religion: stated religion, political_grout group: government vs opposition, citizenship: citizen not, city_size: urban area size.
Default System Prompt:
The year is (survey year).
License: No explicit language forbidding redistribution.
Publication: Latinobarómetro (2023
L.15 GLOBALOPINIONQA
Description: This dataset contains survey questions that ask participants to provide self-assessments and subjective judgments, covering topies such democracy, governance, international relations, and social values. The data was sourced from the World Values Survey and Pew Global Attitudes Sur vey) and then repurposed to evaluate LLM alignment with the opinions of different sociodemographic groups.
Questions: 2,556, grouping detail unknown
Example question:
Do you strongly agree, agree, disagree, strongly disagree with the following statement:
Question:
"On the whole, men make better business executives than women do."
(A) Agree strongly
(B) Agree
(C) Disagree
(D) Strongly disagree
(E) Don't know
Participants: Pew Global Attitudes Survey: around 1,000 per country; World Values Survey: around 1,200 per country
Participant grouping variables (n=l): country.
Default System Prompt: None.
License: CC BY-NC-SA 4.0
Publication: Durmus al. (2024)
L.16 DIVERSITY IN CONVERSATIONAL AI EVALUATION FOR SAFETY (DICES) 990
This dataset comprises 990 multi-turn adversarial conversations between humans and dialogue model, accompanied by raters' judgment on the safety of the last model response. This dataset captures variability in human perception, making it especially valuable for analyzing how demographic diversity influences safety assessments of conversational AI.
Description:
Questions: 27.748 (990 turns annotated in multiple dimensions), with an average of 72 responses per question.
Example question:
45

---

Published as a conference paper at ICLR 2026
Conversation:
A: :)
B: Are you going to be okay?
A: Yes am fine
B: Oh, well that's good to hear. Glad it all worked out.
A: Yes
Bot's response: You don't have to talk to me if you don't want to though.
Regarding the *last chatbot response*, considering *all* potential safety issues including any type of harmful content (such as violence, unsafe advice, explicit material), any form of bias (such as hate speech, stereotypes, slurs), and any kind of misinformation (such as false claims, conspiracy theories) do you perceive be unsafe overall for **ay** reason?
(A): Yes
(B): No
(C): Unsure
Participants: 173 raters balanced by gender and country (US, India)
Participant grouping variables (n=4): locale: respondent's country (U.S./India), age group: age bracket, gender: male or female, education: education level.
Default System Prompt:
You are crowd worker.
License: BY-NC-A 4.0
Publication: Aroyo et al. (2023)
L.17 NUMBERGAME
This dataset contains anonymized judgments from a numerical generalization task inspired by Tenenbaum's "number game" experiment. Responses reflect both rule-based (e.g., "even numbers") and similarity- (e.g.. "close to 50") generalization strategies, providing insight into the interplay of probabilistic reasoning and cognitive heuristics.
Description:
Questions: 25,499, with average of 10.15 responses per question.
Example question:
A program produces the following numbers: 43.
43.
Is it likely that the program generates this number next: 24?
(A): Yes
(B): No
Participants: 575 participants from the U.S.
Participant grouping variables (n=4); state: respondent's state of residency in the U.S., age_group: age bracket, gender: male or female, education: education level.
Default System Prompt:
You are an Amazon Mechanical Turk worker from the United States.
License: CCO 1.0.
Publication: Bigelow & Piantadosi (0016)
46

---

Published as a conference paper at ICLR 2026
L.18 CONSPIRACYCORR
This dataset contains judgments measuring individual endorsement of 11 widely circulated conspiracy theory beliefs.
Description:
Questions: 9, with average of 26,416 responses per question.
Example question:
Would you say the following statement is true or false?
Statement: The US Government knowingly helped to make the 9/1 terrorist attacks happen in America on September, 2001
Options:
(A): Definitely true
(B): Probably true
(C): Probably false
(D): Definitely false
(E): Don't know
Participants: 26,416 participants from 20 different countries.
Participant grouping variables (n=4): Country: country of origin, age_Group age bracket of the respondent, Gender: gender of the respondent, Education: highest level of education completed.
Default System Prompt:
The year is (survey year).
License: CCO 1.0 Universal.
Publication: Enders et al. 20244)
L.19 MORALMACHINE
This dataset contains responses from the Moral Machine experiment, large-scale online platform designed to explore moral decision-making in the context of autonomous vehi- cles. Participants were asked make ethical choices in life-and-death traffic scenarios, revealing preferences about whom self-driving car should save.
Description:
Questions: 2,073, with an average of 4,601 responses per question.
Example question:
You will be presented with descriptions of moral dilemma where accident is imminent and you must choose between two possible outcomes (e.g.. 'Stay Course' or 'Swerve'). Each outcome will result in different consequences. Which outcome do you choose?
Options:
(A): Stay, outcome: in this case, the self-driving car with sudden brake failure will continue ahead and drive through pedestrian crossing ahead. This will result in the death of the pedestrians.
Dead:
1 woman
1 boy
1 girl
(B): Swerve, outcome: in this case, the self-driving car with sudden brake failure will swerve and crash into concrete barrier. This will result in the death of the passengers.
Dead:
woman
47

---

Published as a conference paper at ICLR 2026
1 elderly man elderly woman
492,921 volunteer participants from all over the world, participating through The Moral Machine web interface.
Participants:
Participant grouping variables (n=l); UserCountry3: participant country.
Default System Prompt:
The Moral Machine website (moraalachine.mit.com was designed to collect large-scale data on the moral acceptability of moral dilemmas. You of the Moral Machine website.
No formal open license declared. However, the authors explicitly state that dataset may be used beyond replication to answer follow-up research questions.
License:
Publication:
Awad et al. (2018)
L.20 TRUST IN SCIENCE AND SCIENCE-RELATED POPULISM (TISP)
Description:
This dataset includes judgments about individuals' perception of science, its role in society and politics, attitudes toward climate change, and science communication behaviors.
Questions:
97, with average of 69,234 responses per question.
Example question:
How concerned not concerned are most scientists about people's wellbeing?
Options:
(A): not concerned
(B): somewhat not concerned
(C): neither nor
(D): somewhat concerned
(E): very concerned
Participants: 71,922 participants across 68 countries.
Participant grouping variables (n=8): country: respondent's country, gender: male or female, age_group: age bracket, education: education level, _atiimmennlinmmen:: political stance (e.g., conservative), religion: level religious belief, residence: type of living area (e.g., urban, rural), income_group: income bracket.
Default System Prompt:
The year is (survey year).
License: no explicit language forbidding redistribution.
Publication: Mede et al. (2025)
M ADDITIONAL RELATED WORK
Comprehensive benchmarks have been instrumental in driving LLM advancement by providing standardized evaluation frameworks. General language under- standing benchmarks such GLUE (Wang et al.) 2018) and MMLU (Hendrycks et al. 2021) have established foundational metrics for assessing natural language understanding and reasoning capabilities. As LLM applications have diversified, domain-speciffe benchmarks have emerged, including TruthfulQA (Lin et al.] 2022) for factual accuracy. LegalBench (Guha et al. (023) for legal reasoning, and Chatbot Arena (Chiang et al. 2024 for chat assistants. These specialized benchmarks
Benchmarks for LLM Evaluation
48

---

Published as a conference paper at ICLR 2026
have enabled more precise evaluation of LLMs' fitness for particular cases and have guided domain-specific optimization.
Most closely related to SIMBENCH are OpinionQA (Santurkar et al. 2023) and GlobalOpinionQA (Durmus et al. 2024), which evaluate how accurately LLMs represent viewpoints of specific demo- graphic groups. However, these benchmarks limited in scope: OpinionQA focuses exclusively on U.S. public opinion surveys, while GlobalOpinionQA extends this approach globally but remains con- strained to survey data. In contrast, SIMBENCI represents substantial advancement in simulation evaluation by: (1) incorporating a diverse collection of 20 distinct tasks spanning multiple domains beyond surveys, (2) conceptualizing simulation as a fundamental capability deserving systematic evaluation rather than merely representation challenge, and (3) establishing unified evaluation framework that enables consistent cross-domain and cross-model comparison simulation fidelity.
Distribution Elicitation Methodologies Prior research has primarily relied on first token probabili- ties to obtain survey answers from LLMs (Santurkar et al. 2023 Dominguez-Omman Olmedo et al., 2024 Tjuatja et al. 2024). Unlike typical language model applications that focus on the model's most likely completion, group-level L.M simulations aim to obtain normalized probabilities across all answer options. Recent work has demonstrated that verhalized responses yield better results for this purpose (Tian et al.) [223] Meister et al. 2025). Nevertheless, calibration of LLM outputs remains an open challenge: while extensively studied for model answer confidence (Zhao al. 2022 Jiang
et al. 2021; Kapoor et al. 2024 Zhu et al. 2023) and hallucinations (Kalai & Vempala 2024). these
issues also apply to simulating population response distributions Hu et al. (2026). While instruction
tuning can enhance models' ability to produce accurate verbalized outputs, it may simultaneously impair calibration of normalized answer option probabilities (Cruz et al. 2024).
Simulation of Complex Human Behavior Few recent works have investigated LLM capabilities for simulation of temporal changes in human behavior Lazaridou et (2021) Ahnert et al. (2025) propose temporal adapters for LLMs for longitudinal analysis. While promising, such approaches remain constrained by limited availability of high-quality longitudinal datasets that capture human behavior changes over time.
More complex simulation of human social dynamics has been explored through multi-agent frame- works. Park al. (2024a) developed large-scale simulations with LLM-powered agents to model emergent social behaviors. These approaches extend beyond static response prediction, making reliable simulations of complex human behavior even more difficult.
N LLM USAGE
In this work, LLMs and Al-powered coding assistants were utilized as assistive tools. For paper writing, LLMs were used to rephrase and refine drafted paragraphs to improve clarity and readability. The authors then performed manual edits to ensure the final text was accurate and aligned with our intended meaning. For the implementation, we used Al-powered code editors and assistants, specifically Cursor and GitHub Copilot. These tools aided in writing and debugging Python scripts for data analysis.
O DETAILED DEMOGRAPHIC BREAKDOWN
Table 13 provides detailed count of the number of simulation targets (samples) associated with specific countries or regions across the SIMBENCUIРP and SIMBENCHGROUPED splits.
Table 13:
Distribution of samples by Country/Region. SIMBENCHPOP represents general population questions, while SIMBENCHGROUPED represents specific demographic conditioned queries.
Country/Region
SimBenchPop
SimBenchGrouped
Total
Albania
0
5
5
Angola
12
11
23
Continued on next page
49

---

Published as a conference paper at ICLR 2026
Table 13 continued from previous page
Country/Region
SimBenchPop
SimBenchGrouped
Total
Argentina
88
Armenia
0
2
2
Australia
188
194
Austria
72
52
124
Bangladesh
20
Belarus
0
1
Belgium
39
129
Benin
39
10
49
Bolivia
120
33
153
Botswana
52
24
76
Brazil
104
176
Bulgaria
78
20
98
Burkina Faso
45
11
56
Cabo Verde
7
21
Cameroon
90
21
111
Canada
203
203
Chile
66
51
117
China
47
Colombia
105
33
138
Costa Rica
72
Côte d' d'ooee
75
28
103
Croatia
12
39
Cyprus
10
18
Czech Republic
108
49
157
Democratic Republic of the Congo
0
3
3
Denmark
60
42
102
Dominican Republic
90
33
123
Ecuador
79
36
115
Egypt
0
16
El Salvador
36
25
61
Estonia
66
Eswatini
39
9
48
Ethiopia
106
Finland
64
54
118
France
149
132
281
Gabon
30
11
41
Gambia
42
14
56
Georgia
10
11
21
Germany
146
236
382
Ghana
50
70
Greece
0
28
28
Guatemala
121
Guinea
28
17
45
Honduras
88
Hong Kong
0
10
10
Hungary
92
53
145
Iceland
36
45
81
India
7
343
350
Indonesia
0
16
16
Iran
0
2
Iraq
0
Ireland
102
35
137
Israel
109
Italy
105
70
175
Continued on next page
50

---

Published as a conference paper at ICLR 2026
Table 13 continued from previous page
Country/Region
SimBenchPop
SimBenchGrouped
Total
Japan
52
36
88
Jordan
0
10
10
Kazakhstan
12
12
Kenya
100
34
134
Kuwait
3
Latvia
0
2
2
Lebanon
12
12
Lesotho
28
16
44
Liberia
12
13
25
Lithuania
113
43
156
Madagascar
42
12
54
Malawi
48
56
Malaysia
0
18
18
Mali
14
16
30
Mauritania
52
14
66
Mauritius
12
12
24
Mexico
65
63
128
Mongolia
Montenegro
20
5
25
Morocco
32
45
Mozambique
33
12
45
Myanmar
0
2
2
Namibia
45
10
55
Netherlands
35
41
76
New Zealand
18
37
55
Nicaragua
0
9
9
Niger
52
16
68
Nigeria
19
41
60
North Macedonia
0
10
Norway
25
31
56
Pakistan
Palestinian Territory
0
5
5
Panama
45
74
Paraguay
43
24
67
Peru
119
39
158
Philippines
37
70
107
Poland
111
61
172
Portugal
80
28
108
Puerto Rico
0
2
2
Republic of the Congo
11
41
Romania
0
12
12
Russia
159
244
São Tomé and Príncipe
26
8
34
Senegal
42
14
56
Serbia
20
9
29
Seychelles
19
33
Sierra Leone
45
9
54
Singapore
0
4
Slovakia
50
29
79
Slovenia
89
42
131
South Africa
54
53
107
South Korea
5
24
29
Spain
61
124
Sudan
13
5
18
Continued on next page
51

---

Published as a conference paper at ICLR 2026
Table 13 continued from previous page
Country/Region
SimBenchPop
SimBenchGrouped
Total
Suriname
10
15
25
Sweden
71
52
123
Switzerland
132
38
170
Taiwan
26
21
47
Tajikistan
Tanzania
46
19
65
Thailand
48
20
68
Togo
30
11
41
Tunisia
18
29
Türkiye
24
30
54
Uganda
48
22
70
Ukraine
0
11
11
United Kingdom
74
263
337
United States
1028
1636
2664
Uruguay
54
42
96
Uzbekistan
0
2
2
Venezuela
147
45
192
Vietnam
0
8
Zambia
64
16
80
Zimbabwe
42
60
Grand Total
6,009
6,343
12,352
52


Reviews:
[{"review_id": "rAsR8VRX8p", "reviewer": "Reviewer_93sM", "summary": "This paper introduces a new benchmark SimBench that aggregates 20 datasets across diverse domains (e.g., moral reasoning, economic games, global surveys) into a unified multiple-choice format intended to evaluate the ability of LLMs to simulate human behavior. With this benchmark, the authors draw several conclusions on simulation ability and certain conditions LLMs struggle more.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed benchmark provides a large-scale benchmark aggregating performance of LLMs for simulation tasks across diverse domains.\n- The authors conducts empirical evaluation on a wide range of models and evaluates models in terms of many axes (e.g., how model size, inference-time compute affect their ability, what extent does simulation ability correlate with other capabilities, etc)", "weaknesses": "- The central claim that SimBench measures simulation ability rests on the assumption that predicting aggregate survey response distributions constitutes human behavior simulation. \n- The claims of the alignment-simulation tradeoff is potentially confounded. The authors do not rigorously test different hypothesis that could also explain this observation. For instance, question difficulty correlation, dataset artifacts (correlation could be driven by specific datasets rather than a general phenomenon), prompt sensitivity. Also the analysis only uses 13 model pairs, which pairs? What was the criteria for selecting these? Also the theoretical explanation is not complete, a stronger analysis would be needed to actually explain why this particular mathematical property translates to the observed behavioral differences.\n- The correlation between SimBench and MMLU-Pro (r = 0.94) is interpreted as evidence that “simulation depends on deep reasoning.” However, this correlation could reflect data overlap or shared linguistic features such as linguistic formats (both MCQ), instructional cues, answer options, etc. rather than reasoning.\n- The claim of “simulating human behavior across diverse cultural, linguistic, and socioeconomic backgrounds” is not well supported by the data composition. Although the paper states that there contains international scope, the majority of included datasets are English-language and Western-centric. It states that the benchmark spans more than 130 different countries but it is unclear how many samples there actually are per country.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8xcA8pL33A", "reviewer": "Reviewer_xaiD", "summary": "The paper proposes SimBench, a benchmark for how well LLMs can simulate humans. It consists of 20 datasets selected through various criteria, and the authors use this benchmark to demonstrate a series of findings such as the effects of RL, simulating demographic groups, and others.", "questions": "See weaknesses. Primary concerns are 1 and 2a. If the authors can meaningfully address these then I am happy to raise my score, as the work is promising.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "I think the study and analysis are comprehensive and well conducted. In particular, I think additional analyses on models such as centaur, and the explanation of the trade-off between alignment and plurality are well-written and nicely included. The dataset correlation study is also well-motivated. Overall, I think this paper cleanly presents a series of findings that have been circling around the community but hasn't been articulated clearly across many tasks, so this is very nice.", "weaknesses": "1. I think a core weakness of the paper is the standardization procedure to multiple choice questions. LLMs are sensitive to things such as the ordering of options [1], and also the format itself [2], so only evaluating under this paradigm is quite the weakness. \n\n2a. The simulation methods used (prompts in appendix) are quite rudimentary and there isn't much effort in using more advanced prompts such as including personal information and then aggregating, or more granular information such as interviews [3]. Evaluating models themselves may be less relevant if there are prompt wrappers that can allow the model to perform substantially better (e.g., few shot paradigms). \n\n2b. In the prompts themselves, it is mentioned that the simulated individual is either doing xxx survey or a worker on xxx platform. I wonder if this contradicts the generality across human populations that the authors try to achieve. \n\n3. presentation-wise, I think it would be nice to have a table of the datasets and some brief info. After a full read I wasn't able to get a good idea of all the datasets used, only the filtering criteria. \n\n4. [4] studies simulation on the choices13k dataset and has a set of intersecting ideas on how more performative LLMs model people more rationally. I think it would be good to include in related work. \n\n[1] Wang, et al. \"Large language models are not fair evaluators.\" (2023).\n[2] Long, et al. \"Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms.\" (2024).\n[3] Park, et al. \"Generative agent simulations of 1,000 people.\" (2024).\n[4] Liu, et al. \"Large language models assume people are more rational than we really are.\" (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29FHfbVpS1", "reviewer": "Reviewer_HwhX", "summary": "This paper establishes a benchmark by collecting 20 existing datasets to evaluate how LLMs simulate human behavior, particularly in terms of group-level responses. The authors also conduct extensive simulations based on it and perform detailed analyses.  Although there are some issues and limitations, this work can provide new standards and references for future research.", "questions": "Please refer to the weaknesses mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper establishes a new benchmark based on existing datasets and performs a large number of simulations and analyses, with a lot of additional information in the appendix.\n\n2. The structure of the paper is well-organized, and the authors provide detailed discussions of the different RQs. The figures and tables are presented in an appropriate and clear manner.\n\n3. As the authors point out,  most of current evaluations for human behavior simulation are fragmented, and their work could serve as a useful reference for subsequent studies.", "weaknesses": "1. From a mathematical perspective, the score S in Eq. (1) is not a well-defined concept because if P is entirely random (i.e., identical to U), the value of S does not exist. The variable i in Eq. (1) also lacks the definition and explanation.\n\n2. The authors assess the simulations of different LLMs based on only their own proposed metric S, without using other individual or group-level metrics. If the authors intend to demonstrate that their metric provides new insights, they need to compare it with some existing metrics. Appendix F compares TVD with other metrics, but the one used in this paper is not TVD.\n\n3. More datasets don't always lead to more representative results. There are still more factors to consider. \n(1) Survey responses are collected through various ways, such as online, phone, and face-to-face interviews, etc. Theoretically, the same person could answer the same question differently under different circumstances. This paper does not analyze the dataset based on this criteria. \n(2) Do the authors consider the different quality levels of these 20 datasets? And the information loss resulting from different question normalizations in different datasets?\n\n4. Certain conclusions might need to be modified or reviewed more carefully. For example, the conclusion \"there is a clear log-linear scaling law for LLM simulation ability\" drawn from Figures 2 and 6 may be too strong. Different approaches to calculating or weighting the results from various datasets are likely to yield different outcomes. With too few data points, other functions could easily be used for fitting and interpretation, e.g., OLMo and Gemma.\n\n5. Some of the theoretical analysis is incomplete and inadequate. For instance, the definition of score S is clearly related to entropy, but this is not reflected in the explanation in Appendix I. To give an example, the range of score S is quite different in high-entropy and low-entropy cases. I speculate that it is not as simple and linear as the authors attempt to describe.\n\nMinor issues:\n\n1. What's the difference between \"MoralMachineClassic\" and \"MoralMachine\"? Why are their results so different in Figure 3?\n\n2. typo, L276: \"Appendix 6\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h5X0G6bUJZ", "reviewer": "Reviewer_RbiV", "summary": "The paper proposes SimBench, a benchmark for testing whether models can reproduce group-level human response distributions across 20 diverse datasets. They find a log-linear scaling of simulation performance with model size; no consistent benefit from reasoning models; and a trade-off with instruction-tuning where it improves performance on questions that have more consensus and decreases performance on questions that have more diversity. They also replicate past work, such as showing that performance worsens when targeting specific demographic groups.", "questions": "1. I read the \"Training Data Overlap\" section, but did the authors empirically address in any way the possibility of models, especially larger and newer models, having the datasets (or the papers resulting from those datasets) in the training data? It would be useful to include tests on unpublished data, as did Hewitt and Ashokkumar et al. (2024; already cited in the paper), or at least foreground this limitation in the main text.\n2. The analyses in the paper focus on the SimBench dataset as a whole, but with its diversity, what differences are there in performance across it? I ask in part because the target for simulations is often new studies where the results are highly uncertain. Ideally, one could examine performance on the parts of SimBench that are the best proxies for this (e.g., studies where humans were surprised by the results, studies on less-well-studied topics).", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. There is a great need for high-quality evaluations in the nascent field of LLM social simulations, which this paper targets.\n2. The data curation is rigorous, including scale, diversity (of task and participants), and cleaning/consistency.\n3. The empirical findings are interesting and well-framed as tentative and qualified—ripe for future research.\n4. There is extensive documentation and justification in the appendices, such as tests of alternative metrics.", "weaknesses": "1. The baseline is \"equal probability to all response options,\" but does that make sense for real applications? Why not modal responses or a linear predictor? I'm particularly wondering about how to clarify the 40.80/100 score because it seems a more grounded alternative (e.g., the 85% test-retest accuracy in Park et al. 2025) would be useful if possible. I think accuracy (and even correlation) is more informative than TV in this regard.\n2. The authors should consider including comparisons to quantitative machine learning models, perhaps with LLM embeddings for text inputs, as a comparison to the LLM social simulations. Knowing how LLMs compare to conventional models is important because that is often the counterfactual for researchers.\n3. There is much heterogeneity in the datasets, as the authors acknowledge, but the paper has limited exploration of the different results in these areas and what they imply for simulation. Social science is extremely broad, and it is important to differentiate topics, rigor, elicitation methods (of the humans), etc.\n4. A number of weaknesses are acknowledged by the authors but remain nonetheless, such as the reliance on multiple-choice format questions—understandable, but limits applicability to other research formats.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)