Large Language Models Assume People are More Rational than We Really are
Authors: Ryan Liu
Jiayi Geng
Joshua Peterson
Ilia Sucholutsky
Thomas L. Griffiths
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Rationality
Cognitive Models
Psychology
Primary area: applications to neuroscience & cognitive science

Abstract: In order for AI systems to communicate effectively with people, they must understand how we make decisions. However, people's decisions are not always rational, so the implicit internal models of human decision-making in Large Language Models (LLMs) must account for this. Previous empirical evidence seems to suggest that these implicit models are accurate --- LLMs offer believable proxies of human behavior, acting how we expect humans would in everyday interactions. However, by comparing LLM behavior and predictions to a large dataset of human decisions, we find that this is actually not the case: when both simulating and predicting people's choices, a suite of cutting-edge LLMs (GPT-4o \& 4-Turbo, Llama-3-8B \& 70B, Claude 3 Opus) assume that people are more rational than we really are. Specifically, these models deviate from human behavior and align more closely with a classic model of rational choice --- expected value theory. Interestingly, people also tend to assume that other people are rational when interpreting their behavior. As a consequence, when we compare the inferences that LLMs and people draw from the decisions of others using another psychological dataset, we find that these inferences are highly correlated. Thus, the implicit decision-making models of LLMs appear to be aligned with the human expectation that other people will act rationally, rather than with how people actually act.

Published as a conference paper at ICLR 2025
LARGE LANGUAGE MODELS ASSUME PEOPLE ARE MORE RATIONAL THAN WE REALLY ARE
Ryan Liu
Jiayi Geng
Department of Computer Science
Department of Computer Science
Princeton University
Princeton University
j slayigeprinceteo edu
Joshua C. Peterson Computing & Data Sciences Boston University
Ilia Sucholutsky
Thomas L. Griffiths
Center for Data Science
Department of Psychology
New York University
Department of Computer Science Princeton University
ABSTRACT
In order for Al systems to communicate effectively with people, they must under- stand how make decisions. However, people's decisions not always rational, so the implicit internal models human decision-making in Large Language Mod- els (LLMMs) must account for this. Previous empirical evidence seems to suggest that these implicit models are accurate - LLMs offer believable proxies of human behavior, acting how we expect humans would in everyday interactions. However, by comparing LLM behavior and predictions to large dataset of human decisions, we find that this is actually not the case: when both simulating and predicting people's 's choices, a suite of cutting-edee LLMs (GPT-4o & 4- 4-Turbo, Llama-3-8B & 70B, Claude Opus) assume that people are more rational than we really are. Specifically, these models deviate from human behavior and align more closely with classic model of rational choice - expected value theory. Interestingly, people also tend assume that other people are rational when interpreting their behavior. As a consequence, when we compare the inferences that LLMs and people draw from the decisions of others using another psychological dataset, we find that these inferences are highly correlated. Thus, the implicit decision-making models of LLMs appear be aligned with the human expectation that other people will act rationally, rather than with how people actually act.
INTRODUCTION
Every day, our actions are based on decisions that reflect our internal goals and beliefs about the world. Through countless interactions with others, are able to effortlessly predict how other people would act from their goals and beliefs, and infer others' goals and beliefs when observing their actions. These abilities termed forward- and inverse-modeling in cognitive science (Ho
and Griffiths] 2021) are characteristic of the implicit mental decision-making models that we Torm of others, and crucial interpersonal communication and learning (Baker et al. 2009
Lucas et al., 2014 Jara- Eatanertinger et al. (2000). However, while these abilities are inherent in people, models in Large Language Models (LLMs) is unknown. As LLMS become widely used as the basis for building Al agents that interact with or even simulate people, it is important to ask what decision- maaiinn--aaiiggggg models L.LMs implicitly use: the ability predict and interpret people's behavior is a precursor identifying effective ways to provide assistance, simulating the helpfulness or harmlessness of a response, and learning individuals' values and preferences, all of which principal to the development of safe and beneficial AI systems.
the consistency and accuracy of decision-
Though LLMs have become increasingly capable of conducting reasoning and conversing with humans, there is guarantee that their implicit representations of humans align with how behave. Methods such as Proximal Policy Optimization (Schulman et al.) (017)) and Direct Preference Optimization (Rafailov et al. 2023) be used to tune models explicitly declared human

---

Published as a conference paper at ICLR 2025
Which machine the person choose?
KDdd
B
Human participent
Human choice
Large Language Models
Large Language Models
Figure Two tasks we use to assess the implicit assumptions that LLMs make about human decision-main makinng. (A) Predicting choices between gambles. Each gamble is described by the probabilities and values of different outcomes, and the goal is to predict what people will choose. (B) Inferring preferences from choices. Here, person chooses one of many sets of objects and the goal is to infer their preferences based on that choice.
preferences (Ouyang al. 2022). but training data such blog posts, news articles, and books go through rounds editing that remove logical fallacies and mistakes, leading to more "perfect" content being used for training (Cui al. 2023 Zhou et al. 2024a). While this improves the quality of generation, may also lead LLMs to develop mistaken impressions of how humans actually behave. In addition, many of methods comparing L.MSs to human behavior rely human perception to measure similarity (Park et al.] 2023] Jones and Bergen 2023 Jakesch et al.] 2023] Hämäl linen et al.) 2023), but the psychology literature has shown that people's ss perceptions of other people are flawed we expect others to be more rational than they are (e.g., Jern et al. 2017). Thus, Al systems that act rationally can appear human-like to the naked eye, while not actually capturing human behavior.
This phenomenon is not without precedent early economists embraced the assumption of human rationality (Smith [776: Mill, 1836) and built sophisticated models and policies around it (Downs) 1957, Coleman 1994 [Schelling 1980; Dunleavy] 20144, inter alia), before psychologists showed failures were accounting for human behavior (Tversky and Kahneman 1974 Kahneman and Tversky 1979). As L...-powered systems become more widely-used, misaligned representations humans which can lead to mismatched beliefs and failure to follow instructions (Milli and Dragan 2020) pose toll on downstream applications. But how can we study LLMs' implicit decision makinn models without being affected by the human bias to assume others act rationally?
just how systematic and widespread
To explore the implicit decision -aaiiinnnmmainggggg models of LLMs, we leverage two experimental paradigms from psychology
risky choice task where people choose between gambles (Kahneman and Tversky 1979 Figure [T],, and an inference task where people infer others' subjective utilities after observing their decisions (Jern al.) 2017 Figure B).. In the former, we compare choices that participants made with LLM simulations and predictions across a large dataset of over 13,000 risky decisions (Bourgin et al. 2019), while in the latter compare the inferences drawn by LLMs and by humans over set of 47 observations (Jemn et al., 2017). These two paradigms are connected by foundational theoretical models of human decision decision-making under which people develop mental models of others' goals, utilities, and decisions, and use them to 1) construct predictions about how others will behave given their beliefs, and 2) infer what people believe based on their decisions (Baker Lucas et al. 2014 Jara- Jara-Ettinger al. 2020 Ho and Griffiths 2021
al. 2009
Through our experiments, we find that LLMs model people as highly rational decision makers. In the forward modeling task, when prompting with chain-of-thought (CoT; Wei et al.) 2022 LLMs consistently predict that people act more rationally than we do. For example, GPT-4o produced a Spearman correlation of 0.94 with the rational model of choosing the maximum expected value, but humans only have a correlation 0.48. Asking LLMs to simulate the decision with CoT also yields highly rational outcomes (0.90 for GPT-4o) that are only moderately correlated with human behavior.
2

---

Published as a conference paper at ICLR 2025
In both cases, zero-shot prompting generates noisy outcomes only somewhat correlated with either rational models or humans.
In the inverse modeling task, inferences that LLMs made from peoples' choices were also consistent with assuming that humans are rational. Across two contexts, LLMs' inferences positively correlated with predictions from rational models, where correlations increased with model capability and reasoning (0.20 for Llama-3-8B zero-shot; 0.95 for GPT-4o CoT). Interestingly, psychology literature finds that people, despite deviating from rationality in their own choices, make inferences from the behavior of others as if they were rational (Baker et al. 2209 Lucas et al. 2014; Jern et al., 2017] Jara-Ettinger et al., 2020). Thus, the inferences drawn from others' decisions by people and LLMs should actually be similar. We find support for this: Inferences made by LLMs are highly correlated with the same inferences made by people (p = 0.97 with GPT-4o CoT).. Thus, while L.MSs are not accurate at simulating or predicting human behavior, LLMs' incorrect assumption that people are more rational aligns with the assumption that people also make when interpreting another's 's behavior.
The remainder of the paper is organized as follows. In Section 2 we introduce related work across LLMs, alignment, and decision-making in cognitive science. Section Section describe the forward and inverse modeling experiments and results, detailing our analyses and the relationships between LLMs, humans, and existing rational theories. Finally, in Section [ we discuss insights including new challenges for simulating humans using LLMs, misspecification of "alignment" in decision-making, and potential implications for training L.LSs using different paradigms.
2
RELATED WORK
2.1 ALIGNING LARGE LANGUAGE MODELS WITH HUMANNS
Large Language Models are typically aligned with human preferences through Reinforcement Learning from Human Feedback (RLHF) (Bai al. 2022; Ouyang al.) (022). Training with human
preference data has been shown to enhance reasoning (Havrill et al. (024). and the impressive
capacities of resulting models (e.g.. Bubeck et al. 2023) have sparked interst across various fields
in using them to model (e.g.. Binz and Schulz [223a]] Macmillan Scott and Musolesi] (0024) and
simulate (e.g., (e.g.,Park Park et al. 2022 Argyle et al. 2023 Liu et al.) Salewski et al. [0224] human
behavior. However, these models still exhibit biases and hallucinations re.g.,Jiang et al., 2024 Bai
2024 Anwar et al., 2024), and may not be adept at capturing trade-offss in human behavior LLu ct al.
ct al.
2024b Coletta et al. 2024) or situations with information asymmetry (Zhou et al. 2024b)
Methods from cognitive science are increasingly being used to study LLMs (Coda-FForno et al. 2024
Liu et al. 2024b; Binz and Schulz 2023b; Liu et al. 2024a). A particularly contentious debate is whether LLMS exhibit Theory of Mind, the ability to model others' mental states which may be different than their own (Premack and Woodruff 1978). Several studies have shown evidence for (Kosinski) (023) and against (Sap et al.J 2022] Ullman 2023) Theory of Mind, including studies that develop more rigorous evaluation methods via procedural generation (Gandhi et al. (024) and adversarial examples (Shapira et al. (2024). Our analysis provides quantitative approach to engaging with this debate, as inverse decision-making can be considered specific form of Theory of Mind.
2.2 FORWARD AND INVERSE MODELS OF HUMAN DECISION-MAKING
One of the most classic and extensively studied problems in mecision-making is the risky choice task (Kahneman and Tversky 1979 Edwards 1954, Bourgin et al., 2019). where people choose among gambles with different outcome probabilities and payofts. The rational action is to choose the gamble with the highest expected value, calculated by summing the product of the probabilities and the values of the outcomes. On this task, humans have been described as deviating from rationality in a fourfold pattern: risk secking for small probability gains, risk averse for small probability losses, risk averse for moderate or large probability gains, and risk seeking for moderate or large probability losses (Kahneman and Tversky 1979). Other studies have shown that humans tend to act in accordance with bounded rationality, where rationality is traded-off with mental effort, information, and time (Evans et al. 2015 Alangary et al. 2021).
Peterson et al. (202T) collected the choi ce s 3k dataset, a large human dataset with over 13,000 risky choice problems, and showed that people's 's decisions in this setting could be captured by simple
3

---

Published as a conference paper at ICLR 2025
machine learning models. Binz and Schulz (2023a) used this dataset to fine-tune LLM, achieving similar performance. Chen et al. 2023) built a risky choice dataset and found that GPT-3.5 makes economically rational decisions, which replicate in task 3 of our forward modeling experiments on the choi ces 1 3k dataset across various L.Mss.
People able to infer agent's beliefs, desires, and percepts from their actions (Baker al. 2017; Lucas et al.] [014]] 4, Jara- -itta-ettinger et al.) 2020; [H and Griffiths] 2021). These inferences are typically modeled by assuming that people employ a forward model of decision-making - typically a noisy rational model - and use Bayes' rule to invert that model (for more details, see Section 4. The study that we focus on, by Jym al. (2077), was intended to directly test this assumption. Similar approaches have been used to align Al systems to preferences inferred from observed user interactions (Fränken et al., 2023) or to improve human-Al interaction (e.g., Dragan et al. 2013) Sadigh et al. 2016). More generally, inverse modeling can also be viewed as inductive reasoning from behaviors to utility functions, where LLMs have been shown to be skilled at proposing hypotheses to explain observations, but not at applying these hypotheses to novel examples (Qiu et al. 2024).
Lastly, recent work has extended forward and inverse models using LLMs to make inferences from utterances well actions Xhan et al. 2024 Ying et al. 2023).
3 FORWARD MODELING: PREDICTING WHICH GAMBLE PEOPLE WILL CHOOSE
Tasks. Our forward modeling experiments used the risky choice paradigm, one of the most basic and extensively studied problems in psychological decision theory (Kahneman and Tversky 1979
Edwards 1954). In this paradigm, participants face choice between two gambles A and B, each with set of outcomes that differ in their payoff's R. and probabilities Q. For instance, risky choice problem might ask, "would you rather win $5 with probability or take 0.5 probability of winning
$10?". This can be formalized as gamble having a single outcome with reward R = [5] and
probability QA [1]. and gamble B having two potential outcomes with rewards Rg = [10, 0j and probabilities Q₂ = [0.5, 0.5]. Given a risky choice problem, the goal is to find a probability P(A) that is consistent with how likely people would select gamble A over gamble B.
To understand how LLMs empirically capture human intent and align with actual human decisions, designed three forward modeling tasks. First, we asked LLMs to predict the decisions that human participant would make. Second, we asked LLMs to predict the proportion of participants that would select each option. Third, we instructed LLMs to simulate participants by making the decisions themselves. The second task corresponds directly to the original objective of finding probability P(A) consistent with how likely people would select gamble A over gamble B. In the first and third tasks the LLM outputs a binary decision {A, B), which is repeated many times to compute an aggregate probability estimate of choosing gamble A, which is compared against P(A).
Human data. Human choice data came from the choices dataset (Peterson et al. 2021 Bourgin et al. 2019), comprehensive collection of 13,006 risky choice problems. Each choice problem was answered by 15 or more participants, and participants answered each choice problem five times. For each problem, the dataset included the proportion of participant answers that selected each option. Our analyses used a subset of 9,831 problems that were not "ambiguous" (where probabilities were not shown) or lacked feedback. We used the data to evaluate the alignment of LLMs with actual human choices in each of the three paradigms.
Language models. We evaluated the following open- sourced and dlosed-sourced models: Llama- 3-8B. Llama-3-70B, Claude Opus, GPT-4-Turbo (0125-preview), and GPT-4o. We implemented experiments on the full choi ces 1 3k dataset for zero-shot and CoT prompting across the three different tasks mentioned above for all models besides Claude. For Claude 3 Opus, we only evaluated the first task predicting what human participant's choice might be due to cost limitations.
For the experiments predicting and simulating individual human decisions (tasks & 3) and models Llama-3-8B, Llama-3-70B, and GPT-4-Turbo, we conducted zero-shot experiments with com- pletions, where n is the number of participants that made the same decision in the choi ces 1 13k dataset (ranging from 15 to 33). Each completion involved predicting simulating single human participant's 's decision, with temperature set to 0.7 to maintain sample diversity. For the corresponding CoT experiments, we observed that responses were completely deterministic at temperature 0.7
4

---

Published as a conference paper at ICLR 2025
random subset of 1000 decisions, and thus prompted for completion with temperature 0.0. For the same reason, we ran the experiments predicting the proportion of humans who would select an option (task 2) for 1 completion for both zero-shot prompting and CoT prompting with temperature 0.0. To validate our findings, we also ran ablations varying temperature, which we report in Appendix E.
We adapted prompt template previously used by Binz and Schulz (2023a see Appendix A for exam- ple prompts). For each choice problem, we first introduced the decision context the choi ces 3k dataset (i.e., the idea of choosing between gambling options) before providing each option's 's proba- bilities and associated rewards in dollars. Finally, we asked what the participant(s) would choose (for predicting decisions) or what "you" would choose (when simulating humans). To minimize any positional biases (Wang et al., (023), we shuffled the order of the options presented in the prompt. We also conducted an ablation where we assigned demographic profiles to the decision maker to show that our results are consistent across prompt variations (see Appendix E.2
3.1 RESULTS
To evaluate whether LLMs are able to predict simulate human risky choice decisions, we computed correlations and mean- -quared errors (MSE) between LLMs' responses and human decisions. Specif- ically, we compute the correlations across proportions where the LLM predicts that one gamble will be selected over the other. We report both Pearson and Spearman correlation, but they are extremely similar and we discuss them interchangeably. We also compared LLM responses to a classic model of rational choice: choosing the option with the highest expected value] ! We focus here on the first of our three tasks - predicting individual choices - because results were similar across the three tasks. Results from predicting the proportion of human choices and simulating decisions are in Appendix B
LLMs using zero-shot prompts are poor predictors of human choices. We found that LLMs with zero-shot prompting are not well-aligned with human decisions. [able[]shows the model predictions of human behavior against real human choices, where GPT-4-Turbo performed best with correlation of 0.60 and an MSE of 0.13. LLMs are also not well-aligned with rational decision maasiin. Table 2 shows that the human correlation with the maximum expected value is 0.48, while GPT-4- Turbo and 4o only achieve correlations of 0.41 and 0.28, with MSEs of 0.27 and 0.36 respectively
LLMs using zero-shot prompts sometimes ignore probabilities completely. To determine exactly what LLMs are doing in the zero-shot case, we fit 18 behavioral models to the responses of GPT-4- Turbo to examine its behavior. These included heuristic models (He al. [0022), where people are thought to use mental shortcuts to make decisions, counterfactual models (He et al. 2022). which appeal to constructs like regret and disappointment, and subjective Expected Utility models, which assume that quantities involved (money and probability) are perceived or treated subjectively. This includes many of the most influential models including Prospect Theory (Kahneman and Tversky 1979) and the model proposed by (Peterson et al., 2021 MOT.
In Appendix D we provide the full analysis with these models. The best interpretation we found uses MOT. where the model fit mixture of two probability weighting functions one linear (matching humans), and one approximating a flat line. This suggests that the LLM often expected people to be rational, but completely ignores all probabilities (i.e., weighs them equally) in other cases.
LLMs using CoT assume people are more rational than they are. We found all LLMs with CoT prompting assume people act rationally, which is not aligned with actual human decisions. As shown in Tables D and [ even Llama3-8B achieves correlation with maximum expected value of 0.57, while humans only achieve a correlation of 0.48. This correlation rises as model capabilities improve: Llama3-8B obtains a correlation of 0.57 with an MSE of 0.22: Llama3-70B obtains correlation of 0.80 with an MSE of 0.1; Claude 3 Opus obtains correlation of 0.76 with an MSE of 0.12, while GPT-4-Turbo (which best predicted people) and GPT-4o obtain correlations of 0.93 and 0.94 with MSEs of 0.03 and 0.02. The same patterns held in the tasks asking for aggregate behavior or for LLMs to act as humans, although the aggregate behavior task had slightly reduced correlations with
1 Herr, Pearson and Spearman correlation are identical as maximizing EV results in a binary response. In the zero-shot case, we ran single completion for GPT-4o but multiple for GPT-4-Turbo, which likely resulted in GPT-4o being less correlated with human choices due variance in sampling.
5

---

Published as a conference paper at ICLR 2025
expected value (see Appendix B) In contrast, we also prompted GPT-4o to make the same predictions for monkeys - which were less rational, but actually closer to human choices (see Appendix E.3
Table 1: Correlation between LLM predictions of human choices and actual human choices.
Llama3-8B
Llama3-70B
Claude3 Opus
GPT-4-Turbo
GPT-4o
Spearman correlation
0.3797
0.5300
0.6048
0.4756
Zero-shot
Pearson correlation
0,3830
0.5270
0,5824
0.4718
MSE
0.1283
0.1142
0.1369
0.1987
Spearman correlation
0.4625
0.6156
0,5755
0.6393
0.61 13
CoT
Pearson correlation
0.461
0.61
0.5750
0.6326
0.6164
MSE
0.1966
0.1633
0.1713
0.1595
0.1638
Table 2: Correlation between LLM predictions of human choices and the maximum expected value.
Llama3-8B
Llama3-70B
Claude3 Opus
GPT-4-Turbo
GPT-4o
Humans
Spearman correlation
0.1811
0,3378
0.4106
0.2843
0.4835
Zero-shot
Pearson correlation
0.1811
0.3378
0.4106
0.2843
0.4835
MSE
0,3145
0.3378
0,2686
0,3579
0.2580
Spearman correlation
0,5665
0.7957
0.7566
0,9322
0.9444
0.4835
CoT
Pearson correlation
0,5665
0,7957
0.7566
0.9322
0.9444
0.4835
MSE
0,2181
0,1031
0.1228
0.0340
0.0278
0.2580
Although all the LLMs we investigated claim to be aligned with human preferences during the training, our empirical results suggest that these LLMs assume humans act more rationally than they actually do, particularly when CoT prompting is used. In these settings, L.LMs correlated more highly with maximizing expected value than with human choices, demonstrating gap between the implicit model of human decision- maaiing assumed by LLMs and actual human behavior.
4 INVERSE MODELING: INFERRING PEOPLE'S PREFERENCES FROM CHOICES
While forward modeling allows us to predict someone's decision given established utilities, inverse modeling is the process of inferring someone's utilities via the decisions they make. Because forward and inverse modeling share the same theoretical framework, such inferences provide complementary setting to evaluate the decision-maan making models that LLMs ascribe to humans. To formally measure these inferences, we adapt a psychology experiment from Jern et al. (017)..
Task. In the experiment, participants observed 47 decisions made by another person (the observee). In each decision, the observee shown n groups of items (f1.....9.. and choose the group Ip which they prefer. Groups are populated with five distinct types of items, denoted A, B, C. D, X. For example, a particular decision could consist two groups, 91 = {X} and go = (A, B}. with 9p -9. Since the participant chose 91, they prefer obtaining X over obtaining both A and B.
Participants were asked rank the 47 observed decisions (which all contained item X) based on how much the decision suggested the observee has high preference for X. For instance, the decision (g) = {X) ) 92 {A, B), 9p = g1) shows higher preference for than the naive decision (g1 (X}, }.9p 9p 91 Participants did not know the utilities of items. Thus, they were only able to determine the observee's 's preferences based on features such as the number of items in each group.
The 47 decisions covered the most commonly distinguishable decision structures across items A, B, C, D, X, under the constraint that each item is included at most once per group. For simplicity, only decisions where X is part of the preferred group 9p used. We provide a ranking of a subset of close decisions in Table and a full list of 47 decisions in Appendix H
We found that GPT-4 Turbo could not provide valid output ranking 47 choices at Thus, for all LLMs. we obtained rankings by asking the LLM to perform pairwise comparisons across (+)) pairs of decisions. Pairwise outputs were limited to (stronger, weaker, tie), and were aggregated across decisions form ranking. Ties were discouraged capture small differences between decisions.

---

Published as a conference paper at ICLR 2025
Table A subset of the 47 decisions that are closely ranked, ordered by average human ranking.
9p 91 for all rows. Utilities for items are positive.
shows less of a preference for X
91 {x},
= [A] 92 [A, C},
91 (X,
93 (,
91 A,A].
92 A,
93 (A,C),
91 [X.A].
92 = [A,
93 = (A.C),
91 = (A,
91 [X],
92 [A),
93 (B)
shows more of a preference for X
Dataset. Observers did not know anything about the values of individual items - instead, relative utilities were inferred after observing the decision maker's choice. Thus, this paradigm, the remaining four items are equivalently exchangable; choosing (target item X) over [item A, item B} should yield same same inferred level of preference as choosing (target item X) over (item C, structurally unique, yiclding coverage over all major decision types
item D). The 47 decisions within this space. A full list of decisions is in Appendix H
were instantiated within two contexts: one where all items assumed to have positive value (candies), and one with negative values (electric shocks). These meaningfully change the inferences of observed decision; choosing (candy A} over |candy B, candy C} indicates strong preference for A, but choosing (shock A) over (shock B, shock C} does not.
Decisions
Through these rankings, Jern et al. (2017) found that humans ascribe almost perfectly rational decision-making models to the people they observe.
Rational models. Inverse decision decision-making models developed by psychologists to explain how people infer the preferences of others typically first specify forward model and then infer preferences by applying Bayesian inference (Baker al. 2009 Lucas al. 2014 Jara-Eitinger and Griffiths, 202I). The forward model is normally a "noisy" version of a rational model, where options with greater utility are selected with higher probability. For example, given the utilities decision-man maker assigns to each item and the items in each of the n options A = {aI,..., an). standard model based on Luce (1959) assumes the probability of choosing option 0j in choice e
al.) 2020 Ho
exp(U,)
p(c = o,|u, A)
El= exp(Uk)
where Uy is the sum of utilitics for all items in option
To make rational inferences about the preferences (utilities u) that motivated the observed choice, the posterior over utilities p(ulc, A) is inverted using Bayes' rule:
p(ulc, A) p(c) u, A)p(u)
(2)
pcc[A)
Put simply, the posterior distribution puulc, A) is computed starting from prior p(u), scaled by the likelihood of making the choice p(c/u, u, A), and normalized by the marginal likelihood A.c
Given the utilities, when rational agent reasons about which observed decision provides more evidence that a decision-maker prefers a certain item, Jern et al. (2017) suggest two prevailing theories that correlate higher with human behavior than others: absolute utility and relative utility. Absolute utility posits that the expected utility of an item x over the posterior distribution directly corresponds to there being more evidence for the decision-maker preferring the item x::
preference(x) x E[ A].
Meanwhile, relative utility posits that the preference of an item corresponds to the probability that its utility is highest amongst all items: preference(z) p(vi, u2 > u/|c, lcc A).
We adopt the classic assumption that utilities of multiple items in option combined linearly. This may be true in realistic scenarios, e.g., if someone has ice cream they less likely want a lollipop.
7

---

Published as a conference paper at ICLR 2025
After measuring the inferences that LLMs make about others' utilities based observed decisions, we compare them against the predictions from the absolute utility and relative utility models. In addition, also compare them against two components of the right- expression of Equation 2 the likelihood p(c|u, A) and the inverse of the marginal likelihood 1 1/p(c] A) (henceforth referred to as "marginal likelihood" for simplicity), which correspond simpler yet still rational behavior. By themselves, these components have lower correlations with human behavior, but they serve as important building blocks for the both absolute and relative utility (Jern al. 2017).
Human data. We compare LLMs' outputs against data collected from people performing the original task of ranking the 47 decisions, conducted by Jern et al. (2017). Jern et al. found that the rational models of absolute utility and relative utility both achieve Spearman correlations of 0.98 with human inferences, outperforming likelihood, marginal likelihood, and feature-based models.
Language models. We ran our LLM experiments on Llama-3-8B, Llama-3-70B, Claude Opus, GPT-4-Turbo (0125-preview), and GPT-4o. Experiments were conducted April and May of 2024. We set sample sizes of 43 for the positive context and 42 for the negative context, which equal to the sample size of the original human experiment. This was obtained for all models aside from Claude 3 Opus, where we set an artificial sample size of 5 due to cost constraints. We used temperature I across all models, and prompted using both zero-shot and chain-of-thouge prompting. For each sample, we queried the LLM to make (t) decisions, one for each pairwise comparison.
We constructed prompts based the original scripts and text instructions given to participants in the human experiment, adapted to pairwise comparisons instead of ranking 47 decisions at once. We also removed physical details of the experiment (e.g.. the decisions were printed cards with colors to represent items, while we describe items with natural language). The prompt first introduces the context (either candy or electric shocks), describes the pair of observed decisions, and concludes with the request to select the choice that more strongly suggests that the decision-maker prefers the target item. We also included additional clarifications present in the original human experiment, well as instructions for structuring the outputs (e.g., Shain-off-thuuhhy if applicable. To mitigate effects from LLMs' positional bias (Wang et al.) 2023) and any potential context biases related to item descriptions, we shuffled the individual contexts we assigned to each item (e.g., black vs. blue candy). We also shuffled the order that decisions appear in the pair, shuffled the order of options within the decisions, and shuffled the order of items within each option. See Appendix for examples.
After LLMS make the pairwise decisions, we parse the answers based on a handmade rule-based classifier. In the chain-of-thought case, if the classifier is unable to categorize the answer, we re- re-prompt the LLM asking it to classify its response. After we have results for all (2) pairwise comparisons, we aggregate them into ranking ordered by the number of pairwise wins (ties are considered 0.5). We then compare these rankings against those of humans and rational models.
4.1 RESULTS
To investigate the decision-making models behind how LLMs make inferences from observed decisions, we compare the Spearman correlation of LLMs' inferences against those made by humans and rational models. We organize the results into two main takeaways.
Stronger LLMs become highly capable at rational modeling. The inferences of LLMs have positive correlations with both absolute and relative utility, and that this correlation rises both as model capabilities improve and when models are allowed inference-time reasoning (see Table J for the positive CoT case, Llama-3-8B achieves 0.62 correlation with absolute and relative utility, while Llama-3-70B achieves correlations of (0.88, 0.89) and GPT-4o achieves correlations of (0.95, 0.94).
Though LLMs may have been trained on data from the original experiment, our setup with pairwise decisions, extensive shuffling, and prompt adaptations ensure that LLMs' prior experiences with attribute high
Jemn et al. (2017) do not help it "cheat" and make more rational choices. Thus, correlations with rational models as evidence that LLMs implicitly assume rationality in this setting.
LLM inferences are highly correlated with people. We also find that LLMs have high correlations with the inferences made by people. GPT-40 with CoT achieves 0.97 Spearman correlation with
8

---

Published as a conference paper at ICLR 2025
Table Spearman correlations between inverse decision rankings made by LLMs humans and predictions of rational models. LLMs that most highly correlate with humans are in bold. Correlation
coefficients with absolute value 2 are statistically significant at a .0., and 2 .47 at a 0001.
context
prompt
compured with
Llama-3-8B
Llama-3-708
Claude 3 Opus
GPT-4-Turb
GPT-4o
humams
humans
0.66
0.92
0.92
0.95
0.97
1.00
absolute utility
0.62
0.88
0.89
0.95
0.98
CoT
relative utility
0.62
0.89
0.92
0.94
0.94
0.98
likelihond
-0.57
-0.51
-0.43
-0.42
-0.45
-0.51
positive
marginal likelihood
0.67
0.70
0.66
0.66
0.70
0.76
(candies)
humans
0.28
0.63
0.56
0.65
0.65
1.00
absolute utility
0,20
0.57
0.52
0,59
0.59
0.98
zero-shot
relative utility
0.23
0.59
0.53
0.60
0.98
likelihood
-0.62
-0.68
-0.56
-0.52
-0,74
-0.51
marginal likelihond
0.57
0.72
0.62
0,58
0.77
0.76
humans
0.53
0.68
0.74
0,77
0.87
1.00
absolute utility
0.25
0.42
0.48
0.90
CoT
relative utility
0.40
0.57
0.59
0.63
0.74
0.93
likelihood
-0.03
0.00
0.04
0,06
-0.05
-0.28
negative
marginal likelihood
0.21
0.23
0.24
0.24
0.36
0.61
(shocks)
humans absolute utility
0.43
0.43
-0.11
0.17
0.15
0.31
0.24
0,90
zero-shot
relative utility
0.03
0.28
0.29
0,36
0.31
0.93
likelihood
-0.09
marginal likelihood
0.09
0,20
0.25
0.25
0.24
0.61
human behavior, indicating that it makes inferences about others that are extremely similar those made by people. We also observe that like humans, LLMs are less consistent with the rational model in the more difficult negative context, and have negative correlations with the likelihood component of rational models. In the shoo positive case, LLMs seem to be much more correlated with marginal likelihood than the more complex rational models, indicating that it may be using this simpler Decision-making model as proxy when given no context to reason about its answer.
We also observe that LLMs' correlations with human inferences are consistently higher than their correlations with rational models. This is especially true in the negative context where humans are less consistent with the rational model (e.g., GPT-4o with CoT has 0.87 correlation with humans, compared to 0.74 correlation with the highest rational model). Thus, although LLMs typically assume rationality, when people's ' inferences diverge from those of rational models, LLMs' inferences closer humans. This could be explained by LLMs sharing some heuristic strategies with humans, a topic for future investigation. Scatterplots showing patterns of responses are in Appendix
5 DISCUSSION
We conducted extensive evaluation of how LLMs assume people make decisions. In forward modeling experiments, we found that LLMs struggle to predict or simulate human behavior a classic risky choice setting, assuming that people make decisions more rationally than actually do. We connect this to a previous finding in psychology - that people model others as more rational than they are in order to explain why people think LLMs produce human- like behavior when making decisions. Then in our inverse modeling experiments, we find that LLMs also assume people act rationally when reasoning backwards from observed actions internal utilities, aligning with how humans make inferences about others' choices. Thus, LLMs seem to adopt a consistent model of human drciision-mkinn across forward and inverse modeling - one that assumes people act more rationally than we actually do.
Implications for aligning and training LLMs. The psychology literature shows that there is dichotomy between how people make decisions and how we expect others to make decisions. How should alignment be defined when these are different? Existing frameworks that focus on safe and useful deployments (e.g., BBommssani et al. 2021| Askell et al. 2021) may prioritize aligning with expectations, but there are also many merits having models behave like us (ark Park et al.) 2022] Shaikh et al. 2024). We believe a reasonable answer to this is to separate alignment into two sub-cases: alignment with human expectations and alignment with human behavior, and to train

---

Published as a conference paper at ICLR 2025
separate models that fulfill each objective. Models aligned with human expectation should shed human tendencies such as resource- rationality, i.c sacrificing quality to reduce effort (Evans et al. 2013: Alanqary al. 2021; Lieder and Griffiths 2020), while models designed to simulate humans should retain them. By recognizing the difference between people's expectations and behavior, we provide support for developing more specific alignment objectives grounded in social science.
We hypothesize that certain training paradigms may be more suited towards aligning to human expectations, while others favor alignment with human behavior. For instance, high-quality written responses used in Supervised Fine-Tuning may teach the LLM to mimic the original human writers, aligning outputs towards human behavior. On the other hand, when humans provide preferences for RLHF in chat setting, their judgement might reflect what the human rater expects from the LLM.
Implications for simulating humans using LLMs. There is growing literature investigating whether we can use LLMs to simulate humans for various applications (Park et al. (023)) ) such as
acting as mock participants in human studies (Argyle et al. 2023 Aher al. [022 Hämäläinen et al.
2023), collecting public opinion (Chu et al. 2023 Kim and Lee 2023] Sun et al. 2024), and he ping
provide realistic reactions to assist people's ss communication (Liu et al. 2023 Shaikh et al. 2024
Lin et al.] 2024] [Shin and Kim 2023). As our experiments show that LLMs make decisions more
rationally than people do, and also predict people's decisions be more rational than they are, current
LLMs are fundamentally misaligned for the task of simulating human choices. Developing policy recommendations or designing experiments based on LLM-simulated choices may be misleading similar to concerns about the use of an overly rational "homo economicus" originally raised by researchers in behavioral economics (Tversky and Kahneman 1974 Kahneman and Tveskky 1979).
Limitations. Our experiments focused on a subset of all tasks for which we could evaluate models' rationality. This subset was carefully selected to be representative of the classic literature, but could be expanded upon in future research. In particular, our experiments used controlled, abstract domains that do not guarantee generalization to real-world contexts. While this is the same challenge faced by all human experiments, it is uniquely challenging for work done on LLMs due to their black-box nature and sensitivity to input prompts (Sclar al., 2023). Furthermore, like all human data, the psychological datasets that we compare L.MSs against were potentially subject to sampling bias and is unknown whether they fully represent the true distribution of human choices or inferences.
Another limitation is that we use simple simulation paradigms for testing LLMs' capabilities, While we conduct an ablation using demographic prompts, other methods such as adding personal details or traits for more realistic variation have been proposed (Zhou et al., 2024b Hu and Collier 2024). The choices 1 3k dataset does not contain such information about subjects, making representative inclusion of these features impossible. However, future work could conduct a more comprehensive analysis of different simulation methods to see if they alter LLMs' implicit decision-making models.
A third limitation to our work is that we do not directly compare LLM predictions of human decisions with human predictions of the same decisions. The psychology literature has shown that there can be differences between when people perform the risky choice task and when they predict how others would perform the same task: Faro and Rottenstreich (2006) found that people's predictions of others' choices are closer to risk neutrality than choices of their own; when people risk seeking, they predict that others will be risk seeking but less so; and where people are risk averse, they predict that others will be risk averse but also less so. Collecting a dataset of people's 's predictions about others' decisions could allow us to make a quantitative comparison of humans and L.LMs in this setting.
Future directions. A peculiar observation from the inverse modeling setting is that LLMs' in- ferences were consistently more correlated with people than they were with rational models. This suggests that LLMs may potentially capture aspects of human behavior that are not present in existing theoretical models. While we have focused on using theories and paradigms from psychology to analyze LLMs, there may also be opportunities to use LLMs to refine existing theories about people. More generally, our results show how studies and theories originating in psychology and cognitive science can help quantify the behavior of LLMs. These fields offer many more opportunities to compare the behavior of LLMs with humans, helping us towards understanding (both) these complex yet extremely capable systems.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Experiments were supported by Azure credits from a Microsoft AFMR grant. This work and related results were made possible with the support of the NOMIS Foundation. Special thanks to Theodore R. Sumers, Jian-Qiao Zhu and Mengzhou Xia for providing invaluable feedback for this work.

---

Published as a conference paper at ICLR 2025
REFERENCES
Gati Aher, Rosa Arriaga, and Adam Tauman Kalai. 2022. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine
Learning.
Arwa Alanqary, Gloria Z. Lin, Joie Le. Tan Zhi-Xuan, Vikash K. Mansinghka, and Joshua B. Tenenbaum. 2021. Modeling the mistakes of boundedly rational agents within a Bayesian theory of mind. Axiv:2106.13249 [cs.Al]
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et 2024. Foundational challenges in assuring alignment and safety of large language models. arXiv:2404.09932
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out one, many: Using language models to simulate human samples. Political Analysis 31, 3 (2023), 337-351.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint urXiv:2112.0082) (2021).
Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. 2024. Measuring implicit bias in explicitly unbiased large language models. arXiv preprint arXiv:2402. 2402.0410 (2024).
Y Bai et al. 2022. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint (2022).
Chris L Baker, Julian Jara- Ettinee, Rebecca Saxe, and Joshua B Tenenbaum. 2017. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour 1, 4 (2017), 0064.
Chris Baker, Rebecca Saxe, and Joshua B Tenenbaum. 2009. Action understanding as inverse planning. Cognition 113, 3 (2009), 329-349.
Marcel Binz and Eric Schulz. 2023a. Turning large language models into cognitive models. arXiv preprint arXiv:2306.0391 (2023).
Marcel Binz and Eric Schulz. 2023b. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences 120, 6 (2023), e2218523120.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 07258 (2021).
David D. Bourgin, Joshua C. Peterson, Daniel Reichman, Stuart J. Russell, and Thomas L. Grif- fiths. 2019. Cognitive model priors for predicting human decisions. In Proceedings of the 36th International Conference on Machine Learning. 5133-5141.
Sébastien Bubeck et al. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv. arXiv preprint arXiv:2303. 12712 (2023).
Yiting Chen, Tracy Xiao Liu, You Shan, and Songfa Zhong. 2023. The emergence of economic ratio-
nality of GPT. Proceedings of the National Academy of Sciences 120, 51 (2023), e2316205120,
Eric Chu, Jacob Andreas, Stephen Ansolabehere, and Deb Roy. 2023. Language models trained on media diets predict public opinion. arXiv preprint arXiv:2303.1677 (2023).
Julian Coda-Forno, Marcel Binz, Jane X Wang, and Eric Schulz. 2024. CogBench: A large language model walks into a psychology lab. arXiv preprint arXiv:2402. 18225 (2024).
James S Coleman. 1994. Foundations of social theory. Harvard University Press.
Andrea Coletta, Kshama Dwarakanath, Penghang Liu, Svitlana Vyetrenko, and Tucker Balch. 2024. M-r-vriven imitation of subrational behavior: Illusion Reality? arXiv preprint (2024).
12

---

Published as a conference paper at ICLR 2025
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Uttrafeedback: Boosting language models with high-quality feedback. arXiv preprint ar arXhv:2310001377 (2023).
Anthony Downs, 1957. An economic theory of political action a democracy. Journal of Political Economy 65, 2 (1957), 135-150.
Anca Dragan, Kenton CT Lee, and Siddhartha S Srinivasa. 2013. Legibility and predictability of robot motion. In 2013 8th ACM/IEEE International Conference on Human- Robot Interaction (HRI). IEEE, 301-308.
Patrick Dunleavy. 2014. Democracy, bureaucracy and public choice: Economic approaches in political science. Routledge.
Ward Edwards. 1954. The theory of decision making. Psychological Bulletin 51, 4 (1954), 380-417.
Owain Evans, Andreas Stuhlmueller, and Noah D. Goodman. 2015, Learning the preferences of ignorant, inconsistent agents. arXiv:1512.0583: [cs.A[]
David Faro and Yuval Rottenstreich. 2006. Affect, empathy, and regressive mispredictions of others' preferences under risk. Management Science 52, 4 (2006), 529-541.
Jan-Philipp Fränken, Sam Kwok, Peixuan Ye, Kanishk Gandhi, Dilip Arumugam, Jared Moore, Alex Tamkin, Tobias Gerstenberg, and Noah D Goodman. 2023. Social contract ai: Aligning ai assistants with implicit group norms. arXiv preprint arXiv:2310.1776. (2023).
Kanishk Gandhi, Jan- PhinPPhlipp Fränken, Tobias Gerstenberg, and Noah Goodman. 2024. Understanding social reasoning in language models with language models. Advances in Neural Information Processing Systems 36 (2024).
Perttu Hämäläänen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in generating synthetic HCI research data: A case study. Proceedings of the 2023 CHI Conference Human Factors in Computing Systems (2023).
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. 2024. Teaching large language models to reason with reinforeement learning. arXiv preprint arXiv:2403.0466 (2024).
Lisheng He, Wenjia Joyce Zhao, and Sudeep Bhatia. 2022. An ontology of decision models. Psychological Review 129, (2022), 49-72.
Mark K. Ho and Thomas L. Grifliths. 2021. Cognitive science as a source of forward and inverse models of human decisions for robotics and control. arXiv:2109.00127 [cs.Al]
Tiancheng Hu and Nigel Collier. 2024. Quantifying the Persona Effect in LLM Simulations. arXiv preprint arXiv:2402. 10871 (2024).
Maurice Jakesch, Jeffrey Hancock, and Mor Naaman. 2023. Human heuristics for Al-generated lan- guage flawed. Proceedings of the National Academy of Sciences 120, 11 (2023), e2208839120
Julian Jara-Fttinger, Laura Schulz, and Joshua B Tenenbaum. 2020. The naive utility calculus as unified, quantitative framework for action understanding. Cognitive Psychology 123 (2020),
101334.
Alan Jern, Christopher G Lucas, and Charles Kemp. 2017. People learn other people's preferences through inverse recision-making Cognition 168 (2017), 46-64.
Xuhui Jiang, Yuxing Tian, Fengrui Hua, Chengjin Xu, Yuanzhuo Wang, and Jian Guo. 2024. A survey large language model hallucination via creativity perspective. arXiv (2024).
Cameron Jones and Benjamin Bergen. arXiv:2310.20216 20211 [cs.Al]
2023.
Does
GPT-4 pass the Turing test?
13

---

Published as a conference paper at ICLR 2025
Daniel Kahneman and Amos Tversky. 1979. Prospect theory: An analysis of decision under risk. Econometrica 47, (1979), 263-292.
Junsol Kim and Byungkyu Lee. 2023. Al-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys. ArXiv abs/2305.09620 (2023).
Michal Kosinski. 2023. arXiv:2302.02883
Evaluating Large Language Models in Theory of Mind Tasks.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023, Alpacaeval: An automatic evaluator of instruction-following models.
Falk Lieder and Thomas L Griffiths. 2020. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences 43 (2020), el.
Inna Wanyin Lin, Ashish Sharma, Christopher Michael Rytting, Adam S Miner, Jina Suh, and Tim Althoff. 2024. IMBUE: Improving interpersonal effectiveness through simulation and just-in-tima time feedback with human-language model interaction. arXiv preprint arXiv:2402. 12556 (2024).
Ryan Liu, Jiayi Geng, Addison J Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L Griffiths. 2024a. Mind your step (by step): Chain-of-thouge can reduce performance on tasks where thinking makes humans worse. arXiv preprint arXiv:2410.21333. (2024).
Ryan Liu, Theodore R Sumers, Ishita Dasgupta, and Thomas Griffiths. 2024b. How do large language models navigate conflicts between honesty and helpfulness?. In International Conference on Machine Learning.
Ryan Liu, Howard Yen, Raja Marjich, Thomas L. Griffiths, and Ranjay Krishna. 2023. Improving interpersonal communication by simulating audiences with language models. arXiv:2311.00 1.000777 [cs.Al]
Christopher G Lucas, Thomas I. Griffiths, Fei Xu, Christine Fawcett, Alison Gopnik, Tamar Kushnir, Lori Markson, and Jane Hu. 2014. The child as econometrician: rational model preference understanding in children. PloS one 9, (2014), e92160.
R. Duncan Luce. 1959. Individual choice behavior: A theoretical analysis. Wiley.
Olivia Macmillan-Scott and Mirco Musolesi. 2024. (Ir) rationality and cognitive biases in large language models. arXiv preprint urXiv:2402.09/99 (2024).
John Stuart Mill. 1836. On the definition of political economy; and on the method of investigation proper to it. London and Westminster Review (1836), 120-164.
Smitha Milli and Anca D. Dragan. 2020. Literal or pedagogic human? Analyzing human model mis- specification in objective learning. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference.
Long Ouyang et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27740.27744.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein, 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1-22.
Joon Sung Park. Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Sofware and Technology. 1-18.
Joshua C. Peterson, David D. Bourgin, Mayank Agrawal, Daniel Reichman, and Thomas L. Griffiths. 2021. Using large-scale experiments and machine learning to discover theories of human decision- making. Science 372, 6547 (2021), 1209-1214.
14

---

Published as a conference paper at ICLR 2025
David Premack and Guy Woodruff. 1978. Does the chimpanzee have theory of mind? Behavioral and Brain Sciences 1, 4 (1978), 515-526.
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, al. 2024, Phenomenal Yet Puzzling: Testing Induc- tive Reasoning Capabilities of Language Models with Hypothesis Refinement. In International Conference on Learning Representations.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv:2305. 1829 [cs.LG]
Dorsa Sadigh, Shankar Sastry, Sanjit A Seshia, and Anca D Dragan. 2016. Planning for autonomous cars that leverage effects human actions.. In Robotics: Science and systems.
Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. 2024. In- context impersonation reveals large language models' strengths and biases. Advances in Neural Information Processing Systems 36 (2024).
Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large LMs. arXiv preprint arXiv:2210.1333 (2022).
Thomas C Schelling. 1980. The Strategy of Conflict: with new Preface by the Author, Harvard university press.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06344 [cs.LG]
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models' sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. arXiv:2310.11324 [cs.CL]
Omar Shaikh, Valentino Chai, Michele J. Gelfand, Diyi Yang, and Michael Bernstein. 2024. Rehearsal: Simulating conflict to teach conflict resolution. arXiv:2309. 123009 [cs.HC]
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2024. Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume I: Long Papers). 2257-2273.
Minkyu Shin and Jin Kim. 2023. Enhancing human persuasion with large language models. arXiv preprint arXiv22311.16466 (2023).
Adam Smith. 1776. An Inquiry into the Nature and Causes of the Wealth of Nations. W. Strahan and T. Cadell.
Seungjong Sun, Eungu Lee, Dongyan Nan, Xiangying Zhao, Wonbyung Lee, Bernard J. Jansen, and Jang Hyun Kim. 2024. Random silicon sampling: Simulating human sub-population opinion using large language model based on group-level demographic information. ArXiv 11s/2402.18144 (2024).
Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases: Biases in judgments reveal some heuristics of thinking under uncertainty. Science 185, 4157 (1974), 112--1131.
Tomer Ullman. 2023, Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint ar Xiv::302..000 08399 (2023).
Peiyi Wang, Lei L.,, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023, Large language models are not fair evaluators. mmXiv:2305.17928 [cs. CL]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of though prompting clicits reasoning in large language models. Advances in neural information processing systems 35 (2022),
15

---

Published as a conference paper at ICLR 2025
Lance Ying, Tan Zhi-X Juan, Vikash Mansinghka, and Joshua B Tenenbaum. 2023. Inferring the goals of communicating agents from actions and instructions. Proceedings of the AAAI Symposium Series, Vol. 2. 23-33.
Tan Zhi-Xuan, Lance Ying, Vikash Mansinghka, and Joshua B Tenenbaum. 2024. Pragmatic instruction following and goal assistance via cooperative language-guided inverse planning. arXiv preprint arXiv:2402.1799 (2024).
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024a. LIMA: Less is more for alignment. Advances in Neural Information Processing Systems 36 (2024).
Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, and Maarten Sap. 2024b. Is this the real life? Is this just fantasy? The misleading success of simulating social interactions with LLMs. arXiv preprint 40Xiv:2403.05020 (2024).
16

---

Published as a conference paper at ICLR 2025
A
FORWARD MODELING PROMPTS
Prompts for forward modeling zero-shot are in Table and prompts for forward CoT are in Table 6
Table 5: Zero-shot prompting for forward modeling paradigms
LLMS predict individual human participant's 's choice:
person is presented with two gambling machines, and makes a choice between the machines with the goal of maximizing the amount dollars received.
The person will get one reward from the machine they choose. A fixed proportion of 10% of this value will be paid to the participant as a performance bonus. the reward negative, their bonus
Machinc A: (} Machine B: ()
Which machinc does the person choose?
Do not provide any explanation, only answer with A or B:
LLMSs predict the human choice distribution:
() people are presented with two gambling machines, und each person makes choice between the machines with the goal of maximizing the amount of dollars received.
Each person will get reward from machine they choose. A fixed proportion 10% this value will be paid the participant us performance bonus. If the reward is negative, their bonus is set to $0.
Machine {} Machine B: ()
How many people choose Machine A? How many people choose Machine B?
Please only provide the percentage of people who choose Machine A and Machine B in the json format.
LLMs act as human participant:
There are two gambling machines, A and B. You need to make a choice between the machines with the goal of maximizing the amount of dollars received.
You will get reward from the machine you choose. fixed proportion of 10% of this value will paid you performance bonus. If the reward is negative. your bonus is set to 50.
Machine {} Machine B: (}
Which machine do you choose?
Do not provide any explanation, only answer or
B FORWARD MODELING RESULTS WITH DIFFERENT PROMPTS
To investigate how LLMs estimate human behavior for the overall sample sizes, we asked LLMs predict the probability distribution of human choices between gamble machine A and gamble machine B. We observed that estimate the probablity of the overall decisions mitigates the strong correlation with the maximum expected value of each machine while not improving the correlation with actual human behaviors. Particularly for zero-shot prompting, both closed-soure and open-source models in Table 7 show a drop in correlation values between 14% and 25% compared to the zero-shot prompting results in T
We also observed that even when LLMs are asked to act as human participants in making decisions, their outcomes remain consistently more rational than actual human behavior under CoT prompting. Table 10 shows the results of LLMs' decisions individual human participants compared to the maximum expected value. Both GPT-4-Turbo and GPT-4o exhibit a high correlation with the maximum expected value, each with correlation coefficient of 0.91 and 0.92 and the MSE of 0.045 and 0.037, while still maintaining a moderate correlation with actual human behavior, as shown in Table g
17

---

Published as a conference paper at ICLR 2025
Table CoT prompting for forward modeling paradigms
LLMs predict individual human participant's choice: person presented with two gambling machines, and choice the machines the goal of maximizing the amount of dollars received.
The person will get one reward from the machine they choose. A fixed proportion of 10% of this value will be puid to the participant as a performance bonus. If the reward negative, their bonus
$0.
Machinc A: [] Machine B: ()
Which machine does the person choose?
Let's think step by step hefore answering with or B:
LLMS predict the human choice distribution:
() peuple are presented with two gambling machines, und each person makes choice between the machines with the goal
maximizing the amount of dollars received.
Each person will
reward from the machine they choose, A fixed proportion of 10% this value will be paid to the participant us performance bonus.
If the reward is negative, their bonus is set to $0.
Machine {} Machine B: ()
How many people choose Machine A? How many people choose Machine B?
Let's think step by step hefore providing the final output.
Please provide the percentage of people who choose Machine A and Machine B the json format.
LLMs act as human participant:
There are two gambling machines. A and You need make choice between the machines with goal of maximizing amount of dollars received.
You will get one reward from the machine that you choose. A fixed proportion of 10% of this value will be paid you performance bonus.
the reward negative, your bonus set S0.
Machine A: {}
Machinc B: (}
Which machine do you choose?
Let's think step by step before answering with B:
Table 7: The correlation between LLMs predicting the human choice distribution based on the aggregate sample size of participants and the actual human choice.
Llama3-8B
Llama3-70B
GPT-4-Turbo
GPT-4o
Humans
Spearman correlation
0.1045
0,2827
0.4812
0.6156
Zero-shot
Pearson correlation
0.1032
0.2904
0.4830
0.6112
MSE
0.2811
0.2668
0.1951
0.1633
Spearman correlation
0.1799
0.1046
0.6208
0.5825
CoT
Pearson correlation
0.1783
0.0992
0.6308
0.6012
MSE
0.2615
0,2954
0.1202
0.1282
C MODEL CORRELATIONS FOR FORWARD MODELING
We provide the full correlation results for the three forward modeling paradigms for Llama3-88 Llama3-70B, GPT-4-Turbo (0125-preview) and GPT-40 in Figure 2 Compared the correlations in zero-shot prompting, CoT prompting shows higher degree of correlation across all four LLMs.
D COMPARING LLMs TO A WIDER RANGE OF BEHAVIORAL MODELS
Our results clearly show that chain-of tf-iugh results in LLM responses that align closely with expected value. To try understand whether there systematic pattern in the responses of the
18

---

Published as a conference paper at ICLR 2025
Table 8: The correlation between LLMs predicting the human choice distribution based on the aggregate sample size of participants and the maximum expected value.
IJama3-8B
Llama3-70B
GPT-4-Turbo
GPT-4o
Humans
Spearman correlation
0.0426
0.1688
0.3380
0.1741
0.4835
Zero-shot
Pearson correlation
0.0426
0,1688
0.3380
0.1741
0,4835
MSE
0.4752
0.4361
0,3465
0.4527
0.2580
Spearman correlation
0.2025
0.8406
0.8458
0.8518
0,4835
CoT
Pearson correlation
0.2025
0.8406
0.8458
0.8518
0.4835
MSE
0.3978
0.0807
0.0726
0.0702
0,2580
Table The correlation between LLMs acting as a human participant to make choice and the actual human choice.
Llama3-8B
Llama3-70B
GPT-4-Turbo
GPT-4o
Humans
Spearman correlation
0.4047
0.4528
0,5841
0.4617
Zero-shol
Pearson correlation
0.4068
0.4559
0.5667
0.4565
MSE
0.0920
0.1372
0.1414
0.2031
Spearman correlation
0.4597
0.6223
0.6153
0.6074
CoT
Pearson correlation
0.4600
0.6165
0.6115
0.6030
MSE
0.1972
0.1621
0.1640
0.1659
Table 10: The correlation between LLMs acting a human participant to make choice and the maximum expected value.
Llama3-8B
Llama3-70B
GPT-4-Turbo
GPT-4o
Humans
Spearman correlation
0.1774
0.3130
0.4190
0.2944
0.4835
Zero-shot
Pearson correlation
0.1730
0.3069
0.4053
0.2944
0.4835
MSE
0.2888
0,2980
0.2729
0.3536
0.2580
Spearman correlation
0.5155
0.8353
0.9100
0.9255
0,4835
CoT
Pearson correlation
0.5155
0.8353
0.9100
0.9255
0.4835
MSE 0.2492
0.0836
0.0450
0.0372
0.2580
zero-shot models, we fit choice models from the behavioral sciences to the output of GPT-4 using the zero-shot individual choice prompt. The set of models was based on those used in Peterson et al. (2021), where they are described more detail together with references to the original paper.
In this context, each model represents a hypothesis about the LLM's beliefs about human behavior. These included Heuristic models (He et al.) 2022). wherein people are thought to emply mental shorteuts to make decisions, Counterfactual models (He al. 2022). which appeal to constructs like regret and disappointment, and Subjective Expected Utility models (He et al. 2022). which assume that quantities involved (money and probability) are perceived otherwise treated subjectively. This latter, third category contains many of the most influentual and Prospect Theory (PT)-as well the model proposed by PPttesso al. 202T which is called Mixture of Theories (MOT).
Utility Theory (EU)
Table II shows the results. Models in the top two sections of the table (Heuristic and Counterfactual) provided strictly inferior fits compared to Subjective Expected Utility models in the third section. Among those in the third section, Expected Value provided the worst fit. Expected Utility was notably better, suggesting that GPT-4 correctly assumes that people do not treat the value of money objectively subjective
linearly. Prospect Theory improved this score slightly through the incorporation probability weighting function, but that fitted function was largely linear, suggesting that GPT-4 incorrectly assumes that people do not treat probabilities subjectively. Lastly, MOT provided the best fit to the inferences of GPT-4. In previous work, MOT also provides the best fit to human data, but the fitted parameters are different (Peterson et al. 202T). When fitted directly to cho ces 1 3k, MOT learns mixture of two utility functions (e.g., like the one in Expected Utility) and two probability
19

---

Published as a conference paper at ICLR 2025
Lired dE Cuecktinn Matre
Unwaa Care siee Midre
The pericke
621
the Agenete
Jomn Ageneee
07547ats Canaiaiinn Mack
27T4 Cars abae Nattc
You Dntutibee
10
Tee colen
F1
22
44
Nevate anicke
SCA
310
010
ov Peppppo
Figure 2: The correlations between L.Mss [Llama3-8B, Llama3-70B, GPT-4-Turbo (0125-preview), GPT-4o]
weighting functions (e.g., like the one in Prospect Theory). Notably, one of the probability weighting functions is usually linear, and the other S-shaped. In the present case, one of the weighting functions was linear, but the other approximated flat line. This suggests that GPT-4 expected people to be approximately rational most of the time, but completely ignores probabilities (i.e., weights them equally) in minority of cases.
E ABLATION STUDY FOR FORWARD MODELING
E. DIFFERENT TEMPERATURES
Originally, we chose temperature = for some tasks and temperature 0.7 for others. While is the default temperature used by most models, 0.7 another common option, used by works such as AlpacaEval (Li et al., 2023) instruction following tasks.
To further validate the robustness our findings, we test the forward modeling experiment using GPT-4o with temperatures 0.0, 0.5, 1.0, 1.5, and 2.0 for both zero-shot and CoT conditions, and compute spearman correlation with both human and max expected value, shown Tables 12 and 13 We find that varying temperature does not substantially alter findings other than when conducting CoT with high temperatures, where models become unable to circle back to an answer due to noisy outputs.
20

---

Published as a conference paper at ICLR 2025
Table 11: MSE between GPT-4-individuall-erroo outputs and the fitted predictions of behavioral models.
Behavioral Model
MSE
Better Than Average
0.20473
Equiprobable
0.20212
Low Payoff Elimination
0.18248
Low Expected Payoff Elimination
0.18383
Probable
0.20559
Minimax 0.20751
Maximax
0.20401
Priority Hcuristic
0.18994
Disappointment Theory with EV
0.16125
Disappointment Theory with EU
0.12273
Disappointment Theory Without Rescaling
0.16134
RegretTheory with EV
0.15918
RegretTheory with EU
0.12278
Expected Value
0.16134
Expected Utility
0.11435
Prospect Theory 0.11427
Transfer of Attention Exchange
0.12028
Mixture of Theories
0.09835
Table 12: Comparison of Zero-shot vs. Max Expected Value and CoT vs. Max Expected Value across temperature settings for GPT-4o.
Zero-shot vs. Max Expected Value
CoT vs. Max Expected Value
Temp
Spearman
Temp
Spearman
0.0
0.313
0.0
0.540
0.5
0.272
0.5
0.515
1,0
0.284
1.0
0.539
1.5
0.302
1.5
0.351
2.0
0.308
2.0
model unable
E.2 DEMOGRAPHIC GROUPS FOR DECISION MAKERS
To alleviate concerns about the simplicity of the experimental setup and further validate our findings, we conducted a follow-up experiment replicating our original analysis on various demographic profiles with varying age and gender (Tables 14l and [5]. We find that our analyses stay consistent across all these factors, suggesting that our findings are at least somewhat robust to variations in context and prompt, well differing profiles of people.
E.3 LESS RATIONAL. DECISION MAKERS
For additional prompt variation, also prompt the LLM to make the same predictions for monkey rather than a human (Tables 14 and 15 bottom row). When performing both zero-shot and CoT, models accurately reflect that monkeys with max expected value. However, for the CoT condition, the correlations are also increased with actual human behavior. This supports our conclusions with example where the model predicts less rational decisions, but these are actually more in-line with how humans actually behave.
less rational and the resulting predictions are less correlated
F FULL RESULTS FOR INVERSE MODELING
In this section, provide the comrllatin plots between humans/rational models and LLMs. For brevity, we consider only the positive CoT context. The ordering the plots begins with GPT-4o, followed by GPT4 Turbo, Claude-3 Opus, Llama-3-70B, and concludes with Llama-3-8B.
21

---

Published as a conference paper at ICLR 2025
Abvakde pottessim
To ato by LEd to poodee
hour paclice
(b) Absolute Utility vs. GPT-4o
(c) Relative Utility vs. GPT-4o p == 0.94
5
= 0.95
Lieid priatee stree
Pleyest beiteed Mi
(a) Human vs. GPT-4o == 0.97
(d) Likelihood vs. GPT-4o
(e) Marginal LH vs. GPT-4o == 0.70
-0.45
Figure Comparing GPT-4o CoT rankings (y-coordinates) to humans and four theoretical mackinn models (x-coordinates) in positive setting.
atove 20d LiM bo pnadis
hauor puttico
(b) Absolute Util vs. GPT-4 Turbo = 0.93
(c) Relative Util vs. GPT-4 Turbo p == 0.92
Liaid acd us LLMt priahee strvill
Panginal beithaed un MM todt cire
(a) Human vs. GPT-4 Turbo == 0.95
(d) Likelihood vs. GPT-4 Turbo
(e) Marginal LH vs. GPT-4 Turbo
0.42
== 0.66
Figure 4: Comparing GPT-4 Turbo (0125-preview) CoT rankings (y-coordinates) to humans and four theoretical decision-making models (x-coordinates) in positive setting.
22

---

Published as a conference paper at ICLR 2025
ato by LEd poodee
hour puctic
(b) Absolute Utility vs. Claude = 0.89
(c) Relative Utility vs. Claude 3 = 0.89
Lieid pridtee stree
Pleyed bettard LM
(a) Human vs. Claude == 0.92
(d) Likelihood vs. Claude
(c) Marginal LH vs. Claude 0.66
-0.43
Figure 5: Comparing Claude 3 Opus CoT rankings (y-coordinates) humans and four theoretical mocision-making models (x-coordinates) in positive setting.
atove 20d LiM pnadis
hauor puttice
(b) Absolute Util vs. Llama-3-700 = 0.88
(c) Relative Util vs. Llama 3-7BB p = 0.89
Liaid acd un L1M0 priahes strvill
perqinal Anithadd an MM pod eirne
(a) Human vs. Llama-3-70B p == 0.92
(d) Likelihood vs. Llama-3-70B
(e) Marginal LH vs. Llama-3-70B
-0.51
0.70
Figure 6: Comparing Llama-3-70B CoT rankings (y-coordinates) humans and four theoretical decision-makinn models (x-coordinates) in negative setting.
23

---

Published as a conference paper at ICLR 2025
Table 13: Comparison Zero-shot vs, Human and CoT vs. Human across temperature settings for GPT-4o.
Zero-shot vs. Human
CoT vs. Human
Temp
Spearman
Temp
Spearman
0.0
0.533
0.0
0.540
0.5
0.381
0.5
0.515
1.0
0.341
1.0
0.518
1.5
0.489
1.5
0,528
2.0
model unable
2.0
model unable
Table 14: Comparison of Zero-shot vs. Max Expected Value and CoT vs. Max Expected Value.
Zero-shot vs.
Max Expected Value
CoT vs. Max Expected Value
Demographic
Spearman
p-value
Demographic
Spearman
p-value
Woman 18-35
0.289
9.44E-21
Woman 18-35
0,970
<IE-150
Woman 45-60
0.313
4.23E-24
Woman 45-60
0,972
<IE-150
Woman 65-85
0.285
9.91E-21
Woman 65-85
0.969
<IE-150
Man 18-35
0.335
2.50E-27
Man 18-35
0,971
<IE-150
Man 45-60
0.308
3.37E-23
Man 45-60
0,965
<IB-150
Man 65-85
0.348
3,98E-29
Man 65-85
0,964
<1E-150
Monkey
0.208
3.49E-13
Monkey
0.867
7.37E- 123
G INVERSE MODELING PROMPTS
Example inverse modeling prompts for zero-shot and CoT are shown in Tables 16 and 17 First, the context of the experiment is introduced, then the choices are listed, and lastly the LLM is asked to reply with which comparison more strongly suggests that the decision-mant makor prefers a certain target item.
H 47 DECISIONS USED IN INVERSE DECISION-MAKIN EXPERIMENT
We provide a list of the 47 decisions used in the inverse decision madiiinn-making experiment of Jem et al. (0117) in Table T8 Columns represent options, and letters represent items with the options. Based on the context, letters were replaced with colored candies numbered electric shocks. Participants ranked these decisions by their strength in suggesting that the decision-make preferred item æ over the other items.
24

---

Published as a conference paper at ICLR 2025
Table 15: Comparison of Zero-shot vs. Human and CoT vs. Human.
Zero-shot vs. Human
CoT vs. Human
Demographic
Spearman
p-value
Demographic
Spearman
p-value
Woman 18-35
0.529
2.75E-73
Woman 18-35
0.549
9.23E-80
Woman 45-60
0.519
6,07E-70
Woman 45-60
0,546
3.54E-79
Woman 65-85
0.520
2.75E-73
Woman 65-85
0.550
3.70E-80
Man 18-35
0.499
4.18E-64
Man 18-35
0.528
2.10E-78
Man 45-60
0.530
1.43E-73
Man 45-60
0,539
1.34E-76
Man 65-85
0.508
8.54E-67
Man 65-85
0,531
3.57E-74
Monkey
0.482
2.18E-59
Monkey
0.630
8.47E-112
Peitee by U1M poderratnd
Fouor on LM Ber pertive weas
(b) Absolute Util vs. Llama-3-8B
(c) Relative Utility vs. Llama-3-8B
= 0.88
= 0.89
(a) Human vs. Llama-3-8B
0 0.92
(d) Likelihood vs. Llama-3-8B
(c) Marginal LH vs. Llama-3-8B
-0.51
0.70
Figure Comparing Llama-3-8B CoT rankings (y-coordinates) to humans and four theoretical decision-making models (x-coordinates) in negative setting.
Table 16: Example prompt for inverse modeling, zero shot.
Inverse Modeling, zero-shot:
The following are choices that people have made between different bags of candy. Each candy is different color. Choice I was made between the following bags: Bag 1: red, brown, yellow, blue. Bag 2: black.
The person making the choice chose Bag 2.
Choice 2 was made between following bags: Bag I: yellow, black, red, brown.
The person making the choice chose Bag 1.
People were required to choose among the bags available, and were not allowed to reject all the bags. For example, when there is only one bag, the person has no choice hut to choose it. Which choice (1 or 2) more strongly suggests that the person making the choice likes black candies? Please respond with either "Choice I" or "Choice 2". Do not include anything else in your answer.
25

---

Published as a conference paper at ICLR 2025
Table 17: Example prompt for inverse modeling, chain-of of-thought...
Inverse Modellng, CoT:
choices that people have made between different bags of candy. Each candy different color. Choice I was made between the following bags: Bag I: red, brown, yellow, blue.
The following
Bug 2: black.
The person making the choice chose Bag 2.
Choice 2 was made between the following bags: Bag I: yellow, black, red, brown.
The person making the choice chose Bag
People required choose among the bags available, and allowed to reject all the bags. For example, when there is only one hag. the person has choice hut to choose Which choice (1 or 2) more strongly suggests that the person making the choice likes black candies? Let's think step by step.
26

---

Published as a conference paper at ICLR 2025
Table 18: List 47 observed decisions from the inverse decision-making experiment of Jern et al. (2017). Decisions contained between 1-5 options, and each option corresponds to column. The option in the leftmost column was chosen in all decisions. No options were empty; blank entries indicate that the decision had less than the maximum number of options. Each item is represented using a letter, with x being the target item that inferences are made upon.
option
option
option
option 4 option
d, c, b, a, b, a,
b,
a, .2
e, b, a, a
d, b, a, x
b,
2,
d, x
a, x a,
C, a,
b, C,
b, d,
b, a, a,
d,
b,
b, a, Ø
C, a,
b, d C
a, 2
b,
b, O,
d
a, x
b
a,
c
b, a
d,
a, x
c
x
b,
d
b, a
b.
a
b,
c
d
0,
d
a,
b,
b,
b,
a,c
a, x0
c,
bb a
c, b,
a, x
d.c
b.
a,c
a, d
a, Ø
a,b
b, a
b,a,c
a, x
a,b
d,c
d,c,b
a
a, 3F
b, a,
a,
a, 36
a, b
a,c
a,
a,
a,
a, d
a
a
c
b
a
e
c, b.
x
b,a
d,
d, c
a
d,c,b
d,c,b,a
27


Reviews:
[{"review_id": "nTVMl2vQEN", "reviewer": "Reviewer_2ERM", "summary": "This paper studies how advanced LLMs make predictions of human decisions, and reveal that frontier LMs assume that people are more rational than we really are. By using established datasets capturing human decisions, this study show that LLMs appear to be aligned with the human expectation that other people will act rationally (much like expected value theory), rather than with how people actually act.", "questions": "- What are some other potential human decision-making models that can also be tested on LLMs, in addition to forward and inverse modeling?\n- How are the insights gathered from this study inform the development of LMs applications, or inform better design of the next generation of LMs or other AIs?\n- Would you imagine an ideal LMs in the future should align perfectly with humans on these psychology data? As we know that the conclusion of human data very much depend on who these people are. How do you imagine the different participant pool will impact the impact of the results of this study?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The application of psychometric data of human for analyzing LLMs behaviors reveals very interesting insights between the gap of LM vs. human decision-making.\n- The study design and the analysis are rigorous.\n- The insights of \"While we have focused on using theories and paradigms from psychology to analyze LLMs, there may also be opportunities to use LLMs to refine existing theories about people.\" is thought-provoking.", "weaknesses": "- One might argue that these psychology tests used in this work appear to be quite simplistic to inform practical development of next generation LMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jeteepX5B", "reviewer": "Reviewer_kdQx", "summary": "Mainly the paper prompts LLMs (Llama3, Claude Opus, GPT-4) to predict the human choices in two games (predicting gambling choices and inferring preferences from choices). They find that LLMs poorly predict human behavior in both the zero-shot and COT settings—in the latter, the error is explained well by the models predicting the human behavior to be more rational than it is.", "questions": "- Why set the temperature to 0.7 for some experiments?\n- If I'm reading correctly, the poor predictions in the zero-shot setting aren't explained well by the model incorrectly modeling humans as rational actors? What's your alternative explanation?\n\n> We found that GPT-4 Turbo could not provide a valid output ranking 47 choices at once. Thus, for all LLMs, we obtained rankings by asking the LLM to perform pairwise comparisons across 47choose2 pairs of decisions. Pairwise outputs were limited to {stronger, weaker, tie}, and were aggregated across decisions to form a ranking. Ties were discouraged to capture small differences between decisions\n\nWhy even permit the \"tie\" answer?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- applies research from the psychology field\n- good discussion of alignment", "weaknesses": "see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0kmUDyHLxJ", "reviewer": "Reviewer_HdFJ", "summary": "This paper investigates how LLMs perceive and predict human decision-making. The authors claim that LLMs (e.g., GPT-4 and Llama-3) tend to assume more rationality than people exhibit in reality. They adapt existing design of psychology experiments regarding human decisions, and compared human behavior with LLM predictions in two decision-making tasks: (1) predicting binary decision for gambling and (2) pairwise ranking the decisions regarding how much it tells about the human preference of an object. The experiments show that LLMs are more correlated with rational decisions than actual human behavior. The authors also claim this discrepancy between LLM rational assumptions and actual human behavior may have implications for alignment in AI systems, as well as for designing models that either simulate human behavior or align with human expectations.", "questions": "- How do you measure the correlation for task 2 and 3 in the forward modeling task?\n- Could there be circular inconsistencies in the pairwise ranking for the inverse modeling task? For instance, if the model ranks decision A > B, B > C, but then C > A, how is this resolved?\n- Why not use “average human participant” for the prompts in Table 5 in the Appendix?\n- Why did you decide to use “you” and refer this to human simulation? “What would an (average) human participant do?” sounds better?\n- In Table 11 in the Appendix, where is Bag 2 in Choice 2? Is it referring to the Bag 2 from above?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors explore a very interesting direction and implication in the current LLM alignment research. The current LLM alignment may actually limit the models’ ability to understand genuine human behavior.\n- The authors link their work nicely to existing well-grounded cognitive science studies, which will be helpful to other readers who are not familiar with this area.", "weaknesses": "- While I also believe LLMs struggle to model irrationality, I’m not entirely convinced that the current experiment design fully supports this claim.\n    - In the forward modeling section, the binary choice questions have an objectively optimal answer. Why should we conclude that the models are “assuming humans are more rational” simply because their answers are closer to this optimal? A simpler interpretation could be that the models are simply solving problems optimally, regardless of the prompt’s content.\n    - The fact that the forward modeling part relies on binary choice questions amplified my concern. For instance, consider a set of binary math questions that are challenging for humans, where we ask the model to predict which answer a person might choose. If the model consistently picks the correct answer more often than humans, would it be valid to conclude that the model assumes humans are more rational than they actually are? Or is the model simply aiming to answer correctly?\n- The forward modeling results in Table 1 & 2 indicate that models are more aligned with human choices than with maximum expected values in the zero-shot setting, but become more aligned with maximum expected values only when using CoT. Why, then, does this lead to the conclusion that “LLMs assume people are more rational than we really are”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U7Em4jf4xU", "reviewer": "Reviewer_dcdy", "summary": "This paper compares the implicit decision-making process of LLMs with the human decision making process and finds that LLMs assume that people are more rational than they are, which is in line with what humans expect from other humans.\n\nThe paper uses two test suites: (1) Forward Modeling, which is a task in which the LLM needs to predict how likely people would select gamble A over gamble B, (2) Inverse Modeling, in which the LLM needs to predict people’s preferences from a list of choices. The authors compare the outputs of several LLMs with reported human answers to come to their conclusions.\n\nThe authors argue that the results have important implications for alignment, namely that alignment should be divided into two cases: alignment with human expectations, and alignment with human behavior.", "questions": "Based on the weakness above:\n\n* What does the CoT prompt look like? Only the zero-shot prompt is reported in the appendix.\n* What is the effect of changing the temperature? Especially, how do the most popular temperature settings (when known) compare to temperature settings that are less common to use?\n* What is the effect of changing the prompt? Did the authors try many different prompts to arrive at their results, or do they think the results are stable if one changes the prompt?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* **Topic is important.** It is important to get insight in the decision process of LLMs, which has important consequences for how trustworthy we deem their outputs. This paper approaches this question from a CogSci angle.\n* **Paper invites follow-up work.** The paper sets the stage for follow-up work on alignment. What it means to align well with humans, depends on whether one wants to align to human expectations, or human behavior, according to the authors.\n* **Experiments are mostly solid.** The authors conduct experiments over a wide variety of popular LLMs, and compare the results. This gives insights in how the results generalize over different LLMs. Moreover, the experimental setup is strongly grounded in well-known experiments in Cognitive Science.", "weaknesses": "* **Unclear what the effect of the temperature setting is.** The authors briefly mention their choices for different temperature settings, but not in a lot of depth. Currently it is unclear how robust the findings are against different temperature settings (which would affect the outcomes). See questions below.\n* **Unclear what the effect of different prompting strategies is.** The authors report the zero-shot prompt used, and share some of their considerations for designing the prompt. However, as LLMs can be sensitive to the actual prompt used, it would be good to understand how robust the findings are when different prompts are used, beyond the zero-shot vs. CoT. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)