Large Language Models Assume People are More Rational than We Really are

Authors: Ryan Liu, Jiayi Geng, Joshua Peterson, Ilia Sucholutsky, Thomas L. Griffiths
Primary area: applications to neuroscience & cognitive science
Keywords: Large Language Models, Rationality, Cognitive Models, Psychology

Abstract:
In order for AI systems to communicate effectively with people, they must understand how we make decisions. However, people's decisions are not always rational, so the implicit internal models of human decision-making in Large Language Models (LLMs) must account for this. Previous empirical evidence seems to suggest that these implicit models are accurate --- LLMs offer believable proxies of human behavior, acting how we expect humans would in everyday interactions. However, by comparing LLM behavior and predictions to a large dataset of human decisions, we find that this is actually not the case: when both simulating and predicting people's choices, a suite of cutting-edge LLMs (GPT-4o \& 4-Turbo, Llama-3-8B \& 70B, Claude 3 Opus) assume that people are more rational than we really are. Specifically, these models deviate from human behavior and align more closely with a classic model of rational choice --- expected value theory. Interestingly, people also tend to assume that other people are rational when interpreting their behavior. As a consequence, when we compare the inferences that LLMs and people draw from the decisions of others using another psychological dataset, we find that these inferences are highly correlated. Thus, the implicit decision-making models of LLMs appear to be aligned with the human expectation that other people will act rationally, rather than with how people actually act.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730150944259

thread_type:
official_review

<thread_object>
creator: reviewer_dcdy

<content>
summary:
This paper compares the implicit decision-making process of LLMs with the human decision making process and finds that LLMs assume that people are more rational than they are, which is in line with what humans expect from other humans.

The paper uses two test suites: (1) Forward Modeling, which is a task in which the LLM needs to predict how likely people would select gamble A over gamble B, (2) Inverse Modeling, in which the LLM needs to predict people’s preferences from a list of choices. The authors compare the outputs of several LLMs with reported human answers to come to their conclusions.

The authors argue that the results have important implications for alignment, namely that alignment should be divided into two cases: alignment with human expectations, and alignment with human behavior.

strengths:
* **Topic is important.** It is important to get insight in the decision process of LLMs, which has important consequences for how trustworthy we deem their outputs. This paper approaches this question from a CogSci angle.
* **Paper invites follow-up work.** The paper sets the stage for follow-up work on alignment. What it means to align well with humans, depends on whether one wants to align to human expectations, or human behavior, according to the authors.
* **Experiments are mostly solid.** The authors conduct experiments over a wide variety of popular LLMs, and compare the results. This gives insights in how the results generalize over different LLMs. Moreover, the experimental setup is strongly grounded in well-known experiments in Cognitive Science.

weaknesses:
* **Unclear what the effect of the temperature setting is.** The authors briefly mention their choices for different temperature settings, but not in a lot of depth. Currently it is unclear how robust the findings are against different temperature settings (which would affect the outcomes). See questions below.
* **Unclear what the effect of different prompting strategies is.** The authors report the zero-shot prompt used, and share some of their considerations for designing the prompt. However, as LLMs can be sensitive to the actual prompt used, it would be good to understand how robust the findings are when different prompts are used, beyond the zero-shot vs. CoT. See questions below.

questions:
Based on the weakness above:

* What does the CoT prompt look like? Only the zero-shot prompt is reported in the appendix.
* What is the effect of changing the temperature? Especially, how do the most popular temperature settings (when known) compare to temperature settings that are less common to use?
* What is the effect of changing the prompt? Did the authors try many different prompts to arrive at their results, or do they think the results are stable if one changes the prompt?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706796300

thread_type:
official_review

<thread_object>
creator: reviewer_hdfj

<content>
summary:
This paper investigates how LLMs perceive and predict human decision-making. The authors claim that LLMs (e.g., GPT-4 and Llama-3) tend to assume more rationality than people exhibit in reality. They adapt existing design of psychology experiments regarding human decisions, and compared human behavior with LLM predictions in two decision-making tasks: (1) predicting binary decision for gambling and (2) pairwise ranking the decisions regarding how much it tells about the human preference of an object. The experiments show that LLMs are more correlated with rational decisions than actual human behavior. The authors also claim this discrepancy between LLM rational assumptions and actual human behavior may have implications for alignment in AI systems, as well as for designing models that either simulate human behavior or align with human expectations.

strengths:
- The authors explore a very interesting direction and implication in the current LLM alignment research. The current LLM alignment may actually limit the models’ ability to understand genuine human behavior.
- The authors link their work nicely to existing well-grounded cognitive science studies, which will be helpful to other readers who are not familiar with this area.

weaknesses:
- While I also believe LLMs struggle to model irrationality, I’m not entirely convinced that the current experiment design fully supports this claim.
    - In the forward modeling section, the binary choice questions have an objectively optimal answer. Why should we conclude that the models are “assuming humans are more rational” simply because their answers are closer to this optimal? A simpler interpretation could be that the models are simply solving problems optimally, regardless of the prompt’s content.
    - The fact that the forward modeling part relies on binary choice questions amplified my concern. For instance, consider a set of binary math questions that are challenging for humans, where we ask the model to predict which answer a person might choose. If the model consistently picks the correct answer more often than humans, would it be valid to conclude that the model assumes humans are more rational than they actually are? Or is the model simply aiming to answer correctly?
- The forward modeling results in Table 1 & 2 indicate that models are more aligned with human choices than with maximum expected values in the zero-shot setting, but become more aligned with maximum expected values only when using CoT. Why, then, does this lead to the conclusion that “LLMs assume people are more rational than we really are”?

questions:
- How do you measure the correlation for task 2 and 3 in the forward modeling task?
- Could there be circular inconsistencies in the pairwise ranking for the inverse modeling task? For instance, if the model ranks decision A > B, B > C, but then C > A, how is this resolved?
- Why not use “average human participant” for the prompts in Table 5 in the Appendix?
- Why did you decide to use “you” and refer this to human simulation? “What would an (average) human participant do?” sounds better?
- In Table 11 in the Appendix, where is Bag 2 in Choice 2? Is it referring to the Bag 2 from above?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721930183

thread_type:
official_review

<thread_object>
creator: reviewer_kdqx

<content>
summary:
Mainly the paper prompts LLMs (Llama3, Claude Opus, GPT-4) to predict the human choices in two games (predicting gambling choices and inferring preferences from choices). They find that LLMs poorly predict human behavior in both the zero-shot and COT settings—in the latter, the error is explained well by the models predicting the human behavior to be more rational than it is.

strengths:
- applies research from the psychology field
- good discussion of alignment

weaknesses:
see questions

questions:
- Why set the temperature to 0.7 for some experiments?
- If I'm reading correctly, the poor predictions in the zero-shot setting aren't explained well by the model incorrectly modeling humans as rational actors? What's your alternative explanation?

> We found that GPT-4 Turbo could not provide a valid output ranking 47 choices at once. Thus, for all LLMs, we obtained rankings by asking the LLM to perform pairwise comparisons across 47choose2 pairs of decisions. Pairwise outputs were limited to {stronger, weaker, tie}, and were aggregated across decisions to form a ranking. Ties were discouraged to capture small differences between decisions

Why even permit the "tie" answer?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730936322482

thread_type:
official_review

<thread_object>
creator: reviewer_2erm

<content>
summary:
This paper studies how advanced LLMs make predictions of human decisions, and reveal that frontier LMs assume that people are more rational than we really are. By using established datasets capturing human decisions, this study show that LLMs appear to be aligned with the human expectation that other people will act rationally (much like expected value theory), rather than with how people actually act.

strengths:
- The application of psychometric data of human for analyzing LLMs behaviors reveals very interesting insights between the gap of LM vs. human decision-making.
- The study design and the analysis are rigorous.
- The insights of "While we have focused on using theories and paradigms from psychology to analyze LLMs, there may also be opportunities to use LLMs to refine existing theories about people." is thought-provoking.

weaknesses:
- One might argue that these psychology tests used in this work appear to be quite simplistic to inform practical development of next generation LMs.

questions:
- What are some other potential human decision-making models that can also be tested on LLMs, in addition to forward and inverse modeling?
- How are the insights gathered from this study inform the development of LMs applications, or inform better design of the next generation of LMs or other AIs?
- Would you imagine an ideal LMs in the future should align perfectly with humans on these psychology data? As we know that the conclusion of human data very much depend on who these people are. How do you imagine the different participant pool will impact the impact of the results of this study?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732489834529

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
## General Response [1/2]
We thank all the reviewers for their insightful comments!

We are glad that they found that our paper’s “topic is important” (dcdy), is “thought-provoking” (2ERM), and that it “invites follow-up work” (dcdy). We appreciate that they unanimously believe that we “link [our] work nicely to existing well-grounded cognitive science studies” (HdFJ), that our “setup is strongly grounded in well-known experiments” (dcdy), “applies research from the psychology field” (kdQx), and the “application of psychometric data [...] reveals very interesting insights” (2ERM). 

We also appreciate that our reviewers found our work to have “very interesting direction and implication in the current LLM alignment research” (HdFJ), including a “good discussion of alignment (kdQx) and “important consequences for how trustworthy we deem their outputs” (dcdy). 

We appreciate the positive feedback regarding our experiments, and are glad that reviewers have found our paper’s “design and analysis are rigorous” (2ERM), featuring a “wide variety of popular LLMs” leading to “insights in how the results generalize over different LLMs” (dcdy). 

We also deeply appreciate their questions regarding generalizability across temperatures (kdQx, dcdy), participant groups / simulated human profiles (2ERM, HdFJ), and prompting strategies (dcdy). To address the reviewers’ concerns, we provide a set of **new experiments covering the issues that were raised**. We highlight the results for each experiment below. 

We also appreciate the comments about lack of clarity in interpreting the zero-shot results (kdQx, HdFJ). The corresponding discussion (originally in the 2nd paragraph of 3.1 and Appendix D) was not properly presented in the main text, and we have **revised this section to elucidate our findings**. We have added an explanation below. 

For reviewer-specific questions on evaluation decision details (e.g., measuring correlation from HdFJ), please refer to review-specific responses!

## 1. Deep dive into zero-shot results for Forward Modeling
When we found that models’ forward modeling zero-shot results were not well explained by maximum expected value, we conducted a fit across a wider range of 18 behavioral models to try to understand this behavior. 

Specifically, we replicated the original analyses conducted by the authors of the choices13k dataset [1], which included 
Heuristic models [2], where people are thought to use mental shortcuts to make decisions 
Counterfactual models [2], which appeal to constructs like regret and disappointment
Subjective Expected Utility models, which assume that quantities involved (money and probability) are perceived or treated subjectively. This includes many of the most influential models including Expected Utility Theory, Prospect Theory [3], and the model proposed by [1], the Mixture of Theories (MOT).

In Appendix D, we provide the full analysis of the results of fitting these various models. The best interpretation we found uses MOT, where the model fit a mixture of two probability weighting functions — one which is linear (matching humans), and one that approximated a flat line. This suggests that the LLM often expected people to be rational (ie. the linear function), but also completely ignores all probabilities (i.e., weights them equally) a significant proportion of the time. 

We have rewritten Section 3.1 to better highlight this result. This is in blue in the **updated PDF**. We thank the reviewers for pointing this out for us. 

[1] Joshua C. Peterson, David D. Bourgin, Mayank Agrawal, Daniel Reichman, and Thomas L. Griffiths. 2021. Using large-scale experiments and machine learning to discover theories of human decision-making. Science 372, 6547 (2021), 1209–1214.
[2] Lisheng He, Wenjia Joyce Zhao, and Sudeep Bhatia. 2022. An ontology of decision models.
Psychological Review 129, 1 (2022), 49–72.
[3] Daniel Kahneman and Amos Tversky. 1979. Prospect theory: An analysis of decision under risk. Econometrica 47, 2 (1979), 263–292.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732507514615

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 2/2

comment:
## 2. Additional experiment: Temperature
Originally, we chose temperature = 1 for some tasks and temperature = 0.7 for others, which the reviewers point out is inconsistent. 0.7 is another common temperature setting, used by works such as AlpacaEval [4] on instruction following tasks. 

To further validate the robustness of our findings, we test the forward modeling experiment using temperatures 0.0, 0.5, 1.0, 1.5, and 2.0 for both zero-shot and CoT conditions, and compute spearman correlation with both human and max expected value, shown in the table below. We find that varying temperature does not substantially alter our findings other than CoT with high temperatures, where the model becomes unable to complete long responses due to noisy outputs. 

| Zero-shot vs.|Max Expected Value | CoT vs. Max |Expected Value | 
|----------|----------|----------|-----|
| Temp | Spearman | Temp | Spearman |
| 0.0  | 0.313 | 0.0   | 0.540 |
| 0.5  | 0.272 | 0.5   | 0.532 |
| 1.0  | 0.284 | 1.0   | 0.539 |
| 1.5  | 0.302 | 1.5   | 0.351 |
| 2.0  | 0.308 | 2.0   | model unable |

| Zero-shot vs.|Human | CoT vs. |Human| 
|----------|----------|----------|-----|
| Temp | Spearman | Temp | Spearman |
| 0.0  | 0.533 | 0.0   | 0.540 |
| 0.5  | 0.381 | 0.5   | 0.477 |
| 1.0  | 0.341 | 1.0   | 0.518 |
| 1.5  | 0.489 | 1.5   | 0.528 |
| 2.0  | 0.389 | 2.0   | model unable |

[4] Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., & Hashimoto, T. B. (2023). AlpacaEval: An Automatic Evaluator of Instruction-following Models. GitHub repository. url: https://github.com/tatsu-lab/alpaca_eval

## 3. Additional experiment: Participant Groups and Prompting Variations
To alleviate some concerns about the simplicity of the experimental setup and lack of representation across demographic groups, we conducted a follow-up experiment replicating our original analysis on various demographic profiles varying age and gender. We find that our analyses stay consistent across all these factors, suggesting that our findings are at least somewhat robust to variations in context as well as differing profiles of people. 

For additional prompt variation, we also prompt the LLM to make the same predictions for a monkey rather than a human. While we do not see differences in zero-shot results, when performing CoT models accurately reflect that monkeys are less rational and the resulting predictions are less correlated with max expected value. However, the correlations are also increased with actual human behavior, supporting our hypothesis with an example where the model predicts less rational decisions, but these are actually more in-line with how humans actually behave. 

**Demographic + Prompt Variation Results**
| Zero-shot vs. |Max EV | |CoT vs. Max EV | | |
|----------|----------|----------|-----|-----|----|
| Demographic | Spearman | p-value    | Demographic | Spearman | p-value | 
| Woman 18-35 | 0.289        | 9.44E-21 | Woman 18-35 | 0.970        | <1E-150 |
| Woman 45-60 | 0.313        | 4.23E-24 | Woman 45-60 | 0.972        | <1E-150 |
| Woman 65-85 | 0.289        | 9.44E-21 | Woman 65-85 | 0.969        | <1E-150 |
| Man 18-35      | 0.335        | 1.37E-27 | Man 18-35      | 0.971        | <1E-150 |
| Man 45-60      | 0.327        | 2.50E-26 | Man 45-60      | 0.965        | <1E-150 |
| Man 65-85      | 0.348        | 9.19E-30 | Man 65-85      | 0.972        | <1E-150 |
| Monkey           | 0.348        | 8.41E-30 | Monkey          | 0.654        | 7.37E-123 |

| Zero-shot vs. |Human | |CoT vs. Human | | |
|----------|----------|----------|-----|-----|----|
| Demographic | Spearman | p-value    | Demographic | Spearman | p-value | 
| Woman 18-35 | 0.529        | 2.75E-73 | Woman 18-35 | 0.549        | 9.23E-80 |
| Woman 45-60 | 0.519        | 6.07E-70 | Woman 45-60 | 0.546        | 8.54E-79 |
| Woman 65-85 | 0.529        | 2.75E-73 | Woman 65-85 | 0.550        | 3.70E-80 |
| Man 18-35      | 0.499        | 4.18E-64 | Man 18-35      | 0.545        | 2.10E-78 |
| Man 45-60      | 0.530        | 1.43E-73 | Man 45-60      | 0.539        | 1.34E-76 |
| Man 65-85      | 0.508        | 8.54E-67 | Man 65-85      | 0.541        | 3.10E-77 |
| Monkey           | 0.482        | 2.18E-59 | Monkey          | 0.630        | 8.47E-112 |
</content>
</thread_object>

</thread>
<thread>
creation date:
1733306053265

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion Period Summary

comment:
To conclude the discussion period, we would like to provide a quick summary comprising of the discussions:

### 1. Better explanation for zero-shot case in forward modeling

Initially, reviewers kdQx and HdFJ expressed concerns about how to interpret the zero-shot results for the forward modeling. We have since updated the writing to provide readers a better understanding of this (see Section 3.1 blue text in PDF). 

### 2. Strong results on additional experiments testing generalizability: {temperature, demographic prompts, prompting strategies}

In the original reviews, reviewers had concerns about generalizability across temperatures (kdQx, dcdy), participant groups / simulated human profiles (2ERM, HdFJ), and prompting strategies (dcdy). We conducted additional experiments on temperature and demographics (highlighted in the general response) that show that our findings are consistent across these attributes, which reviewers HdFJ and dcdy have since acknowledged: 

"I appreciate the authors' engagement and their extensive additional experiments. As a result, I am raising my score." (HdFJ)

"Thank you for answering my questions and providing these additional experiments. Would be great to see those in the CR as well." (dcdy)

Both these reviewers have expressed the desire to see these additional experiments included in the main text, and reviewer HdFJ has requested additional models for these experiments, which we will happily comply for the camera ready version if the paper is accepted. 

### 3. Discussion with reviewer 2ERM on practical impact on informing the development of LMs

Reviewer 2ERM brought up that 'there lacks clear practical direct impact of these study results to what the authors identify as the successor applications---"identifying effective ways to provide assistance, simulating the helpfulness or harmlessness of a response, and learning individuals’ values and preferences"'. We have provided an explanation on why we believe these cases are well-motivated, and plan to add this as a discussion section in the final version of the paper if it is accepted to help readers better understand the grounded impacts of this work. 

Separately, it is worth mentioning that Reviewer kdQx has not responded to our rebuttal, so they were not able to update their opinion given our points 1 and 2 --- which they request in their original review.

### Conclusion
We thank our reviewers for engaging with our paper and providing invaluable feedback in their reviews and during the discussion period, and as our reviewers have found our work to have impacts in impacts towards alignment, strong connections to psychology literature, that the "topic is important" (dcdy) and the paper "reveals very important insights" (2ERM), we hope to be able to share our findings with a larger audience.
</content>
</thread_object>

</thread>


Metareview:
metareview:
PC is entering meta-review on behalf of SAC/AC:

The reviewers felt that the paper was an interesting research direction, and had important implication in LLM alignment research demonstrating the limitations of models’ ability to understand genuine human behaviour.