Turning large language models into cognitive models
Authors: Marcel Binz
Eric Schulz
Venue: ICLR 2024 poster (2024)
Keywords: cognitive modeling
large language models
neural networks
cognitive psychology
decision-making
Primary area: applications to neuroscience & cognitive science
TL;DR: We finetune a large language model on data from psychological experiments and find that doing so produces models that are more aligned with human decision-making.

Abstract: Large language models are powerful systems that excel at many tasks, ranging from translation to mathematical reasoning. Yet, at the same time, these models often show unhuman-like characteristics. In the present paper, we address this gap and ask whether large language models can be turned into cognitive models. We find that -- after finetuning them on data from psychological experiments -- these models offer accurate representations of human behavior, even outperforming traditional cognitive models in two decision-making domains. In addition, we show that their representations contain the information necessary to model behavior on the level of individual subjects. Finally, we demonstrate that finetuning on multiple tasks enables large language models to predict human behavior in a previously unseen task. Taken together, these results suggest that large, pre-trained models can be adapted to become models of human cognition, which opens up future research directions toward building more general cognitive models.

Published as a conference paper at ICLR 2024
TURNING LARGE LANGUAGE MODELS INTO COGNI- TIVE MODELS
Marcel Binz
Eric Schulz
Max Planck Institute for Biological Cybernetics Tübingen, Germany
Max Planck Institute for Biological Cybernetics Tübingen, Germany e eric.schul 12t .
ma www.nntuu............o..ue www.tueeee.m....mmmmm de
ABSTRACT
Large language models are powerful systems that excel at many tasks, ranging from translation to mathematical reasoning. Yet, at the same time, these models often show unhuman-like characteristics, In the present paper, we address this gap and ask whether large language models can be turned into cognitive models. We find that after finetuning them on data from psychological experiments these models offer accurate representations of human behavior, even outperforming tra- ditional cognitive models in two pecision-making domains. In addition, we show that their representations contain the information necessary model behavior on the level of individual subjects. Finally, we demonstrate that finetuning on multi- ple tasks enables large language models to predict human behavior in a previously unseen task. Taken together, these results suggest that large, pre-trained models be adapted become models of human cognition, which opens up future research directions toward building more general cognitive models.
INTRODUCTION
Large language models are neural networks trained on vast amounts data to predict the next token for a given text sequence (Brown et al. 2222)). These models display many emergent abilities that were not anticipated by extrapolating the performance of smaller models (Wei al.] (022). Their abilities are impressive and far-reaching that some have argued that they even show sparks of general intelligence (Bubeck al., (023). We may currently witness one of the biggest revolutions in artificial intelligence, but the impact of modern language models is felt far beyond, permeating into education (Kasneci et al. 2023), medicine (Li ct al. 2023), and the labor market (Eloundou al. 2023).
In-context learning the ability to extract information from a context and to use that information to improve the production of subsequent outputs is one of the defining features of such models. is through this mechanism that large language models are able to solve a variety of tasks, rang- ing from translation (Brown al. 2020) analogical reasoning (Webb et al. 2022). Previous work has shown that these models can even successfully navigate when they are placed into clas- sic psychological experiments (Binz Schulz 2023 Shiffrin & Mitchell| 2023 Coda-Forno al.) 2003 Dasgupta et al. 2022, Hagendortf et al. 2022). To provide just one example, GPT-3 - an autoregressive language model designed by OpenAl (Brown et al., 2020) outperformed human subjects in a sequential decision-making experiment that required to balance between exploitative and exploratory actions (Binz& Schulz 2023).
Even though these models show human-like behavioral characteristics in some situations, this is not always the case (Uilman| 2023) Mitchell & Krakauer] 2023). In the sequential decision-making experiment mentioned above, for instance, GPT-3 relied heavily exploitative strategies, while people applied a combination claborate exploration strategies (Wilson et al. 2014). Moreover, GPT-3 stopped improving after only few trials, while people continued learning the task pro- gressed.
In the present paper, investigate whether it is possible to fix the behavioral discrepancy between large language models and humans. To do so, we rely on the idea of finetuning on domain-speciff data. This approach has been fruitful across number of areas (annh al. 2019 Ouyang al.

---

Published as a conference paper at ICLR 2024
2022) and eventually led to the creation of the term foundatio models (Bommasani et al. 2021 models trained on broad data at scale and adapted to a wide range of downstream tasks. In the context of human cognition, such domain- -emain--pecifcc data can be readily accessed by tapping the vast amount of behavioral studies that psychologists have conducted over the last century. We exploited this fact and extracted data sets for several behavioral paradigms which we then used finetune large language model.
We show that this approach can be used create models that describe human behavior better than traditional cognitive models. We verify our result through extensive model simulations, which con- firm that finetuned language models indeed show human-like behavioral characteristics. Further- more, we find that embeddings obtained from such models contain the information necessary to capture individual differences. Finally, we highlight that a model finetuned on two tasks capable of predicting human behavior on third, hold-out task. Taken together, our work demonstrates that is possible to align large language models with human behavior by finetuning them on data from psychological experiments. We believe that this could open up new opportunities to harvest the power of large language models to inform our theories of human cognition the future.
METHODS
We tested whether is possible capture how people make decisions through finetuning a large language model. For our analyses, we relied on the Large Language Model Meta AI, or in short: LLaMA (Touvron et al. 2023). LLaMA is a family of state-of-the -frtt-rrtttttttt foundational large language models (with either 7B, 13B, 33B, or 65B parameters) that were trained on trillions of tokens coming from exclusively publicly available data sets. We focused on the largest of these models the 65B parameter version in our upcoming analyses. LLaMA is publicly available, meaning that researchers are provided with complete access to the network architecture including its pre-trained weights. We utilized this feature to extract embeddings for several cognitive tasks and then finetuned a linear layer on top of these embeddings to predict human choices (see Figure for a visualization). We call the resulting class of models CENTaUR, in analogy to the mythical creature that is half human and half ungulate.
For our analyses. first passed the corresponding text prompt (see Figure for stylized examples and Appendix I for exact prompts) through LLaMA. We then extracted the activations of the final hidden layer and repeated this procedure for each participant and trial. We fitted separate regular- ized logistic regression models on the standardized data via a maximum likelihood estimation. Final model performance was measured through the predictive log-likelihood on hold-out data obtained using a 100-fold cross-validation procedure. In each fold, we split the data into a training set (90%), a validation set (9%), and a test set (1%). The validation set was used identify the parameter a that controls the strength of the & regularization term using grid search procedure. We considered discrete a-values of [O, 0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0]. The optimization procedure was implemented in PYTORCH (Paszke et al.] 2019) and used the default LBFGS op- timizer (Liu Nocedal, 1989). Data and code for our study are available through the following GitHub repository; https:/ //ggtgttttb//////////////
/CENTAUR
3 RESULTS
We considered two paradigms that have been extensively studied in the human decision-making lit- erature for our analyses: decisions from descriptions (Kahneman & Tversky 1972) and decisions from experience (Hertwig et al.) 2004). In the former, decision-maker is asked to choose be- tween one of two hypothetical gambles like the ones shown in Figure Db. Thus, for both options, there is complete information about outcome probabilities and their respective values. In contrast, the decisions from experience paradigm does not provide such explicit information. Instead, the decision-makes has to learn about outcome probabilities and their respective values from repeated interactions with the task as shown in Figure Td. Importantly, this modification calls for a change in how an ideal decision-makse should approach such problems: it is not enough to merely exploit cur- rently available knowledge anymore but also crucial to explore options that are unfamiliar (Schulz) & Gershman 2019).
2

---

Published as a conference paper at ICLR 2024
(a)
You made the following obser-
made the following You made the following obser-
You made the following obser- vations in the past:
Large language model
Linear model
-Machine delivered S34,
-aachne delivered 841.
Machine delivered SS7.
-Machine delivered S37.
Q: Which machine do you choose?
A: Machine [insert]
(d)
choices1.3k
Horizon task
Machine $90 with 10.0% chance, -$12 with 90.0% chanee.
Negative log- log-likellhood
120000
You made the following obser-
Negative dgg-ilkellhodd
50000
vations the past:
90000
40000
0
-Machine delivered $34.
Machine 2: -$13 with 40.0% chanee, $22 with 60.0% chance.
-Machine delivered $41.
60000
-Machine delivered $57.
30000
-Machine delivered $37.
30000
20000
Q: Which machine do you choose?
Random BERT
LLaMA
CENTaUR Transformel BEAST
Q: Which machine you
Random BERT
LLaMA
CENTAUR Transforme Hybrid
choose?
A: Machine [insert]
A: Machine [insert]
Figure I: Illustration of our approach and main results. (a) We provided text-based descriptions of psychological experiments to a large language model and extracted the resulting embeddings. We then finetuned a linear layer on top of these embeddings to predict human choices. We refer to
the resulting model as CENTaUR, (b) Example prompt for the choices data set. (c) Negative
log-likelihoone for the choices data set. (d) Example prompt for the horizon task. (e) Negative
log-likeling ge-likkllihodds for the horizon task. Prompts shown in this figure are stylized for readability, Exact prompts can found Appendix A.I
For both these paradigms, we created a data set consisting of embeddings and the corresponding human choices. We obtained embeddings by passing prompts that included all the information that people had access
given trial through LLaMA and then extracting the hidden activations of the final layer. We relied on publicly available data from earlier studies in this process. In the decisions from descriptions setting, used the choices 13k data set (Peterson et al. 2021), which is a large-scale data set consisting of over 13,000 choice problems (all in all, 14,711 participants made million choices these problems). In the decisions from experience setting,
used data from the horizon task (Wilson et al. 2014) and a replication study (Feng et al. 2021), which combined include 60 participants making total of 67.200 choices. The horizon task a stationary two-armed bandit problem with normally- -rimallydistributedd rewards. Each task consists of either five or ten trials. The first four require the participant select a predetermined option (forced-chorl They are followed by either one or six free-choice trials in which participants can freely decide between the two options. The number of free-choice trials also known as the horizon. Participants are aware of the remaining choices and can use that information guide their exploration behavior.
trials).
3.1 FINETUNED LANGUAGE MODELS BEAT DOMAIN-SPECIFIC MODELS
With these two data sets at hand, we fitted regularized logistic regression model from the extracted embeddings to human choices described above. In this section, restricted ourselves to joint model for all participants, thereby neglecting potential individual differences (but see one of the following sections for analysis that allows for individual differences).
3

---

Published as a conference paper at ICLR 2024
We compared the goodness-of-fh of the resulting models against five baselines (for further details see Appendix I.2
1. a random guessing model.
2. a regularized logistic regression model with embeddings obtained from a simpler word embedding model (BERT, (Devlin et al., 2018)).
3. LLaMA without finetuning (obtained by reading out log-probabilities of the pre-trained model).
4. a regularized logistic regression model with embeddings obtained from a randomly initial- ized transformer (Transformer, (Schrimpf et al. (20TT).
5. a domain-specinn data set (Erev et al. 2017) and a hybrid model (Gershman 2018) that involves combina- tion of different exploitation and exploration strategies for the horizon task)
model (Best Estimate and Sampling Tools, or BEAST, for the choices
We found that LLaMA did not capture human behavior well, obtaining a negative log- log-likelihood (NLL) close to chance-level for the choices 13k data set (NLL = 96248.5) and the horizon task (NLL = 46211.4). However, finetuning led to models that captured human behavior better than the domain-specific models under consideration. In the choices13k data set, CENTaUR achieved negative log-likelihood of 48002.3 while BEAST only achieved a negative log-likelihood of 49448.1 (see ) In the horizon task, CENTAUR achieved negative log- ligllikelihodd of 25968.6 while the hybrid model only achieved negative log-likelihood of 29042.5 (see Figure [[b). The two other baselines (BERT and Transformer) were both significantly better than chance but did not reach the level of CENTAUR. Figure 6 in Appendix A.3 further corroborates these results using accuracy as measurement of goodness-off -fff-.
To ensure that our fitting procedure is robust to random noise, we repeated this analysis with five different seeds. CENTAUR's results were almost identical across seeds with an average negative log-likelihood of 48002.3 (SE= = 0.02) for the choices 13k data set and an average negative log- likelihood of 25968.6 6 (SE= 0.02) for the horizon task. In both cases, these values were significantly lower than those of the Nomain-speciffe models (p < 0.001). Together, these results suggest that the representations extracted from large language models are rich enough to attain state-of-the-as results for modeling human decision- decision-making.
3.2 MODEL SIMULATIONS REVEAL HIUMAN-LIKE BEIIAVIOR
We next verified that CENTAUR shows human-like behavioral characteristies. To do so, we simu- lated the model on the experimental tasks. Looking at performance, we found that finetuning led to models that closely resemble human performance as shown in Figure and b. For the choices- data set, CENTAUR obtained a regret (defined as the difference between the highest possible reward and the reward for the action selected by the model) of 1.35 (SF = 0.01), which was much closer to the human regret (M 1.24, SE 0.01) than the regret of LLaMAA (M 1.85, SE = 0.01). The results for the horizon task showed an identical pattern, with CENTaUR (M = 2.38, SE = 0.01) matching human regret (M = 2.33, SE = 0.05) more closely than LLaMA (M 7.21, SE = 0.02).
In addition to looking at performance, we also inspected choice curves. For this analysis, we took the data from the first free-choice trial in the horizon task and divided it into two conditions: (1) an equal information condition that includes trials where the decision-maker had access to an equal number of observations for both options and (2) an unequal information condition that includes trials where the decision-maker previously observed one option fewer times than the other. We then fitted a separate logistic regression model for each condition with reward difference, horizon, and their interaction as independent variables onto the simulated choices. Earlier studies with human subjects (Wilson et al., 2014) identified the following two main results regarding their exploratory behavior: (1) people's choices become more random with a longer horizon in the equal information condition (as shown in Figure 2 and (2) people in the unequal information condition select the more informative option more frequently when the task horizon is longer (as shown in Figure 2t). While LLaMA did not show any of the two effects (see Figure 2 and f), CENTaUR exhibited both of them (see Figure and h), thereby further corroborating that it accurately captures human behavior.
'There are a number of other Nomain-speciffic models that could be used as baselines instead (Bourgin al. 2019 He et al. 2022 Zhang & Yu 2013 Wilssn et al. 2014). While the ones we selected are good representatives of the lnterature,, future work should strive conduct more exhaustive comparison.
4

---

Published as a conference paper at ICLR 2024
(a)
(c)
(c)
cholces13k
Human
LLaMA
CENT&UR
Horizon Horizon
Horizon
Horizon
Regrets
Horizon
Horizon
pifirst option)
1.0
pffirst option)
1.0
pifirst option)
1.0
Random
LLaMA
CENTAUR
Human
0.0
0.0
0.0
25
25
25
Reward difference
Reward difference
Reward difference
(b)
(d)
(1)
(h)
task
Human
LLaMA
CENTBUR
Horizon 1 Horizon
Horizon 1
Horizon 1 Horizon
Regrets
p(more informative)
p(more informative)
Horizon
p(more informative)
1.0
1.0
1.0
0.5
0.5
0.5
Random
LLaMA
CENTaUR
Human
0.0
0.0
0.0
25
25
25
25
Reward difference
Reward difference
Reward difference
Figure 2: Model simulations. (a) Performance for different models and human participants on the choices13k 3k data set. (b) Performance for different models and human participants on the horizon task. (c) Human choice curves in the equal information condition of the horizon task. (d) Human choice curves the unequal information condition of the horizon task. (e) LLaMA choice curves in the equal information condition of the horizon task. (f) LLaMA choice curves in the unequal information condition of the horizon task. (g) CENTaUR choice curves in the equal information condition of the horizon task. (h) CENTAUR choice curves in the unequal information condition of the horizon task.
3.3 LANGUAGE MODEL EMBEDDINGS CAPTURE INDIVIDUAL DIFFERENCES
We also investigated how well CENTaUR describes the behavior of each individual participant. Note that this form of analysis is only possible for the horizon task as choice information on the participant level is not available for the choices13k 13k data (N= 52 out of 60) was best modeled by CENTAUR (see Figure for a detailed visualization). We furthermore entered the negative log-likelihoods into random-effects model selection procedure which estimates the probability that a particular model is the most frequent explanation within a set of candidate models (Rigoux et al. 2014). This procedure favored CENTaUR decisively, assigning a probability that it is the most frequent explanation of close to one.
In total, the majority of participants
Thus far, we have finetuned LLaMA jointly for all participants. However, people may exhibit indi- vidual differences that are not captured by this analysis. To close this gap and test whether LLaMA embeddings can account for individual differences, we incorporated random effects the finetuned layer. We added random effect for each participant and embedding dimension while keeping the re- maining evaluation procedure the same. Figure 3p illustrates the resulting negative lielliollss..ds. The mixed-effect structure improved goodness-of- of considerably (NLL= 23929.9 5) compared the effecttffett-nlly model (NLL= 25968.6) 6)) Furthermore, CENTaUR remained superior to the hybrid model with identical mixed-effect structure (NLL 24166.0). Taken together, the findings ported in this section highlight that embeddings of large language models contain the information necessary to model behavior the participant level.
5

---

Published as a conference paper at ICLR 2024
(a)
(b)
Negative log-likelihood difference
Random
log-likel aggllikllhhod
26000
LLaMA
Negative
25000
CENTaUR
Hybrid
24000
10
23000
CENTaUR
CENTAUR (mixed)
Hybrid (mixed)
Random
LLaMA
CENTaUR
Hybrid
Participant
Figure 3: Individual differences. (a) Negative log-likelihood difference to the best-fitting model for each participant. Black coloring highlights the best-fitting model, while white coloring corresponds difference larger than (b) Negative log- ligeliiissoods for models that were finetuned using the mixed-effects structure described in the main text.
3.4 EVALUATING GOODNESS-OF-FI ON HOLD-OUT TASKS
Next, we examined whether CENTaUR after being finetuned multiple tasks is able to predict human behavior in an entirely different task. This evaluation protocol provides a much stronger test for the generalization abilities of our approach. Following our initial analyses, layer on top of LLaMA embeddings. However, this time, we fitted a joint model using both the data from the choices 13k data set and the horizon task. and then evaluated how well the finetuned model captures human choices on a third task. Further details about the fitting procedure are provided in Appendix A.4 For the hold-out task, considered data from recent study that provided participants with a choice between one option whose information is provided via a description and another option for which information is provided via list of experienced outcomes (Garcia al., 2023). Figure 4 shows an example prompt for this experimental paradigm. In total, the experiment involved 98 participants making 25,872 choices. We only considered the mixed trials of the post- learning phase for our analysis, leading to data set size of 8,624.
finetuned linear
Finetuning was generally beneficial for modeling human behavior on the hold-out task: negative log-likelhhoods for CENTAUR (NLL= 4521.1) decreased both in comparison to a random guessing model (NLL= = 5977.7) and LLaMA (NLL 6307.9). We were thus curious whether CENTaUR also captures human behavior a qualitative level. To test this, took look at the key insight from the original study: people tend to overvalue options that are provided through a description (known symbolic or S-options) over the options that come with list of experienced outcomes (known as experiential or E-options) as illustrated in Figure 44 and c. LLaMA does not show this characteristic and instead weighs both option types equally (Figure 41 and e). In contrast this, CENTaUR shows human-like behavior, taking the S-option more strongly into account (Figure 4 and g). This is especially remarkable because we never presented data from the experiment under consideration during finetuning.
3.5 USING CENTAUR TO INFORM COGNITIVE THEBRIES
CENTAUR may also be used to gain insights into the cognitive processes that underlie observed behaviors. For instance, it can be used benchmark against which traditional cognitive models can be compared. This can help us to identify behavioral patterns that are predictable but that are not captured by existing cognitive models (Agrawal al. 2020). In the following, present proof of concept analysis which we have applied this idea to the two paradigms considered this paper.
We first computed differences between the log- li-likikihhods of CENTaUR and the domain-spenn specific model (BEAST for the choices 13k data set and the hybrid model for the horizon task) for each data point. We then ranked the data points and inspected those with the largest log- oogliikel difference.

---

Published as a conference paper at ICLR 2024
(a)
(b)
(d)
(f)
Human
LLaMA
CENTAUR
mude following
observations in the past:
1.0
1.0
1.0
E-option (win)
-Machine delivered
p(E-option)
p(E-option)
p(E-option)
-Machine delivered S1.
-Machine
delivered -$1.
0.5
0.5
0.5
-Machine
delivered
-Machine
delivered S1.
-Machine
delivered -$1.
0.0
0.0
0.0
-Machine
delivered S1.
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
-Muchine delivered S1,
-Machine
delivered -$1.
Soption p(win)
S-option p(win)
S-option p(win)
-Machnne delivered
(c)
(e)
(g)
Machine 2:
1.0
Indifference point
1.0
Indifference point
1.0
-!1 with 30.0% chance. $1 with 70.0% chunce.
Indifference
0.5
0.5
Q: Which machine do choose?
A: Machine [insert]
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
E-option p(win)
E-option p(win)
E-option p(win)
Figure 4: Hold-out task evaluations. (a) Example prompt for the ssperiential-symbolk task of Gareia et al. (2023). (b) Human choice curves as a function win probabilities for both options. (c) Human indifference points as a function of win probability for the E-option. Indifferent points express the win probabilities at which a decision-maker is equally likely to select both options. (d) ILLMA choice curves function of win probabilities for both options.
(e) LLaMA indifference points as a function of win probability for the E-option. (f) CENTaUR choice curves as a function of win probabilities for both options. (g) CENTaUR indifference points as function of win probability for the E-option.
These points correspond to cases where CENTaUR accurately predicted human behavior but the domain-specifie model did not. Figure shows three examples for each paradigm.
For the choices 13k data set, we observed that the data points with the largest log lilkkeiiioodd dif- ference correspond to problems where (1) one option consists of purely negative outcomes, (2) the other option contains least one positive outcome, and (3) the expected value of the purely negative option is higher. Humans, like CENTaUR, prefer the option with the positive outcome, whereas BEAST does not. We interpret this as a form of loss aversion (Kahneman) 1979) that is context- dependent. This bias could be integrated into the BEAST model to further advance our theories of human decision -akin....
For the horizon task, observed that the data points with the largest log- likeliidldood difference correspond to problems with (1) a high mean reward difference between options, (2) a long horizon, and (3) where people have repeatedly selected the option with the lower mean value. CENTAUR able to capture that people will stick to the less valuable option while the hybrid model is not, suggesting that the hybrid model could be improved by adding stickiness component (Gershman) 2020 Jagadish et al. 2023).
4 DISCUSSION
We have demonstrated that large language models can be turned into cognitive models by finetuning their final layer. This process led models that outperformed traditional cognitive models their goodness-of-fit human behavior. Furthermore, these models were able capture behavioral differences at the individual participant level. Finally, we have shown that our approach generalizes to previously unseen tasks. In particular, model that was finetuned on two tasks also exhibited human-like behavior on a third, hold-out task.
These results complement earlier work showing that large language model embeddings allow to predict behavior and neural activations in linguistic settings (Schrimpf et al.) 22,2002]] Kumar et al.)
2022 Tuckute et al. 2023 Antonello al. 2023). For example, (Schrimpf et al. (021 showed that
7

---

Published as a conference paper at ICLR 2024
(a)
Machine delivers
Machine delivers dollars 100% chance.
Machine 1 delivers
dollars with chance. -14 dollars with 25% chance.
9 dollars 100% chance.
Machine 2 delivers
Machine 2 delivers
Machine delivers
-33 dollars with 50% chanec,
-11 dollars with 40% chance,
-3.5 dollars with 12.5% 22.5% chance,
-2.5 dollars with 1.5625%% chance, 7.8125% chunce, -0.5 dellars with 15.625% chance.
-5 dollars with 30% chance,
dollars 37.5% chance, -1.5 dollars with 37.5% chance,
.1.5
33 dollars
15% chance,
dollars with 7.5% chunce.
-0.5 dollars 12.5% chance.
dellars with 15.625% chance,
dollars 3.75% chance,
1.5 dollars with 7.8125% chance,
25 dollars with 1.875% chance,
2.5 dollars with 1.5625% chunce.
57 dollars with 0.9375% chunce,
121 dollars with 0.9375% chance.
Your goal to maximize amount of reccived dollars.
Your goal to maximize amount of received dollars.
Your goal to maximize the amount of received dollars.
Q: Which machine da you choose?
Q: Which machine do you choose?
Q: Which muchine do you choose?
A: Machine
A: Machine 2
A: Machine 2
(b)
You mude the following observations in past:
You made following observations
You made following observations
the past:
the past:
Machine 2 delivered 34 dollars.
Machine delivered 74 dollars.
Maccine delivered 50 dollars.
Machine delivered 62 dollars.
Machine 2 delivered 75 dollars.
Machine 2 delivered 2 dollars.
Machine 1 delivered 64 dollars.
Machine
delivered 52 dollars.
Machine delivered dollars.
Machine 2 delivered 30 dollars.
Machine 1 delivered 40 dellars.
Machine delivered 39 dollars.
Machin: 1 delivered 59 dollars.
Machine 2 delivered 88 dollars.
Machine delivered dollars.
Machine 2 delivered 39 dollars.
Machine 2 delivered 74 dollars.
Machine 2 delivered 15 dollars,
Machine 2 delivered 26 dollars.
Machine 2 delivered 70 dollars.
Machine 2 delivered dollars.
Machine 2 delivered 16 dollars.
Machine delivered 34 dollars.
Machine delivered 20 dollars. Machine 2 delivered 4 dollars.
Machine 2 delivered 12 dollars.
Machine delivered 45 dollars.
Your goal to maximize the sum of received dollars within one additional choice.
Your goal to maximize the sum of received dollars within one additional choice.
Your goal to maximize the sum of received dollars within one additional choice.
Q: Which machine do you choose? A: Machine
Q: Which machine do you choose?
Q: Which machine do you choose? Machine
Machine
Figure 5: Prompts with the largest difference between the log- likelioos of CENTaUR and the domain-specific model. Human choices are highlighted in green. CENTAUR matched human choices, while the domain-specific model did not. (a) Examples from the choices 13k data sets. (b) Examples from the horizon task. Formatting adjusted for readability.
large language models can predict neural and behavioral responses in tasks that involved reading short passages with an accuracy that was close to noise ceiling. While it may be expected that large language models explain human behavior in linguistic domains (after all these models are trained to predict future word occurrences), the observation that these results also transfer to more cognitive domains like the ones studied here is highly non-trive con-trivial and has not been demonstrated before.
We are particularly excited about one feature of CENTaUR: embeddings extracted for different tasks all lie in a common space. This property allows finetuned large language models to solve multiple tasks in a unified architecture. We have presented preliminary results in this direction, showing that a model finetuned on two tasks can predict human behavior on a third. However, we believe that our current results only hint at the potential of this approach. Ideally, we would like to scale up our approach to finetuning on a larger number of tasks from the psychology literature. If one would include enough tasks in the training set, the resulting system should in principle - generalize to any hold-out task. Therefore, our approach provides path towards a domain- domain-general general model of human cognition, which has been the goal of psychologists for decades (Newell 1992 Yang et al. 2019 Riveland & Pouget 2022, Binz et al. 2023).
We believe that having access to such a nomain--eneral model would transform psychology and the behavioral sciences more generally. Recently, researchers even started to wonder whether large language models can replace human participants in experimental studies (Dillion et al. 2002)), i.c., whether they can be used as proxies for human behavior (Aher et al.) 2023 Jiang et al.) 2022) While we would not go as far, there are certainly compelling arguments for why to rely on such quicker, cheaper, and less noisy. Imagine, for instance, that you
agents for prototyping: they have just designed a new experimental paradigm. In a typical workflow, you would often run one or multiple pilot studies to gauge whether everything works as intended. However, with accurate
8

---

Published as a conference paper at ICLR 2024
domain-generts model hand, you may conduct the piloting step in silico, thereby saving time and money. In another situation, you may have an application that benefits from predicting how people act. For such use cases, domain-genette running expensive data collection procedures, you could work directly with the simulated data of the model, which could enable many applications that were previously out of reach.
model would be again tremendously useful. Instead of
However, it is important to mention that the present paper still falls short of this goal. We would expect generalist cognitive model be robust modifications in the prompt structure. While this is generally true for people, the same does not always hold for large language models (Frank) 2023a Mitchell 2023). For instance, recent paper found that modern language models can solve 90% of balse-belief tasks, thereby indicating theory of mind-like abilities (Kosinski, 2023). Follow-
up papers, on the other hand, showed that performance breaks down as soon as the task structure is modified in trivial ways, suggesting that the evidence is not as clear-cut (Ullman 2023 Shapira al., 2023). We also found this to the case for CENTaUR: performance breaks down once we evaluated finetuned models tasks with slightly alternated prompt structures (see Appendix | for further details), Identifying if and how these issues can be resolved is an important area for future research. We imagine that using data augmentation techniques and training the models larger data sets can improve their robustness, but it is still unclear to what extent.
Finally, we have to ask ourselves what can learn about human cognition when finetuning large language models. We have presented a proof of concept analysis demonstrating that CENTAUR can be used to inform cognitive theories. While this analysis is interesting in its own right, it is certainly not the end of the story. Looking beyond the current work, having access to an accurate neural network model of human behavior provides the opportunity apply wide range of explainability techniques from the machine learning literature. For instance, we could pick a particular neuron in the embedding and trace back what parts of an input sequence excite that neuron using methods
such as layer-wise relevance propagation (Bach et al. 2015] Chefer et al.] (021). The obtained insights could be then fed back into traditional cognitive models, thereby improving their ability to
characterize human behavior. It should be noted that analyses like this are not possible when working with human subjects alone and thus our work opens up a totally new spectrum techniques that could be applied to study human cognition.
To summarize, large language models are an immensely powerful tool for studying human behavior (Demszky et al. 2023 Frank 2023b). We believe that our work has only scratched the surface of this potential and there is certainly much more to come. Many questions only wait to be explored: larger models generally more suitable for finetuning? How do techniques such as reinforcement learning from human feedback (Ziegler et al. 2019) influence the goodness-of-fit to human behav- ior? Is possible finetune entire networks instead of just their final layer using techniques such as low-rank adaptation (Hu et al. [202]? However, the most crucial aspect that still ought to be studied in more detail is how far finetuned models are able to generalize beyond their training distribution.
ACKNOWLEDGMENTS
This work was funded by the Max Planck Society, the Volkswagen Foundation, as well as the Deutsche Forschungsgemeinsccaatt (DFG, German Research Foundation) under Germany's Excel- lence Stratgy-EXXC214))33377777

---

Published as a conference paper at ICLR 2024
REFERENCES
Mayank Agrawal, Joshua C. Peterson, and Thomas I. Griffiths. Scaling up psychology via scientific regret minimization. Proceedings of the National Academy of Sciences, 117(16):825-8835, 2020.
Gati V Aher, Rosa Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pp. 337-371. PMLR, 2023.
Richard Antonello, Aditya Vaidya, and Alexander G Huth. Scaling laws for language encoding models in fmri. arXiv preprint arXiv:2305.11866 2023.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Robu-R Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer- baser-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 12066))e211853312, 2023.
Marcel Binz, Ishita Dasgupta, Akshay Jagadish, Matthew Botvinick, Jane Wang, and Eric Schulz. Meta-learned models of cognition. arXiv preprint arXiv:2304.06729, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- 2021.
nities and risks of foundation models. arXiv preprint
David D Bourgin, Joshua C Peterson, Daniel Reichman, Stuart J Russell, and Thomas L Griffiths. Cognitive model priors for predicting human decisions. In International conference on machine learning, pp. 5133-51411 PMLR, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. NeurIPS, 33:1877- 1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg. et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 2023.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 782-791, 2021.
Julian Coda-Fomma Forno Kristin Witte, Akshay K Jagadish, Marcel Binz, Zeynep Akata, and Eric Schulz. Inducing anxiety in large language models increases exploration and bias. arXiv:2304. 11111, 2023.
Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James McClelland, and Felix Hill. Language models show human-like content effects on rea- soning. arXiv preprint MXXi::20770075511 2022.
Dorottya Demszky, Diyi Yang, David S Yeager, Christopher J Bryan, Margarett Clapper, Susannah Chandhok, Johannes Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, et al. Using large language models in psychology. Nature Reviews Psychology, pp. 1-14, 2023.
Jacob Devlin, Ming- Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04888 2018.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can ai language models replace human participants? Trendy in Cognitive Sciences, 2023.
10

---

Published as a conference paper at ICLR 2024
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303 10130, 2023.
Ido Erev, Eyal Ert, Ori Plonsky, Doron Cohen, and Oded Cohen. From anomalies forecasts: Toward a descriptive model of decisions under risk, under ambiguity, and from experience. Psy- chological review, 124(4):369, 2017.
Samuel F Feng, Siyu Wang, Sylvia Zarnescu, and Robert C Wilson. The dynamics of explore- exploit decisions reveal signal-to-noise mechanism for random exploration. Scientific reports, 11(1):1-15, 2021.
Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, pp. 1-2, 2023a.
Michael C Frank. Large language models as models of human cognition. 2023b.
Basile Garcia, Maël Lebreton, Sacha Bourgois-GGrondee and Stefano Palminteri. Experiential val- are underweighted in decisions involving symbolic options. Nature human behaviour, pp- 1-16, 2023.
Samuel Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34-42, 2018.
Samuel Gershman. Origin of perseveration in the trade-off between reward and complexity, Cog- nilion, 204:104394, 2020.
Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Machine intuition: Uncovering human-like intuitive decision-making gpt-3.5. arXiv preprint arXiv:2212.05226, 2022.
Lisheng He, Pantelis P Analytis, and Sudeep Bhatia. The wisdom of model crowds. Management
Science, 68(5)33635-3659 2022.
Ralph Hertwig, Greg Barron, Elke Weber, and Ido Erev. Decisions from experience and the effect of rare events in risky choice. Psychological science, 15(8):534-539, 2004.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 006800 2021.
Akshay Kumar Jagadish, Marcel Binz, Tankred Saanum, Jane X Wang, and Eric Schulz. Zero-shot compositional reinforcement learning in humans. PsyArXiv, 2023.
Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluat- ing and inducing personality in pre-trained language models. 2022.
Daniel Kahneman. Prospect theory: An analysis of decisions under risk. Econometrica, 47:278, 1979.
Daniel Kahneman and Amos Tversky. Subjective probability: A judgment of representativeness. Cognitive psychology, 3(3):430 1972.
Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274, 2023.
Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint ar3iv:2302 232002302..0083,, 2023.
Sreejan Kumar, Theodore Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A Norman, Thomas L Griffiths. Robert D Hawkins, and Samuel A Nastase. Reconstructing the cascade of language processing in the brain using the internal computations of transformer- based language model. BioRxiv, pp. 2022-06, 2022.

---

Published as a conference paper at ICLR 2024
Hanzhou Li, John T Moon, Saptarshi Purkayastha, Leo Anthony Celi, Hari Trivedi, and Judy W Gichoya. Ethics of large language models in medicine and medical research. The Lancet Digital Health, 5(6))e333-
2023.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3):503-522 1989.
Melanie Mitchell. How do we know how smart systems are?, 2023.
Melanie Mitchell and David Krakauer. The debate over understanding in ai's large language models. Proceedings of the National Academy of Sciences, 120x13)xx2211590712, 2023.
Allen Newell. Unified theories of cognition and the role of soar. Soar: A cognitive architecture in perspective: A tribute to Allen Newell, pp. 25-79, 1992.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Joshua C Peterson, David D Bourgin, Mayank Agrawal, Daniel Reichman, and Thomas L Grif- fiths. Using large-scale experiments and machine learning to discover theories of human decision- making. Science, 372(6547): 1209-12222111 1214, 2021.
Ori Plonsky, Reut Apel, Ido Erev, Eyal Ert, and Moshe Tennenholtz. When and how social scientists add value to data scientists? a choice prediction competition for human decision making. Unpublished Manuscript, 2018.
Lionel Rigoux, Klaas Enno Stephan, Karl Friston, and Jean Daunizeau. Bayesian model selection Neuroimage, 84:971-985, 2014.
for group
Reidar Riveland and Alexandre Pouget. neural model of task compositionality with natural lan- guage instructions. bioRxiv, 2022.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 2019.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: In- tegrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)=21056461 18, 2021.
Eric Schulz and Samuel J Gershman. The algorithmic architecture of exploration in the human brain. Current opinion in neurobiology. 55:7-14, 2019.
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305..476. 2023.
Richard Shiffrin and Melanie Mitchell. Probing the psychology ai models. Proceedings of the National Academy of Sciences, 120(10):e2300663 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302. 1397// 2023.
Greta Tuckute. Aalok Sathe, Shashank Srikant, Maya Taliaferro, Mingye Wang, Martin Schrimpf, Kendrick Kay, and Evelina Fedorenko. Driving and suppressing the human language network using large language models. bioRxiv, 2023.
12

---

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine Published as a conference paper at ICLR 2024
Tomer Ullman. Large language models fail trivial alterations theory-of of-mind tasks. arXiv preprint arXiv:2302.08399 2023.
learning research, 9(11), 2008.
Taylor Webb, Keith Holyoak, and Hongjing Lu. Emergent analogical reasoning large language 2022.
models. arXiv preprint
Jason Wei, Yi Tay, Rishi Bommasani. Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models. TMLR, 2022. ISSN 2835-8856.
Robert C Wilson, Andra Geana, John M White, Elliot A Ludvig, and Jonathan D Cohen. Humans use directed and random exploration solve the explore-exploit dilemma. Journal of Experimental Psychology: General, 143(6):2074, 2014.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Huggingface's transformers:
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, State-of-the-arr natural language processing. arXiv preprint arXiv:1910003771, 2019.
Guangyu Robert Yang. Madhura R Joglckar, H Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297-306. 2019.
Shunan Zhang and Angela Yu. Forgetful bayes and myopic planning: Human learning and decision-making in a bandit setting. Advances in neural information processing systems, 26, 2013.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint
2019.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 EXAMPLE PROMPTS
For the choices 13k data set, we prompted each decision independently, thereby ignoring the potential effect of feedback. We used the following template:
Machine delivers 90 dollars with 10.0% chance and -12 dollars with 90.0% chance. Machine delivers -13 dollars with 40.0% chance and 22 dollars with 60.09 chance.
Your goal is to maximize the amount of received dollars.
Q: Which machine do you choose?
A: Machine [insert]
For the horizon task, we prompted each task independently, thereby ignoring potential learning effects across the experiment. We used the following template:
You made the following observations in the past:
Machine delivered 34 dollars.
Machine delivered 41 dollars.
Machine delivered 57 dollars.
Machine delivered 37 dollars.
Your goal is to maximize the sum of received dollars within six additional choices.
Q: Which machine do you choose?
A: Machine [insert]
For the experiential-symmolii task, prompted each decision independently and only considered the post-learning phase. We furthermore simplified the observation history by only including the option that is relevant the current decision. We used the following template:
You made the following observations in the past:
Machine delivered dollars.
Machine delivered dollars.
Machine delivered -1 dollars.
Machine delivered dollars.
Machine delivered dollars.
Machine delivered -1 dollars.
[...]
Machine delivered dollars.
Machine delivers -1 dollars with 30.0% chance and dollars with 70.0% chance.
Your goal is to maximize the amount of received dollars.
Q: Which machine do you choose?
A: Machine [insert]
Note that the "[insert]" token is shown only for visual purposes and part of the actual prompt.
14

---

Published as a conference paper at ICLR 2024
A.2 BASELINE MODELS
For the LLaMA baseline, we passed the corresponding text prompt through LLaMA. We then extracted the unnormalized probabilities for the two options (i.e., tokens "I" and "2") and repeated this procedure for each participant and trial. We multiplied the unnormalized probabilities with temperature parameter, which was fitted human choices using the procedure described in the main text.
The fitting procedure for the BERT and Transformer baselines mirrored those of CENTaUR. We
extracted BERT embeddings using the transformers library (Wolf et al.] 2019). We extracted embeddings for the Transformer baseline by resetting LLaMA weights to their initial values before querying the model.
For the BEAST baseline, we relied on the version provided for the choice prediction competition 2018 (Plonsky et al., 2018). We additionally included an error model that selects random choice with a particular probability. We treated this probability as a free parameter and fitted it using the procedure described in the main text. The hybrid model closely followed the implementation of [Grshhman (2018). We replaced the probit link function with a logit link function to ensure comparability CENTaUR.
A.3 ACCURACY RESULTS
conttinss a validation of our main analysis using accuracy a measurement of goodness-of- fit. We again found that CENTaUR was the best performing model, achieving accuracies of 65.15 18% for the choices data set and 83.5% for the horizon task {
We also included GPT-4 (OpenAl 2023) as a baseline in this analysis. GPT-4 performed worse when compared to CENTaUR for both the choices13k 13k data set (2%) and the horizon task (80.3%) but better than LLaMA without any finetuning.
(a) choices13k
(b) Horizon task
1.00 1
1.00
Accuracy
Accuracy
0.75
0.75
0.50
0.50
Random
BERT LLaMA CENTaUR Transformet BEAST
GPT-4
Random
BERT
LLaMA CENTAUR Transformet Hybrid GPT-4
Figure 6: Replication of our main analysis (Figure using accuracy as a measurement of goodness-
of-fit.
A.4 HOLD-OUT TASK ANALYSES
The training set consisted of the concatenated choices 13 and horizon task data. To obtain a vali- dation and test set, we split the data of the experiential-symbolit task into eight folds and repeated the previously described fitting procedure for each fold. The validation set was used identify the
Note that the accuracy differs from the decision preference accuracy reported by (Peterson et al. 2021 The decision preference accuracy is defined as the proportion of problems in which the model prediction for ((A) is 0.5 when the observed proportions also 0.5. CENTaUR achieves decision preference accuracy of 81.9% the choices 133 data
15

---

Published as a conference paper at ICLR 2024
parameter that controls the strength of the b regularization term and inverse temperature pa- rameter. We considered discrete inverse temperature values [0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1] and a-vulues as described in the main text.
For our model simulations, simulated data deterministically based median threshold (again using the predictions on the test set). The resulting choice curves were generated by fitting separate logistic regression model for each possible win probability of the E-option. Each model used the win probability of the S-option and an intercept term as independent variables and the probability of choosing the E-option as the dependent variable.
A.5 ROBUSTNESS CHECKS
We tested how robust CENTaUR is minor modifications in the prompt structure. For this analysis, we finetuned CENTaUR on the choices 33 data set as described in the main text but evaluated on modified prompts. In particular, considered the following two modifications:
1. Moving the instructions ("Your goal is to maximize the amount of received dollars." ") and the question ("Q: Which machine do you choose?") to the beginning of the prompt.
2. Exchanging the order of probabilities and outcomes (i.e., "Machine delivers 90 dollars with 10.0% chance and -12 dollars with 90.0% chance" becomes "Machine 1 has 10.0% chance to deliver 90 dollars and 90.0% chance to deliver 12 dollars.")
In general, found that all of these modifications lead to goodness-of-ff below chance level. The exact results of the analysis are shown Figure 7
Negative log- log-llkellnodd
150000
90000
30000
changed intra option order-
question in the beginning
both modifications
Figure 7: Robustness minor prompt modifications on the choices 13k data set. The dotted red line corresponds to the goodness-off of fit of random guessing model.
16

---

Published as a conference paper at ICLR 2024
A.6 USING COGNITIVE THEORIES INFORM CENTAUR
We have also conducted an inverse of the analysis presented in Section 3.5 For this analysis, inspected the data points with the lowest log-likelihood difference between CENTaUR and the domain-specific model. These points correspond to cases where the domain-specifie model accu- rately predicted human behavior but CENTaUR did not. Figure shows three examples for each paradigm.
For the choices I 3k data set, we were not able to identify clear pattern in these data points. For the horizon task, we observed that the data points with the lowest log-likelihood difference correspond cases where participants switched between choices in the last trial after repeatedly selecting the other option. We could not identify clear pattern that would indicate this switch.
(a)
Machine delivers dollars with chunce. dollars with 99% chunce.
Machine delivers 90 dollars 440 dollars with 50% chance.
Machine delivers
chance.
dollars with 1% chance. 30 dollars with 99% chance.
Machine delivers 15 dollars with 95% chance, dollars with chunce.
Machine delivers 0 dollars with 10% chance, dollars with chance.
Machine delivers
-3 dollars with 1% chance,
3.0918% chunce. 27.5 dellars with 15.4688% chance.
26.5
28.5 dollars 30.9375% chance.
29.5 dollars with 30.9375% chance,
30.5 dollars with 15.4688% chance, 3.0938% chance.
31.5 dollars
goal to maximize amoumt of reccived dollars.
goal to maximize amount of received dollars.
Your goal to maximize amount received dollars.
Q: Which machine do you choose?
Q: Which machine do choose?
Q: Which machine do choose?
A: Machine
A: Machine 2
A: Machine 1
(b)
You made the following observations in
You made following observations
You made following observations
past:
the past:
the pust:
Machine delivered 67 dollars.
Machine delivered 47 dollars.
Maccine delivered 77 dollazs.
Machine delivered 61 dollars.
Machine 2 delivered 40 dollars.
Machine 2 delivered 77 dollars.
Machine 2 delivered 63 dollars.
Machine delivered 38 dollars.
Machine delivered 59 dollars.
Machine 2 delivered 65 dollars.
Machine 2 delivered 36 dullars.
Machine delivered 58 dollars.
Machin: 2 delivered 68 dollars.
Machine delivered 44 dellars.
Machine delivered 67 dollars.
Machine 2 delivered 67 dollars.
delivered dollars.
Machine delivered 63 dollars.
Machin: delivered 65 dollars.
Machine delivered 45 dellars.
Machine delivered 48 dollars.
Machine 2 delivered 79 dollars.
Machine delivered 46 dollars.
Machine delivered 71 dollars.
Machine 2 delivered 72 dollars.
Machine delivered 46 dollars.
Machine delivered 46 dollars.
Your goal to maximize the sum of received dollars within one additional choice.
Your goal to maximize the sum of reseived dollars within one additional choice.
Your goal to maximize the sum of received dollars within one additional choice.
Q: Which machine do you choose?
Q: Which machine do you choose?
Q: Which machine do you choose? Machine
A: Machine
Machine
Figure 8: Prompts with the lowest difference between the log- ligeliiilssoods of CENTaUR and the domain-specific model. Human choices are highlighted in green. (a) Examples from the choices data sets. (b) Examples from the horizon task. Formatting adjusted for readability.
17

---

Published as a conference paper at ICLR 2024
A.7 EMBEDDING ANALYSIS
We performed a simple t-SNE (Van der Maaten & Hinton 2008) analysis to investigate the informa- tion contained in the LLaMA embeddings. Figure shows the results of this analysis. We found that the different tasks can be separated in the t-SNE space and that the embeddings for the experiential- symbolic task lie in between those of the two individual tasks. While the t-SNE space also displays some structure with regard to the difference in values between both options, the overall picture is less clear in this case.
choices13k
Horizon task
Experiential-symbolic task
50
50
50
1.0
0.5
0
0.0
-25 25
-25 25
-25 25 p(Machine 2)
Task identity
Value difference
Figure 9: t-SNE analysis of LLaMA embeddings. The left plot shows 256 data points for each task with their task identity. The middle plot shows the same data points colored based their normalized value difference (Machine minus Machine 1). The right plot shows the same data points colored based on CENTaUR's probability of choosing Machine 2.
18


Reviews:
[{"review_id": "3Uwoii4gYE", "reviewer": "Reviewer_RhDW", "summary": "The authors propose to fine-tune large language models (LLM) on real human decision making behavior / tasks from psychology to potentially better align them with people and provide better predictions of behavior. They find that fine-tuning is largely successful on two such tasks and allow for at least some generalization to a third hold out task.", "questions": "--- What is the practical improvement (e.g., accuracy?) of fine-tuning?\n--- Can BEAST be fine-tuned as well?\n--- How does this compare to more typical feedforward neural networks with no pretraining.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I found this paper really interesting and worthwhile. While I suspect that there is much more work to be done in assessing the results, the main result is clear and fascinating: LLMs can quickly adapt to predict specific human behaviors. Perhaps most interesting is that this only requires linear regression. While this isn't really fine-tuning in the typical sense and could be considered a limitation (the authors could have trained a single layer or so, but with LLMs this is not easy by any means), I find it useful to know that only a simple transformation is needed. In fact, one could argue that the model didn't really have to learn anything about people. Everything the model needs appears to be located in the final input representation. One could further argue that this implies that LLMs already know how to emulate a diverse range of behavior, but that prompting is not the best way to test for it. In that sense, I think the primary contribution is strong. I don't think the authors needed to name their linear model, as the contribution is empirical and not methodological in some sense, but that's not a weakness given the aim of the paper.", "weaknesses": "The main weakness in my view is the difficulty in comparing to past work in the relevant domains and which are cited in the work. Past work appears to use different metrics, different splits of the data, and different baseline models. For example, BEAST does not appear to be the best or only relevant baseline, which is also usually evaluated using mean squared error. There is also a history of work behind the choices13k dataset with machine learning methods that the authors don't review. The authors also don't provide more interpretable metrics such as accuracy. One option to fix this would be make better comparisons using a larger set of relevant models, while another could be to focus on the success of fine-tuning more generally, which is already interesting, and less on benchmarks.\n\nMore generally, the claims of the paper could be toned down a bit. The space of human behavior is massive, and success in fine-tuning to extremely narrow ranges of behavior does not guarantee any particular success rate with other kinds. LLMs already fall short in matching human intelligence in some respects, and thus can't mimic such behavior. This may even be true or more nuanced \"non-intelligent\" behavior as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DHjlHeqM1", "reviewer": "Reviewer_sVTi", "summary": "This paper studies the problem of using large language models to simulate human results in behavioral experiments, with a focus on decision-making tasks. The method is intuitive and effective: linear probing of a llama model with human behavioral experiment data (i.e., extract latent embeddings and feed to a linear classifier to predict the judgment). Experiment results show such a fine-tuned-based approach can well fit human behavioral experiment distribution and generalize to hold-out tasks. This work shed light on some alternative approaches besides Bayesian models for predicting human decision-making results.", "questions": "See weaknesses. I'm open to discussing and increasing my score. The score I'm willing to give now is ~7, but this year's ICLR only has 6 and 8 options :). Moreover, is there any statistical analysis (e.g., significance tests) for comparing the correlation between human and CENTaUR predictions?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is well written, with a clear motivation of using large language models to simulate human behaviors (or at least binary choice results on two types of decision-making tasks). The detailed implementation: extract embeddings and then do a linear probing is good enough for a scalable method.\n\n- I like the idea of using large language models for a proxy model for analyzing human behavior. The crux is how to prob or design proper experimental methods (analogous to methods developed in experiment psychology). The proposed fine-tuning method performs well on behavior prediction tasks.\n\n- The main results are good, showing good predictive matches to human participants compared with other common methods. The hold-out tests also confirm the validity of using large language models as a universal cognitive model for behavioral tests.", "weaknesses": "- Although using open-sourced models (e.g., llama) is a good choice, the most powerful models to date (including instruction fine-tuned ones)  are not tested as a baseline method (e.g., few-shot evaluations on GPT-3.5/GPT-4/PaLM-2/Claude/instruction-finetuned llama 2 variants), it is suggested that some of those models can also demonstrate human-like behaviors in some human decision making tasks, through prompting or few-shot evaluations [A]. The few-shot method might be done by prompting some of the training examples in the CENTaUR method. So, it might be helpful to prove that CENTaUR is better than other prompting-based methods on more powerful LLMs.\n\n- Missing some discussions to previous works using LLMs to simulate and replicate behavior study results: [A-E]. Some similar aspects for using LLM in decision-making evaluations have already been proposed by Aher et al. [A].\n\n- Minor problems (only suggestions, not affecting the score): please use vectorized graphics (Figure 6 is good, but Figures 1-5 look blurry).\n\nref:\n\n[A]. Aher, G. V., Arriaga, R. I., & Kalai, A. T. (2023). Using large language models to simulate multiple humans and replicate human subject studies. In ICML 2023\n\n[B]. Jiang, G., Xu, M., Zhu, S. C., Han, W., Zhang, C., & Zhu, Y. (2023). Evaluating and Inducing Personality in Pre-trained Language Models. In NeurIPS 2023\n\n[C]. Frank, M. C. (2023). Large language models as models of human cognition. PsyArXiv\n\n[D]. Shiffrin, R. and Mitchell, M. (2023). Probing the psychology of AI models. PNAS\n\n[E]. Demszky, D., Yang, D., Yeager, D.S. et al. (2023). Using large language models in psychology. Nat Rev Psychol. *this one is not within the three months before the submission, so it is perfectly fine not to mention it :), just as a suggestion*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YHyk0tmNiT", "reviewer": "Reviewer_nBjG", "summary": "This work presents a novel cognitive model that uses embeddings from an LLM (LLaMa), together with a regression model, to predict human decision-making behavior. The model outperforms both the pre-trained LLM and domain-specific cognitive models, accounts for individual differences, and can generalize to a holdout task.", "questions": "- Have the authors considered whether a more powerful LLM (e.g. GPT-3 or GPT-4) might perform better on these tasks without fine-tuning, or carried out any such evaluation? My sense is that LLaMa is not as effectively instruction-tuned as these other models, and thus performs poorly in the zero-shot setting, which would explain the need for fine-tuning. But I wonder whether a more effectively instruction-tuned model might perform better 'out of the box' (I'm not suggesting that the authors need to perform this evaluation, just curious to hear their thoughts).\n- Have the authors tried to train LLaMa through in-context learning instead of fine-tuning? If this were effective, it might be more useful in settings where training data is limited (as is more generally the case in cognitive science).\n- Is CENTaUR an acronym, and if so what does it stand for?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This work presents an interesting and novel approach to cognitive modeling that outperforms domain-specific cognitive models.\n- The model is shown to be capable of generating qualitative cognitive insights, in addition to superior quantitative performance.\n- The model accounts for individual differences.\n- The model generalizes to a novel task.\n- The paper includes some interesting discussion of the broader implications of this approach for cognitive science.", "weaknesses": "- I am not sure if this is a weakness per se, but the work is primarily oriented toward cognitive science. It may be better suited to a more cog-sci oriented venue. However, I think the work generally makes a strong contribution and would be happy for it be published at ICLR.\n- My primary substantive concern is that the model is only evaluated on publicly available datasets. Do the authors know whether this data is included in LLaMa's pretraining data? I'm not entirely certain, but given the open-source nature of the model I think it should be possible to determine this. This seems like an important factor for considering how generalizable the approach will be to new datasets.\n- For the holdout task, is there an appropriate domain-specific model with which to compare CENTaUR?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Omoa01bD2P", "reviewer": "Reviewer_donP", "summary": "Can language models capture aspects of human behaviour? In this work, these authors take a step towards this question by introducing fine-tuning an LLM (specifically LLaMA) on human behavioural data. The authors find that the resulting model – CENTaUR – is able to well-capture binary human decisions and achieves competitive performance against alternative cognitive models. In particular, the authors find that CENTaUR’s learned embeddings generalise well to new human behaviour data, raising the prospects of leveraging (finetuned) LLMs as stimuli-computable cognitive models.", "questions": "Most of the hesitation in my final score stems from the Weaknesses above; if the authors are able to conduct a deeper dive into the potential gaps between CENTaUR and human performance (not just emphasizing the goodness of the fits) and further clarify the performance of CENTaUR relative to the existing cognitive baselines / broader motivation of these models as (replacement?) predictive cognitive models, I am open to raising my score. For instance, I believe that the authors resoundedly demonstrate that fine-tuning improves on LLaMA for fitting human behaviour, and this is a nice contribution; but I don't think the authors are yet ready to present their model as a stand-alone cognitive model. I realize the authors couch the limitations of this, but I think it could be further improved (per notes above).\n\nI raise a few other points which I believe are important and/or could add to the paper, but these are worth addressing only if the authors have time.\n\n- This is not so much a question, but would be nice for the authors to comment on (and perhaps include in the Discussion): The choices13k dataset appears to have been released in 2019? As such, it’s possible that the models were trained over this data (LLaMA performance is poor out-of-the-gate, so perhaps it is less of an issue; but it would be good for the authors to comment on as it could impact performance, and broadly a nuance in leveraging these models as cognitive models). One idea for the authors to begin to explore this is by repeating the prompt variation in Appendix A.4, but in the training set as well. \n- Was there any interesting visible structure in the embeddings learned? Have the authors run any kind of visualization over the embeddings (tSNE, etc) and perhaps looked into any possible structure there which could inform why some participants rather than others are not captured by CENTaUR? This is not necessary and the lack of such an investigation does not lower my score; but could be an exploration to strengthen the work. \n- Minor note: it would be nice if the authors could visualize the choice curves on the same graph. For instance, it looks like CENTaUR is much sharper in its curve than humans in Fig 4 (top row)? But this is a bit hard to cross-compare when the graphs are side-by-side.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The method proposed by the authors is sweet and simple (in a good way!). The idea of finetuning on human choice data is neat and impressively improves fits to human data. I believe the CENTaUR method therefore has value to the broader community. I think this study is a good first-step towards opening doors for the use of LLMs as fits of human data which would inspire further work in the space going forwards. \n\nI am particularly excited by the generalization results the authors demonstrate in Section 3.4. It would be intriguing to see how performance scales when jointly training CENTaUR on many more human behavior datasets. I believe the methodology the authors introduce in the work, and the fact that they will release their code (and nicely, are using an already open-source model, in contrast to the oft used GPT series) should set their framework up well for extension.   \n\nI also think the inclusion of a random effects component in the finetuning layer to capute individual differences is quite a nice (and seemingly novel? To my understanding?) idea; I would encourage the authors to expand on this further as a contribution, if they indeed believe the method is more generalizable and important for extending LLMs as cognitive models. \n\nI appreciate the authors’ rigour in their qualitative investigations (though I believe the results could be pushed on even further; see Weaknesses). I also appreciate the authors’ couching the broader implications of their work in the Discussion; the authors do a good job of cautioning against too broad a take on their findings of these models *as* cognitive models. Some of this could be brought further to the Intro (see below)).", "weaknesses": "While I think the authors do a good job in couching the generality of their method and limitations in the Discussion, I think this could be set-up further in the Introduction. At times, I feel the motivation was a bit confused. There is a difference, in my opinion, between seeking to bridge the gap between out-of-the-box language models and fits to behavioral data, versus actually using LLMs *as* cognitive models to *predict and study* human behavior. I believe this work nicely supports and expands on the former, and teases the possibility of the latter – but the results at present do not fully convince me that these models are ready yet, or should, serve as stimuli-computable models of human behavior. \n\nExpanding on the above, it’s not clear to me that CENTaUR is better than BEAST (Fig 1c); yes, the NLL is objectively lower – but what about error bars? Is it significantly lower? The authors do a good job of highlighting qualitative strengths of CENTAuR, but it would be nice for a deeper dive on the gaps between CENTaUR and human performance as well? For instance, could the authors construct the inverse of Fig 5? What do examples of cases where CENTaUR did *not* match humans (but perhaps BEAST / hybrid did?) The authors note as well that there are 8 out of 60 individual participants not well-captured by the \n\nMore broadly, it would be nice if the authors could include error bars in their results as much as possible (it should be feasible to obtain these, given the authors used 100-fold CV?). Further, why are the more classical cognitive models not included in the plots in 2a and b? \n\nAs a minor stylistic weakness: I did not find it helpful to read the raw NLL values in the text. I think it’s best to keep these in the figures/tables and discuss the trends. It’s hard to tell what counts as a large or small gap between NLL; again, error bars would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)