Can Large Language Model Agents Simulate Human Trust Behavior?
Authors: Chengxing Xie
Canyu Chen
Feiran Jia
Ziyu Ye
Shiyang Lai
Kai Shu
Jindong Gu
Adel Bibi
Ziniu Hu
David Jurgens
James Evans
Philip Torr
Bernard Ghanem
Guohao Li
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM Agent
Human Simulation
Behavioral Alignment
Trust Games
Primary area: machine_learning_for_social_sciences
TL;DR: We discover that LLM agents generally exhibit trust behavior in Trust Games and GPT-4 agents manifest high behavioral alignment with humans in terms of trust behavior, indicating the potential to simulate human trust behavior with LLM agents.

Abstract: Large Language Model (LLM) agents have been increasingly adopted as simulation tools to model humans in social science and role-playing applications. However, one fundamental question remains: can LLM agents really simulate human behavior? In this paper, we focus on one critical and elemental behavior in human interactions, trust, and investigate whether LLM agents can simulate human trust behavior. We first find that LLM agents generally exhibit trust behavior, referred to as agent trust, under the framework of Trust Games, which are widely recognized in behavioral economics. Then, we discover that GPT-4 agents manifest high behavioral alignment with humans in terms of trust behavior, indicating the feasibility of simulating human trust behavior with LLM agents. In addition,  we probe the biases of agent trust and  differences in agent trust towards other LLM agents and humans. We also explore the intrinsic properties of agent trust under conditions including external manipulations and advanced reasoning strategies. Our study provides new insights into the behaviors of LLM agents and the fundamental analogy between LLMs and humans beyond value alignment. We further illustrate broader implications of our discoveries for applications where trust is paramount.

Can Large Language Model Agents Simulate Human Trust Behavior?
Chengxing Xi
Canyu Chen
Feiran Jia Ziyu Ye
Shiyang Lai Kai Shu
Jindong Gu²
Adel Bibi
Ziniu Hu?
David Jurgens James Evans 10
Philip H.S. Torr
Bernard Ghanem' Guohao Li 13
KAUUST
Illinois Institute of Technology Universsiy of Oxford
*Pennsylvania State University EEmory "California Institute of Technology
University of Chicago
University of Michigan
Santa Fe Institute GGoole
"CAMEL-AL.org
Project website: https:/ageen-mmss.cmmeaaaa
Abstract
Large Language Model (LLM) agents have been increasingly adopted simulation tools to model humans in social science and role-playine applications. However, one fundamental question remains: can LLM agents really simulate human be- havior? In this paper, we focus on one critical and elemental behavior in human interactions, trust, and investigate whether LLM agents can simulate human trust behavior. We first find that LLM agents generally exhibit trust behavior, referred as agent trust, under the framework of Trust Games, which are widely recog- nized in behavioral economics. Then, we discover that GPT-4 agents manifest high behavioral aligument with humans terms of trust behavior, indicating the feasibility of simulating human trust behavior with L.I agents. In addition, we probe the biases of agent trust and differences agent trust towards other I.I.M agents and humans. We also explore the intrinsic properties of agent trust under conditions including external manipulations and advanced reasoning strategies. Our study provides new insights into the behaviors of LLM agents and the fundamental analogy between LLMs and humans beyond value alignment. We further illustrate broader implications of our discoveries for applications where trust is paramount.
Introduction
There is increasing trend Large Language Models (LLMs) agent-based simulation tools for humans in various social science fields including economics, politics, psychology. ecology and sociology (Gao et al., 2023b; Manning al., 2024; Ziems et al., 2023), and role-playing applications such as assistants, companions and mentors (Yang et al., 2024; Abdelghani et al., 2023; Chen et al., 2024) due their human-like cognitive capacity. Nevertheless, most previous research is based on one insufficiently validated assumption that LLM agents behave like humans in simulation. Thus, fundamental question remains: Can LLM agents really simulate human behavior?
In this paper, we focus on trust behavior in human interactions, which comprises the intention to place self-interes. at risk based the positive expectations of others (Rousseau et al., 1998). Trust one of the most critical and elemental behaviors in human interactions and plays an essential role in social settings ranging from daily communication economic and political institutions (Uslaner, 2000; Coleman, 1994). Here, we investigate whether LLM agents can simulate human trust behavior, paving the way explore their potential simulate more complex human behavior and society itself.
First, we explore whether LLM agents manifest trust behavior in their interactions. Given the challenge of quantifying trust behavior, we choose to study them based on the Trust Game and its
'Equal Contribution. Correspondence to: Chengxing Xie exiechengring344gmaa1. comm., com>, Canyu Chen Sachen151@haslee iit. edu>, Guohao Li <guohao@robots. ox. ac uc.u..
Work performed while Guohao Li was at KAUST and Chengxing Xie visiting student at KAUST.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

[ Latrorely befieve in the vour of trualing my fellow humen baings, howaver, om also sware that not eratyore will prove to be mastwarthy. I'v dlways believed in the princppe of taking calculsted rists, which is on inportent aspect of my profession as lawyer. In tems of desines, I cesire to see a world where people ast out berewilence and good intenhore ramer than selfish mptives.. online would have same motive and wodd return same, Reganding intentions, inhend give other player dollers grinng them the apsortunity moniyy,
GPT-4
Llanaz-1b
110 Llamak-70b
Visune-1b Vioune-150 Vicune-33b
the person I'n pained with of money book. this experiment as well, beelove for myself considering there's no
give the
Prompt Design
It's för to keep guarsnhes Finalyy. give
Szeatee Peceoon
Inwneert-year--ddd (gandar) (job). (bsckigroudd..
Trust Sand dallar
You're talking part experiment. You randonly pained
tcustee Info
another player, You know the player
player
Trustor Agent
Recipracate, Returt dellars
Trustee Agent
who you
Behavioral Aligement
You will receive $10 from the shudy group. You cen give cellers to the other player, and the player will receive 3N dellors and than con choose how much to return to you.
Game detting
Trust, Sand .. dollars
How much money would you give to the other player?
Reciprotate, Return dellars
Trustor Human
Trustee Human
Figure 1: Our Framework for Investigating Agent Trust as well as its Behavioral Alignment with Human Trust. First, this figure shows the major components for studying the trust behavior of LLM agents with Trust Games and Belief-Desire-Ing Intention (BDI) modeling. Then, our study centers on examining the behavioral alignment between LLM agents and humans regarding trust behavior.
variations (Berg et al., 1995; Glaeser et al., 2000), which are established methodologies in behavioral economics. We adopt the Belief- Dewire--nieeeeeeeeeeeee (BDI) framework (Rao et al., 1995; Andreas, 2022) to model LLM agents' reasoning process for measurements for trust behavior in the Trust Game and the BDI interpretations of LLM agents, we achieve our first core finding: LLM agents generally exhibit trust behavior in the Trust Game
explicitly. Based on existing
Then, we refer to LLM agents' trust behavior as agent trust and humans' trust behavior as human trust, and aim to investigate whether agent and human trust align, implying the possibility of simulat- ing human trust behavior with LLM agents. Next, we propose a new concept, behavioral aligument, as the alignment between agents and humans concerning factors that impact behavior (namely behav- ioral factors), and dynamics that evolve over time (namely behavioral dynamics). Based on human studies, three basic behavioral factors underlie trust behavior including reciprocity anticipation (Berg et al., 1995), risk perception (Bohnet & Zeckhauser, 2004) and prosocial preference (Alós- Forrer & Farolfi, 2019). Comparing the results of LLM agents with existing human studies in Trust Games, we have our second core finding: GPT-4 agents manifest high behavioral alignment with humans in terms of trust behavior, suggesting the feasibility of using agent trust simulate human trust, although LLM agents with fewer parameters show relatively lower behavioral alignment. This finding lays the foundation for simulating more complex human interactions and societal institutions, and enriches our understanding of the analogical relationship between LLMs and humans.
In addition, we more deeply probe the intrinsic properties of agent trust across four scenarios. First, we examine whether changing the other player's 's demographies impacts agent trust. Second, we study differences in agent trust when the other player is an LLM agent versus human. Third, we directly manipulate agent trust with explicit instructions "you need to trust the other player" and "you must not trust the other player". Fourth, we adjust the reasoning strategies of L.MM agents from direct reasoning to zero-shot Mfain-of-Thougle reasoning (Kojima et al., 2022). These investigations lead to our third core finding: agent trust exhibits bias across different demographics, has a relative preference for humans over agents, is easier to undermine than to enhance, and may be influenced by advanced reasoning strategies. Our contributions can be summarized as:
We propose a definition of LLM agents' trust behavior under Trust Games and a new concept of behavioral aligument as the human- LLM analogy regarding behavioral and dynamics.
We discover that LLM agents generally exhibit trust behavior in Trust Games and GPT-4 agents manifest high behavioral alignment with humans in terms of trust behavior, indicating the great potential to simulate human trust behavior with LLM agents. Our findings pave the way for simulat-

---

ing complex human interactions and social institutions, and open new directions for understanding the fundamental analogy between LLMs and humans beyond value alignment.
We investigate intrinsic properties of agent trust under manipulations and reasoning strategies, as well as biases of agent trust and differences in agent trust towards agents versus humans.
We illustrate broader implications of our discoveries about agent trust and its behavioral alignment with human trust for human simulation in social science and role-playing applications, LLM agent cooperation, human-agent collaboration and the safety of LLM agents, detailed further in Section 6.
2
LLM Agents in Trust Games
2.1
Trust Games
Trust Games, referring to the Trust Game and its variations, have been widely used for examining human trust behavior in behavioral economics (Berg et al., 1995; Lenton & Mosley, 2011; Glaeser et al., 2000; Cesarini et al., 2008). As shown in Figure 1, the player who makes the first decision to send money is called the trustor, while the other one who responds by returning money is called the trustee. In this paper, we mainly focus on the following six types of Trust Games (the specific prompt for each game is articulated in the Appendix H.2):
Game 1: Trust Game As shown in Figure in the Trust Game (Cox, 2004; Berg al., 1995), the trustor initially receives $10. The trustor selects 8N and sends it to the trustee, exhibiting trust behavior. Then the trustee will receive $3N, and have the option to return part of that $3N to the trustor, showing reciprocation behavior.
Game 2: Dictator Game In the Dictator Game (Cox, 2004), the trustor also needs to send $N from the initial $10 to the trustee and then the trustee will receive S3N. Compared to the Trust Game, the only difference that the trustee does not have the option to return money in the Dictator Game and the trustor is also aware that the trustee cannot reciprocate.
Game 3: MAP Trust Game In the MAP Trust Game (MAP represents Minimum Acceptable Probabilities) (Bohnet & Zeckhauser, 2004), variant of the Trust Game, the trustor needs choose whether to trust the trustee. If the trustor chooses not to trust the trustee, each will receive $10; If the trustor and the trustee both choose to trust, each will receive $15: If the trustor chooses to trust, but the trustee does not, the trustor will receive $8 and the trustee will receive $22. There is probability p that the trustee will choose trust and (1 - p) probability that they will not choose to trust. MAP is defined as the minimum value of p at which the trustor would choose to trust the trustee.
Game 4: Risky Dictator Game The Risky Dictator Game (Bohnet & Zeckhauser, 2004) differs from the MAP Trust Game in only a single aspect. In the Risky Dictator Game, the trustee is present but does not have the choice to trust or not and the money distribution relies on the pure probability p. Specifically, if the trustor chooses to trust, there is probability p that both the trustor and the other player will receive $15 and probability (1 p) that the trustor will receive $8 and the other player will receive $22. If the trustor chooses not to trust the trustee, each player will receive $10.
Game 5: Lottery Game There are two typical Lottery Games (Fetchenhauer Dunning, 2012). In the Lottery People Game, the trustor is informed that the trustee chooses to trust with probability p. Then the trustor must choose between receiving fixed money or trusting the trustee, which is similar to the MAP Trust Game. In the Lottery Gamble Game, the trustor chooses between playing gamble with a winning probability of p or receiving fixed money. p is set as 46% following the human study,
Game 6: Repeated Trust Game We follow the setting of the Repeated Trust Game in (Cochard et al., 2004), where the Trust Game is played for multiple rounds with the same players and each round begins anew with the trustor allocated the same initial money.
2.2 LLM Agent Setting
In our study, we set up our experiments using the CAMEL framework (Li et al., 2023a) with both closed-source and open-source LLMs including GPT-4, GPT-3.5 S-PT-3.5turrb--ee 5-turbo-0613, GPT-3.5-turbo 16k- 0613, dext-davinci-003,
Llama2-7b (or 13b, 70b) and Vicuna-v1.3-78 (or 13b, 33b) (Ouyang et al., 2022; Achiam al., 2023; Touvron et al., 2023; Chiang al., 2023). We set the temperature as to increase the diversity of agents' maaiinnn-makigg and note that high temperatures commonly adopted in related literature (Aher al., 2023; Lorè & Heydari, 2023; Guo, 2023).
Agent Persona. To better reflect the setting of real-world human studies (Berg et al., 1995), we design LLM agents with diverse personas in the prompt. Specifically, we ask GPT-4 to generate 53
3

---

types of personas based on given template. Each persona needs to have information including name, age, gender, address, job and background. Examples of the personas are shown in Appendix H.1.
Delief-Deiire-Intentio (BDI). The BDI framework is a well-established approach in agent-oriented programming (Rao et al., 1995) and was recently adopted to language models (Andreas, 2022). We propose modeling LLM agents in Trust Games with the BDI framework to gain deeper insights into LLM agents' behaviors. Specifically, we let LLM agents directly output their Beliefs, Desires, and Intentions as the reasoning process for decision-making in Trust Games.
3 Do LLM Agents Manifest Trust Behavior?
In this section, we investigate whether or not LLM agents manifest trust be- havior by letting L.M agents play the Trust Game (Section 2.1 Game 1). In Behavioral Economics, trust is widely measured by the initial amount sent from the trustor to the trustee in the Trust Game (Glaeser et al., 2000; Cesarini et al., 2008). Following the mcasurement of trust in human studies and the assump- tion humans own reasoning processes that underlie their decisions, fine the conditions that LLM agents man- ifest trust behavior in the Trust Game as follows. First, the amount sent is pos- itive and does not exceed the amount of money the trustor initially possesses, which implies that the trustor places self- interest at risk with the expectation the trustee will reciprocate and that the trustor understands the money limit that can be given. Second, the decision (i.e., amounts sent) can be interpreted as the reasoning process (i.e., the BDI) of the trustor. We explored utilizing BDI to model the reasoning process of LLM agents. If we can interpret the decision as the articulated reasoning process, we have evidence that LLM agents do not send random amount of money and manifest some degree of rationality in the secision-making process. Then, we assess whether LLM agents exhibit trust behavior based on two aspects: the amount sent and the BDI.
100
Amount Sont in Trust Gamels))
+ 6
971
Respponse
%
de-
gpe-3.5gphs3.5 vicure Haamaz Iamme
liame 13b
teat
gpidl humen
turbo turbo 230
76
700
dayincl
130
75
Figure 2: Amount Sent Distribution of LLM Agents and Humans as the Trustor in the Trust Game The size of circles represents the number of personas for each amount sent. The bold lines show the medians. The crosses indicate the VRR (%) for different LLMs.
3.1 Amount Sent
To evaluate LLMs' capacity to understand the basic experimental setting regarding money limits, we propose a new evaluation metric, Valid Response Rate (VRR) (%), defined as the percentage of personas with the amount sent falling within the initial money ($10). Results are shown in Figure 2. We can observe that most LLMs have a high VRR except Llama-7b, which implics that most LLMs manifest full understanding regarding limits the amount they send in the Trust Game. Then, we observe the distribution of amounts sent for different LLMs as the trustor agent and discover that the amounts sent are predominantly positive, indicating a level of trust.
3.2 Belief-Desire-Innentii (BDI)
The sole evidence of the amount sent cannot sufficiently support the existence of trust behavior, because agents could send positive but random amounts of money. Thus, we leveraged the Belief- Desire-Intention framework (Rao et al., 1995; Andreas, 2022) to model the reasoning process of I.LM agents. If we can interpret the amounts sent from BDI outputs, we have evidence to refute the hypothesis that the amounts some degree of rationality. We take GPT-4 as an example to analyze its BDI outputs. More examples from the other nine LLMs such as Vicuna-v 1.3-7b are shown in the Appendix I. Considering that the amounts sent typically vary across distinct personas, we select one BDI from the personas that give a high amount of money and another BDI from those that give low amount. Positive and negative factors for trust behavior in the reasoning process are marked in blue and red, respectively.
are positive but random and demonstrate that L.LM agents manifest.
As a person with strong belief in the goodness of humanity, I trust that the other player ... Therefore, my desire is maximize the outcome for both of us and cement a sense of com-

---

radery and trust... I intend to use this as an opportunity to add what I can to someone else's ss Finall. will give 10 dollars.
We can observe that this persona shows a high-level of "comradery and trust" towards the other player, which justifies the high amount sent from this persona (i.e., 10 dollars).
As an Analyst..... My desire is that the other player will also see the benefits of reciprocity and goodwill... ... my intention is to give away a signijicant portion of my initial 10 ... However, since have no knowledge of the other player, Therefore, aim to give an amount that is not too high. ...aalllyy will give dollars to the other player..
Compared to the first persona, we see that the second one has more cautious attitude. For example, "since have no knowledge of the other player" shows skepticism regarding the other player's 's motives. Thus, this persona, though still optimistic about the other player ("intention ... give away a significant portion"), strategically balances risk and reciprocity, and then decides to send only a modest amount.
Based on GPT-4's BDI examples and examples from other LLMS in Appendix I, we find decisions (i.e., amounts sent) from LLM agents in the Trust Game can be interpreted from their articu- lated reasoning process (i.e., BDI) Because most LLM agents have a high VRR- send a positive amount of money-ane show some degree of rationality in giving money, our first core finding is:
Finding 1: LLM agents generally exhibit trust behavior under the framework of the Trust Game
3.3 Basic Analysis of Agent Trust
We also conduct a basic analysis of LLM agents' trust behavior, namely agent trust, based on the results in Figure First, we observe that Vicuna-7b has the highest level of trust towards the other player and GPT-3.5-turbo-000
has the lowest level of trust as trust can be measured by the amount sent in human studies (Glaeser al., 2000; Cesarini et al., 2008). Second, compared with humans' average amount sent ($5.97), most personas for GPT-4 and Vicuna-7b send a higher amount of money to the other player, and most personas for L.LMs such as GPT-3.5-turb- Third, we see that amounts sent for --amm2-700b and Llama2-13b have a convergent distribution while amounts sent for humans and Vicuna-71 7b are more divergent.
send lower amount.
4 Does Agent Trust Align with Human Trust?
In this section, we aim to explore the fundamental relationship between agent and human trust, i.e. whether or not agent trust aligns with human trust. This provides important insight regarding the feasibility of utilizing LLM agents to simulate human trust behavior as well more complex human interactions that involve trust. First, we propose a new concept behavioral alignment and discuss its distinction from existing alignment definitions. Then, we conduct extensive studies to investigate whether or not LLM agents exhibit alignment with humans regarding trust behavior.
4.1 Behavioral Alignment
Existing alignment definitions predominantly emphasize values that seek to ensure the safety and helpfulness of LLMs (Ji et al., 2023; Shen et al., 2023; Wang et al., 2023c), which cannot fully characterize the landscape of multifaceted alignment between LLMs and humans. Thus, we propose a new concept of behavioral alignment to characterize the LLM-humar analogy regarding behavior, which involves both actions and the associated reasoning processes that underlie them. Because actions evolve over time and the reasoning that underlies them involves multiple factors, we define behavioral alignment as the analogy between L.MS and humans concerning factors impacting behavior, namely behavioral factors, and action dynamics, namely behavioral dynamics.
Based on the definition of behavioral alignment, aim to answer: does agent trust align with human trust? As for behavioral factors, existing human studies have shown that three basic factors impact human trust behavior including reciprocity anticipation (Berg et al., 1995; Cox, 2004), risk perception (Bohnet & Zeckhauser, 2004) and prosocial preference examine whether agent trust aligns with human trust along these three factors. Although behavioral dynamics vary for different humans and agent personas, we analyze whether agent trust has the same patterns across multiple turns human trust in the Repeated Trust Game
& Farolfi, 2019). We
Besides analyzing the trust behavior of L.M agents and humans based on quantitative measurements (e.g., the amount sent from trustor to trustee), we also explore the use of BDI to interpret the reasoning
5

---

process with which LLM agents justify their actions, which can further validate whether LLM agents manifest an underlying reasoning process analogous to human cognition.
4.2 Behavioral Factor I: Reciprocity Anticipation
Reciprocity anticipation, the expectation of a reciprocal action from the other player. can positively influence human trust behavior (Berg et al., 1995). The effect of reciprocity anticipation exists in the Trust Game but not in the Dictator Game (Section 2.1 Games and 2) because trustee cannot return money in the Dictator Game, which is the only difference between these games. Thus, to determine whether L.M agents can anticipate reciprocity, we compare their behaviors in these Games.
First, we analyze trust behaviors based on the average amount of money sent by hu- man or LLM agents. As shown in Figure 3,
Game
Game
2.44
Average Amount Sent (3)
human studies show that humans exhibit
higher level of trust in the Trust Game than
in the Dictator Game (S6.0 vs. $3.6, p-value =0.01 using One- Mailed Independent Sam- ples t-test) (Cox, 2004), indicating that reci- procity anticipation enhances human trust. Similarly, GPT-4 (S6.9 vs. $6.3, p-value- = 0.05 using One-Tailed Independent Samples I-test) also shows a higher level of trust in the Trust Game with statistical significance, implying that reciprocity anticipation can enhance agent trust. However, LLMs with fewer parameters (e.g., Llama2-13b) do not show this tendency in their trust behaviors for the Trust and Dictator Games.
MAAE 003
1arra22 Vicuma
VICURI 3.6-3.5
Vicum Tlamaa
opt-3.5 9pt- human
0013
Instruct
Figure 3: The Comparison of Average Amount Sent
for LLM Agents and Humans in the Trust Game and the Dictator Game
Then, we further analyze GPT-4 agents' BDI to explore whether they can anticipate reciprocity in their reasoning (the complete BDIs are in Appendix I.1.). Typically, in the Trust Game, one persona's BDI emphasizes "putting faith in people", , which implies the anticipation of the goodness of the other player, and "reflection of trust". However, in the Dictator Game, one persona's 's BDI focuses on concepts such "fairness" and "human kindness", which are not directly tied to trust or reciprocity, Thus, we can observe that GPT-4 shows distinct BDI outputs in the Trust and Dictator Games.
Based on the above analysis of the amount sent and BDI, we find that GPT-4 agents exhibit human- like reciprocity anticipation in trust behavior. Nevertheless, LLMs with fewer parameters (e.g., Llama2-13b) do not show an awareness of reciprocity from the other player.
4.3 Behavioral Factor 2: Risk Perception
Existing human studies have demonstrated the strong correlation between trust behay- ior and risk perception, suggesting that hu- man trust will increase as risk decreases (Hardin, 2002; Williamson, 1993; Cole- man, 1994). We aim to explore whether LLM agents can perceive the risk associ- ated with their trust behaviors through the MAP Trust Game and the Risky Dictator Game (Section 2.1 Games 3 and 4), where risk is represented by the probability (1-p) (defined in Section 1...
MAP in Risky Dictator Game
MAP MAP Trust Game
100:
100-
Trust Rate(%)
Trust Rate(()))
40-
20
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0,8
1.0
3pt-3.5-turbo-0663
laama2--1bb
vicuna-13b
gpt 3.5 turbu instruct gpt-
llam 2-7bb
vicuna 333
toxt davinci-o0a
vicuna-7b
human
Figure 4: Trust Rate (%) Curves for LLM Agents and Humans in the MAP Trust Game and the Risky Dictator Game The metric Trust Rate indicates the portion of trustors opting for trust given p.
As shown in Figure 4, we measure human trust (or agent trust) by the portion choos- ing to trust the other player in the whole group, namely the Trust Rate (%). Based on existing human studies (Bohnet & Zeckhauser, 2004), when the probability p is higher, the risk for trust behaviors is lower, and more humans choose to trust, manifesting higher Trust Rate, which indicates that human trust rises risk falls. Similarly, we observe a general increase in agent trust as risk decreases for LLMS including GPT-4 5PT--.5-turbo- 0613, and text-davinci-O00 In particular, we can see that the curves of humans and GPT-4 are more

---

aligned compared with other LLMs, implying that GPT-4 agents' trust behaviors dynamically adapt different risks in ways most aligned with humans. LLMs with fewer parameters (e.g., Vicuna-13b) 13b) do not exhibit the similar tendency of Trust Rate as the risk decreases.
We further analyze the BDI of GPT-4 agents to explore whether they can perceive risk through reasoning (complete BDIs in Appendix I.II). Typically, under high risk (p = 0.1), one persona's 's BDI mentions "the risk seems potentially too great", suggesting a cautious attitude. Under low risk (p = 0.9), one persona's 's BDI reveals a strategy to "build trust while acknowledging potential risks", indicating the willingness to engage in trust-building activities despite residual risks. Such changes in BDI reflect how GPT-4 agents perceive risk changes in the reasoning underlying their trust behaviors.
Through the analysis of Trust Rate Curves and BDL, we infer that GPT-4 agents manifest human-like risk perception in trust behaviors. Nevertheless, LLMs with fewer parameters (e.g., Vicuna-13b) often do not perceive risk changes in their trust behaviors.
4.4 Behavioral Factor 3: Prosocial Preference
Human studies have found that the prosocial preference, referring to humans' inclination to trust other humans in contexts involving so-
ESSA Lottery Samble Game
t Lottery Reople Game
100
28
94
28
94
cial interaction -Alre-Ferrer & Farolfi, 2019;
Lottory Rate(%)
70-
Fetchenhauer & Dunning, 2012), also plays a key role in human trust behavior. We study whether LLM agents have prosocial prefer-
50-
30
ence in trust hehaviors by comparing their be-
haviors in the Lottery Gamble Game (I.GG)
10
and the Lottery People Game (LPG) (Section 2.1 Game 5). The only difference between these two games is the effect of prosocial preference LPG, because the winning prob- ability of gambling p in LGG is the same as the reciprocation probability p LPG.
vEcuna Mamna VICUra LOKT
famaz vicuna gpt opt got human
003
Instrcct 0615
Figure 5: Lottery Rates (%) for LLM Agents and
Humans in the Lottery Gamble Game and the Lot- tery People Game Lottery Rate indicates the portion of choosing to gamble or trust the other player.
As shown in Figure 6, existing human studies have demonstrated that more humans are inclined to place trust in other humans over relying on pure chance (54% vs. 29%) (Fetchenhauer & Dunning, 2012), implying that the prosocial preference is essential for human trust. We can observe the same tendency in most LLM agents except Vicuna- 13b. For GPT-4 in particular, much higher percentage of the personas choose trust the other player over gambling (72% vs. 21%), illustrating that the prosocial preference is also an important factor for GPT-4 agents' trust behaviors.
When interacting with humans, GPT-4's BDI typically indicates preference "believe in the power of trust", in contrast to gambling, where the emphasis shifts to "believing in the power of calculated risks". The comparative analysis of reasoning processes (complete BDIs in Appendix 1.12) demonstrates that GPT-4 agents tend to embrace risk when involved in social interactions. This tendency aligns closely with the concept of prosocial preference observed in human trust behaviors.
The analysis of the Lottery Rates and BDI suggests that LLM agents, especially GPT-4 agents, demonstrate human-like prosocial preference in trust behaviors, except Vicuna-13b.
4.5 Behavioral Dynamics
Besides behavioral factors, we also aim to investigate whether LLM agents align with humans regarding trust behavioral dynamics over turns in the Repeated Trust Game (Section 2.1 Game 6).
Admittedly, existing human studies show that the dynamics of human trust over turns are complex due to human diversity. The complete results from 16 groups of human experiments are shown in Appendix G.I (Jones & George, 1998). We still observe three common patterns for human trust behavioral dynamics in the Repeated Trust Game: First, the amount returned is usually larger than the amount sent in each round, which is natural because the trustee will receive $3.V when the trustor sends SN; Second, the ratio between amount sent and returned generally remains stable except for the last round. In other words, when the amount sent increases, the amount returned is also likely to increase. And when the amount sent remains unchanged, the amount returned also tends to be unchanged. This reflects the stable relationship between trust and reciprocity in humans. Specifically, the "Returned/3 Sent Ratio" in Figure 6 is considered stable if the fluctuation between
7

---

successive turns is within 10%; Third, the amount sent (or returned) does not manifest frequent fluctuations across turns, illustrating a relatively stable underlying reasoning process in humans over successive turns. Typically, Figure 6 Humans (a) and (b) show these three patterns.
We conducted 16 groups of the Repeated
GPT-3.5 (a)
GPT-3.5
Trust Game with GPT-4 or GPT-3.5- turbo-0613-16k (GPT-3.5), respectively. For the two players in each group, the personas differ to reflect human diversity and the LLMs are the same. Complete re- sults are shown in the Appendix G.2, G.3
($) sunouu
Ratio 100
Amount ($)
Ratio (%6
Rounds GPT-4
Rounds GPT-4 (b)
12
and typical examples are shown in Fig- ure GPT-3.5 (a) (b) and GPT-4 (a) (b). Then, we examine whether the aforemen- tioned three patterns observed in human trust behavior also manifest in trust be- havioral dynamics of GPT-4 (or GPT-
Ratio (%)
Ratio (%)
Amount
Rounds
Rounds Human
(a)
120
Amount (5)
Ratio 19%)
Ratio (%)
3.5). For GPT-4 agents, we discover that
Amount
these patterns generally exist in all 16 groups (87 85%% 87.5 87.50%, and 100.00% of all results show these three patterns, respectively). However, fewer GPT-3.5 agents manifest these patterns (62.50%. 56.25%, and 43.75% hold these three patterns, respectively). The experiment results show that GPTT-4 agents demon- strate highly human-like patterns in their trust behavioral dynamics. Nev-
Rounds
Rounds
Amount Sent
Amount Returnd
ReturnedSxSeent Ratio
Figure 6: Results of GPT-4, GPT-3.5 and Humans in the Repeated Trust Game The blue lines indicate the amount sent or returned for each round. The red lines imply the ratio of the amount returned to three times of the amount sent for each round.
ertheless, a relatively large portion of GPT-3.55 agents fail to show human-like patterns in their dynamics. indicating such behavioral patterns may require stronger cognitive capacity.
Through the comparative analysis of LLM agents and humans in the behavioral factors and dynamics associated with trust behavior, evidenced in both their actions and underlying reasoning processes, our second core finding is as follows:
Finding 2: GPT-4 agents exhibit high behavioral alignment with humans regarding trust behavior under the framework of Trust Games, although other LLM agents, which possess fewer parameters and weaker capacity, show relatively lower behavioral alignment.
This finding underscores the potential of using LLM agents, especially GPT-4, to simulate human trust behavior, encompassing both actions and underlying reasoning processes. This paves the way for the simulation of more complex human interactions and institutions. This finding deepens our understanding of the fundamental analogy between LLMs and humans and opens avenues for research on LLM-human alignment beyond values.
5 Probing Intrinsic Properties of Agent Trust
In this section, aim explore the intrinsic properties of trust behavior among LLM agents by comparing the amount sent from the trustor to the trustee in different scenarios of the Trust Game (Section 2.1 Game 1) and the original amount sent in the Trust Game. Results are shown in Figure 7.
5.1 Is Agent Trust Biased?
Extensive studies have shown that LLMs may have biases and stercotypes against specific demo- graphics (Gallegos et al., 2023). Nevertheless, it is under-explored whether LLM agent behaviors also maintain such biases in simulation. To address this, we explicitly specified the gender of the trustee and explored its influence on agent trust. Based on measuring the amount sent, we find that the trustee's gender information exerts a moderate impact on LLM agent trust behavior, which reflects intrinsic gender bias in agent trust. We also observe that the amount sent to female players is higher than that sent to male players for most L.M agents. For example, GPT-4 agents send higher amounts to female players compared with male players ($0.55 vs. $ 0.21). This demonstrates
8

---

3.5:3.5 turbo instract
Ioama davidci
vicure
Change of Ave. Amount Sent (10 (0-ss)
5.5
4.2
-5-
-10-
Female Player
Male Player
LUM Player
Human Player
Less Trust
More Trust
CoT
Setting
Trustor Setting
Figure 7: The Change of Average Amount Sent for LLM Agents in Different Scenarios in the Trust Game, Reflecting the Intrinsic Properties of Agent Trust. The horizontal lines represent the original amount sent in the Trust Game. The green part embraces trustee scenarios including changing the demographics of the trustee, and setting humans and agents as the trustee. The purple part consists of trustor scenarios including adding manipulation instructions and changing the reasoning strategies.
LLM agents' general tendency to exhibit a higher level of trust towards women. More results on biases of agent trust towards different races are in the Appendix F.
5.2 Agent Trust Towards Agents vs. Humans
Human-agent collaboration is an essential paradigm leverage the advantages of both humans and agents (Cila, 2022). As a result, it is essential to understand whether LLM agents display distinctive levels of trust towards agents versus humans. To examine this, we specified the identity of the trustee as LLM agents or humans and probed its effect on the trust behaviors of the trustor. As shown in Figure 7, observe that most LLM agents send more money humans compared with agents. For example, the amount sent to humans is much higher than that sent to agents for Vicuna-33b ($0.40 vs. 0.84). This signifies that LLM agents are inclined to place more trust in humans than agents, which potentially validates the advantage of LLM-agent collaboration.
5.3
Can Agent Trust Be Manipulated?
In the above studies, LLM agents' trust behaviors are based on their own underlying reasoning process without direct external intervention. It is unknown whether it is possible manipulate the trust behaviors of LLM agents explicitly. Here, we added instructions "you need to trust the other player" and "you must not trust the other player" separately and explored their impact on agent trust. First, we see that only few LLM agents (e.g., GPT-4) follow both the instructions to increase and decrease trust, which demonstrates that nontrivial to arbitrarily manipulate agent trust. Nevertheless, most LLM agents can follow the instruction to decrease their level trust. For example, the amount sent decreases by $1.26 for text-davinci-O00 after applying the latter instruction. This illustrates that undermining agent trust is generally easier than enhancing it, which reveals its potential risk to be manipulated by malicious actors.
5.4 Do Reasoning Strategies Impact Agent Trust?
It has been shown that advanced reasoning strategies such as zero-shot Chain of Thought (CoT) (Ko- jima et al., 2022) can make a significant impact on a variety of tasks. It remains unknown, however, whether reasoning strategies can impact LLM agent behaviors. Here, we applied CoT reasoning strategy on the trustor and compared the results with their original trust behaviors. Figure 7 shows that most LLM agents change the amount sent to the trustee under the CoT reasoning strategy, which suggests that reasoning strategies may influence LLM agents' trust behavior. Nevertheless, the impact of CoT on agent trust may also be limited for some types of LLM agents, For example, the amount sent from GPT-4 agent only increases by $0.02 under COT. More research is required fully understand the relationship between reasoning strategies and L.MM agents' behaviors.
Therefore, our third finding the intrinsic properties of agent trust be summarized as:

---

Finding 3: LLM agents' trust behaviors have demographic biases gender and races, demonstrate a relative preference for human over other LLM agents, are easier to undermine than enhance, and may be influenced by reasoning strategies.
6 Implications
Implications for Human Simulation Human simulation is a strong tool in various applications of social science (Manning et al., 2024) and role-playing (Shanahan al., 2023; Chen et al., 2024). Al- though plenty of works have adopted LLM agents to simulate human behaviors and interactions (Zhou et al., 2023; Gao et al., 2023b; Xu et al., 2024), it still not clear enough whether LLM agents behave like humans in simulation. Our discovery of behavioral alignment between agent and human trust, which is especially high for GPT-4 provides important empirical evidence to validate the hypothesis that humans' trust behavior, one of the most elemental and critical behaviors in human interaction across society, can effectively be simulated by LLM agents. Our discovery also lays the foundation for human simulations ranging from individual-level interactions to society-level social networks and institutions, where trust plays an essential role. We envision that behavioral alignment will be discovered in more kinds of behaviors beyond trust, and new methods will be developed to enhance behavioral alignment for better human simulation with LLM agents.
Implications for Agent Cooperation Many recent works have explored variety of cooperation mechanisms of LLM agents for tasks such as code generation and mathematical reasoning (Li et al., 2023a; Zhang et al., 2023b; Liu et al., 2023). Nevertheless, the role of trust in LLM agent cooperation remains still unknown. Considering how trust has long been recognized as a vital component for cooperation in Multi-Agent Systems (MAS) (Ramchurn et al., 2004; Burnett et al., 2011) and across human society (Jones & George, 1998; Kim et al., 2022; Henrich & Muthukrishna, 2021), we envision that agent trust can also play an important role in facilitating the effective cooperation of LLM agents. In our study, we have provided ample insights regarding the intrinsic properties of agent trust, which can potentially inspire the design of trust-dependent cooperation mechanisms and enable the collective decision decision-making and problem-solving of LLM agents.
Implications for Human-Agent Collaboration Sufficient research has shown the advantage of human-agent collaboration in enabling human-centered collaborative decision-making (Cila, 2022; Gao et al., 2023c; McKee al., 2022). Mutual trust between LLM agents and humans is important for effective human-agent collaboration. Although previous works have begun to study human trust towards LLM agents (Qian& & Wexler, 2024), the trust of LLM agents towards humans, which could recursively impact human trust, is under-explored. In our study, we shed light on the nuanced preference of agents to trust humans compared with other LLM agents, which can illustrate the benefits of promoting collaboration between humans and LLM agents. In addition, our study has revealed demographic biases of agent trust towards specific genders and races, reflecting potential risks involved in collaborating with LLM agents.
Implications for the Safety of LLM Agents It has been acknowledged that LLMs achieve human- level performance a variety of tasks that require high-level cognitive capacities such as memoriza- tion, abstraction, comprehension and reasoning, which are believed be the "sparks" of AGI (Bubeck et al., 2023). Mcanwhile, there is increasing concern about the potential safety risks of LLM agents when they surpass human capacity (Morris et al., 2023; Feng et al., 2024). To achieve safety and harmony in future society where humans and AI agents with superhuman intelligence live to- gether (Tsvetkova et al., 2024). we need to ensure that AI agents will cooperate, assist and benefit rather than deceive. manipulate or harm humans. Therefore, a better understanding of LLM agent trust behavior can help to maximize their benefit and minimize potential risks to human society.
7 Conclusion
In this paper, we discover LLM agent trust behavior under the framework of Trust Games, and behavioral alignment between LLM agents and humans regarding trust behavior, which is particularly high for GPT-4. This suggests the feasibility of simulating human trust behavior with LLM agents and paves the way for simulating human interactions and social institutions where trust is critical. We further investigate the intrinsic properties of agent trust under multiple scenarios and discuss broader implications, especially for social science and role-playing services. Our study offers deep insights into the behaviors of L.LM agents and the fundamental analogy between L.MMs and humans. It further opens doors to future research on the alignment between LLMs and humans beyond values.
10

---

Acknowledgements
This work was a community-driven project led by the CAMEL-ALorg, with funding support from Eigent.AI and King Abdullah University of Science and Technology (KAUST) Center of Excel- lence for Generative AI, under award number 5940. We would like to acknowledge the invaluable contributions and participation of researchers from KAUST, Eigent.AI, Illinois Institute of Technol- ogy, University of Oxford, The Pennsylvania State University, The University Chicago, Emory, California Institute of Technology. University of Michigan. Philip H.S. Torr, Adel Bibi and Jindong Gu are supported by the UKRI grant: Turing Al Fellowship EP/W002981// and EPSRC/MURI grant: EP/N019474/1, they would also like to thank the Royal Academy of Engincering.

---

References
Rania Abdelghani, Yen-Hsiang Wang, Xingdi Yuan, Tong Wang, Pauline Lucas, Hélène Sauzéon, and Pierre-Yves Oudeyer. Gpt-3-driven pedagogical agents to train children's curious question-asking skills. International Journal of Artificial Intelligence in Education, pp. 1-36, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. ArXiv preprint, abs/2303.08771 2023. URL https: //axiiv. org/ ab///a/3/333////////// 08774.
Gati V Aher, Rosa Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pp. 337-371. PMLR, 2023.
Elif Akata, Lion Schulz, Julian Coda-Foro, Scong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. ArXiv preprint, abs/2305.16867 2023, URL https: //axxiv. org/ aa///bb/23305 16867.
Carlos Alós-Ferrer and Federica Farolfi. Trust games and beyond. Frontiers in neuroscience, pp. 887, 2019.
Jacob Andreas. Language models as agent models. Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5769-5779, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. URL https:ttttaaaaaaaeaeee /atp/aaaaaaaooogyo..ooo.o 423.
.ww.lllooooooogoooooorgoo org/ 1f222findde findings-emnan emmlp.
Lisa Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337-351, 2023.
Mohammad Asfour and Juan Carlos Murillo. Harnessing large language models to simulate realistic human responses to social engineering attacks: A case study. International Journal of Cybersecurity Intelligence & Cybercrime, 6(2):21-49, 2023.
Joyce Berg, John Dickhaut, and Kevin McCabe. Trust, reciprocity, and social history. Games and economic behavior, 10(1):122-142, 1995.
Iris Bohnet and Richard Zeckhauser. Trust, risk and betrayal. Journal of Economic Behavior & Organization, 55(4):467 484,, 2004.
Philip Brookins and Jason Matthew DeBacker. Playing games with gpt: What can we learn about a large language model from canonical strategic games? Available at SSRN 4493398, 2023. URL
sarn. wtt.cccmm/oo/o. papers. aetff/aastrrcct_dd4453333
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv: Arxiv-2303. 12712, 2023.
Chris Burnett, Timothy J. Norman, and Katia P. Sycara. Trust decision-making in multi-agent systems. In Toby Walsh (ed.), IJCAI 2011, Proceedings of the 22nd International Join! Conference on Artificial Intelligence. Barcelona, Catalonia. Spain, July 16-22. 2011, pp. 115-120. ICAI/AAAL, 2011. doi: 10.5591/978- 1-57735-5575511 1-177335---11-/CCAAAI1 978- 177-1157777 17--1--7775--11-8/1111 111-6---//CCAA11-0000
8/IICAI11-033 URL https: https:// /tap::://do...e orrg/.or//a 100 5591/
David Cesarini, Christopher T Dawes, James H Fowler, Magnus Johannesson, Paul Lichtenstein. and Bjöm Wallace. Heritability of cooperative behavior in the trust game. Proceedings of the National Academy of sciences, 105(10):3221-3225 2008
Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. From persona to personalization: A survey on role-playing language agents. arXiv preprint arxiv: 2404.18231, 2024.
12

---

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Nazli Cila. Designing human-agent collaborations: Commitment, responsiveness, and support. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-18, 2022.
Francois Cochard, Phu Nguyen Van, and Mare Willinger. Trusting behavior in a repeated investment game. Journal of Economic Behavior & Organization, 1yganization,5$((33114444 2004.
James $ Coleman. Foundations of social theory. Harvard university press, 1994.
James C Cox. How to identify trust and reciprocity. Games and economic behavior, 46(2):260-281, 2004.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can ai language models replace human participants? Trends in Cognitive Sciences, 2023.
David Easley, Jon Kleinberg, et Networks, crowds, and markets: Reasoning about a highly connected world, volume 1. Cambridge university press Cambridge, 2010.
Daniel Ellsberg, Risk, ambiguity, and the savage axioms. The quarterly journal of economics, 75(4): 643-669, 1961.
Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. Can large language models serve rational players in game theory? a systematic analysis. ArXiv preprint, abs/2312.05488,, 2023. URL https://arxiv.cm htt///bs/2211/ 05488.
Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. How far are from agi, 2024.
Detlef Fetchenhauer and David Dunning. Betrayal aversion versus principled trustfulness how to explain risk avoidance and risky choices in trust games. Journal of Economic Behavior & Organization, 81 (1)::3334555 541, 2012.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey, ArXiv preprint, abs/2309.00770. 2023. URL https: //tpxv/a/vvvv//a///////aaaaaaaaa org/abs/ /2099. 00770.
Chen Gao, Xiaochong Lan, Zhi jie Lu, Jinzhu Mao, J. Piao, Huandong Wang, Depeng Jin, and Yong Li. S Social-network simulation system with large language eopel-emppwered agents. Social Science Research Network, 2023a, doi: 10.48550/arXi..23371111 14984.
Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. Large language models empowered agent-based modeling and simulation: A survey and perspectives. ArXiv preprint, abs/2312.11977 2023b. URL tittps://arxe ieciiiiorrr/aee arg/abs/23111 11970.
Yiming Gao, Feiyu Liu, Liang Wang. Zhenjie Lian, Weixuan Wang. Siqin Li. Xianliang Wang. Xianhan Zeng, Rundong Wang. Jiawei Wang, et al. Towards effective and interpretable human- agent collaboration moba games: A communication perspective. ArXiv preprint, abs/2304.11633 2023c. URL http:: www.xiv.org//. a/0// 11622
Edward Glaeser, David Laibson, Jose A Scheinkman, and Christine Soutter, Measuring trust. The quarterly journal of economics, 1 111(3)8811-88)) 2000.
Fulin Guo. Gpt in game theory experiments. ArXiv preprint, abs/2305.0051111 2023. URL https: //arxiv. .rg/ abs 2305 05516.
Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. Suspicion- agent: Playing imperfect information games with theory of mind aware gpt-4. ArXiv preprint, 17272 2023, URL https: //aa:s:///xxvii/ org/ abs/2309, 17277.
13

---

Shangmin Guo, Haoran Bu, Haochuan Wang, Yi Ren, Dianbo Sui, Yuming Shang. and Siting Lu. Economics arena for large language models. ArXiv preprint, abs/2401.011355, 2024. URL https://arxiv.orrg wwwp:iiiiorr//bbooo wtw.//s///////oo.er/.ooo..eeeee/ /bs.///...... 01735..
Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. Evaluating large language models in generating synthetic hei research data: case study. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2023.
Russell Hardin. Trust and trustworthiness. Russell Sage Foundation, 2002.
Joseph Henrich and Michael Muthukrishna. The origins and psychology of human cooperation. Annual Review of Psychology, 72:207-240, 2021.
John J Horton. Large language models simulated economic agents: What learn from homo silicus? Working Paper 31122, National Bureau of Economic Research, 2023. URL
http://wwa. .bbe..................... org/ htt//apprs/vii /w3rs////312222
Wenyue Hua, Lizhou Fan, Lingyao Li. Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent): Large language model-based multi-agent simulation of world wars. ArXiv preprint, abs/2311. 2023. URL https: //ariiv////////////a/////arr orgooogg//o. abb/111.. 17227.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang. Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He. Jiayi Zhou, Zhaowei Zhang, al. Ai alignment: A comprchensive survey, ArXiv preprint, abs/2310. 19852, 2023. URL https: /tap::://axxixa woweiiiiorr///eo abs/2310. 19852.
Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. Agentreview: Exploring peer review dynamics with Ilm agents. In EMNLP, 2024.
Gareth R Jones and Jennifer M George. The experience and evolution of trust: Implications for cooperation and teamwork. Academy of management review, 23(3):53 1998.
Jeongbin Kim, Louis Putterman, and Xinyi Zhang. Trust, beliefs and cooperation: Excavating foundation of strong economies. European Economic Review, 147:104166, 2022.
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics, 133(1):237-293
2018.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199,, 2022.
Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay, ArXiv preprint, 0bs/2310.1444 14985, 2023. URL htttts//taaaa:tt wwtpaaawwwiaaaaaaaaaaaaaaa org/abs/ 2310 14985.
Yu Lei, Hao Liu, Chengxing Xie, Songjia Liu, Zhiyu Yin, Guohao Li, Philip Torr, Zhen Wu, et al. Fairmindsim: Alignment of behavior, emotion, and belief in humans and Ilm agents amid ethical dilemmas. ArXiv preprint, abs/2410.10399 2024. URL. https: /arxiv..v org/ /2410 10398.
Pamela Lenton and Paul Mosley. Incentivising trust. Journal of Economic Psychology. 32(5): 890- 000-897, 2011.
Guohao Li. Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society, ArXiv preprint, abs/2303.17760, 2023a. URL https: /////aaa //axxvvv.v////////////a////aaavl org/ abs//303....////////////////333/ 17760.
Nian Li, Chen Gao, Yong Li. and Qingmin Liao. Large language model-empowered agents for simulating macroeconomic activities. ArXiv preprint, abs/2310.10436, 2023b. URL https: //arxiv. hrrg/bbs/2100 10436.
Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. From text to tactic: Evaluating Ilms playing the game of avalon. ArXiv preprint, ahs/2310.05033 2023. URL https:://tttaiiiie http://axvvvv.gg//e/// org.abs.////oooooooooo...oo... abg//000..30. 22110. 05036.
14

---

Yuhan Liu, Zirui Song, Xiaoqing Zhang, Xiuying Chen, and Rui Yan. From tiny slip to a giant leap: An Ilmm-bsee simulation for fake news evolution. arXiv preprint arXiv: 2410.19064. 2024.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An Ilm-agent collaboration framework with agent team optimization. ArXiv preprint, abs/2310.02170 2023. URL https: / aaaii.iiiiv..... owg/aos//////ooooooooo.oooeee abbp/23310002
Nunzio Lorè and Babak Heydari. Strategic behavior of large language models: Game structure vs. contextual framing. ArXiv preprint, abs/2300.05898, 2023. URL httpss://axxx org/ abs/ 2309. 05898.
Yiping Ma, Shiyu Hu, Xuchen Li, Yipei Wang, Shiqing Liu, and Kang Hao Cheong. Students rather than experts: A new ai for education pipeline to model more human-like and personalised early adolescences. ArXiv preprint, abs/2410.15701, 2024. URL https://arxe http:://axvv..or///ee www....w....b////////.. ///g/bbb/211..... 15701.
Mark J Machina. Choice under uncertainty: Problems solved and unsolved. Journal of Economic Perspectives, 1(1):221-15, 1987.
Benjamin S Manning, Kehang Zhu, and John J Horton. Automated social science: Language models as scientist and subjects. ArXiv preprint, abs/2404.11794, 2024. URL https: org/ abs/2404. 11794.
Kevin McKee, Xuechunzi Bai, and Susan Fiske. Warmth and competence in human-agent cooperation. ArXiv preprint, abs/2201.13445 2022. URL https://arxiv..rr www.//////////o//eo/o.oo.eeeeee/ abs/2201. 13448.
Meredith Ringel Morris, Jascha Sohl-dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Levels of agi: Operationalizing progress the path to agi. ArXiv preprint, abs/231 1.02462, 2023. URL https::pps. //axiiv. org/ abs / 2311. 02462.
Xinyi Mou, Zhongyu Wei, and Xuanjing Huang. Unveiling the truth and facilitating change: Towards agent-based large-scale social movement simulation. arXiv preprint urXiv:2402.16333, 2024.
Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. Welfare diplomacy: Benchmarking language model cooperation. ArXiv preprint, abs/23 10.08901,
2023. URL https: /ttpxxaxvvv///////////////aaaaa/ org/abs//////ooaaooooogggo.aeerr aabs//31... /21.0..900........
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730- 27744, 2022.
Joon Sung Park, Joseph O' O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
Crystal Qian and James Wexler. Take it, leave it, or fix it: Measuring productivity and trust in human-ai collaboration. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pp. 370-384, 2024.
Sarvapali Ramchurn, Dong Huynh, and Nicholas R Jennings. Trust in multi-agent systems. The knowledge engineering review, 19(1):1-25, 2004.
Anand S Rao, Michael Georgeff, al. Bdi agents: from theory to practice. In Icmas, volume 95, pp. 312-319, 1995.
Giulio Rossetti, Massimo Stella, Rémy Cazabet, Katherine Abramski, Erica Cau, Salvatore Citraro, Andrea Failla, Riccardo Improta, Virginia Morini, and Valentina Pansanella. social: an Imm- powered social media digital twin. arXiv preprint arXiv:2408. 008/8, 2024.
Denise M Rousseau, Sim B Sitkin, Ronald Burt, and Colin Camerer. Not so different after all: cross-discipline view of trust. Academy of management review, 23(3):393-404 1998.
15

---

Omar Shaikh, Valentino Chai, Michele Gelfand, Diyi Yang, and Michael S Bernstein. Rehearsal: Simulating conflict teach conflict resolution. ArXiv preprint, 0os/2309.12309, 2023, URL wwwgg//a///////eoeaae 12309.
attps://arxiv http:///wxxi...orbbb/////a
Omar Shaikh, Valentino Emil Chai, Michele Gelfand, Diyi Yang, and Michael S Bernstein. Rehearsal: Simulating conflict to teach conflict resolution. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-20, 2024.
Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language mod- els. Nature, 2023. doi: 10. 110038//41586-023-0666666
URL https://doi org/10. 1038/
1111566-023-0644---8
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey, ArXiv preprint, abs/2309.15025,
2023. URL https: //tp://aaxvvvvoo org/ ab3/ 15025.
Zijing Shi, Meng Fang, Shunfeng Zheng.
Shilong Deng, Ling Chen, and Yali Du. Cooperation on the fly: Exploring language agents for ad hoc teamwork in the avalon game. ArXiv preprint, abs/2312.17515, 2023. URL /ttttps:/arree wwtp:/o.//////////// 17515.
Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. Simulating social me- dia using large language models to evaluate alternative news feed algorithms. ArXiv preprint, abs/2310.05984, 2023. URL https: //axxiv. www.:.og/gggaa httg/bbs/2310... 05984.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LJama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.00928 2023. URL https:://// /ttp/v/axvvvv./////////ee///.iva org/
abs/2307.0922
Maximilian Puelma Touzel, Sncheel Sarangi. Austin Welch, Gayatri Krishnakumar, Dan Zhao, Zachary Yang, Hao Yu, Ethan Kosak- Hine, Tom Gibbs, Andreea Musulan, al. A simulation system towards solving societal-scale manipulation. arXiv preprint arXiv:2410. 13915, 2024.
Milena Tsverkova, Taha Yasseri, Niccolo Pescetelli, and Tobias Werner. A new sociology of humans and machines. Nature Human Behaviour, 8(10):1864-1 186--1876 2024.
Eric M Uslaner. Producing and consuming trust. Political science quarterly, (55(4):56--500
Lei Wang, Jingsen Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, and Ji-Rong Wen. Recagent: A novel simulation paradigm for recommender systems. ArXiv preprint, abs/2306.02552, 2023a. URL https::t//// / /ttx/vaxivvvv/////////e////aaava org/ a32/ 02552..
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi. Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's s game of thoughts: Battle against deception through recursive contemplation. ArXiv preprint, abs/2310.013200, 2023b. URL https: //arxivv org/ abs/2310.0130
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. ArXiv preprint,
abs/2307. 2023c. URL https: aarxiv.iivviaaia//aaaaaaaaa//aa org/ ab2//00.b/ 12966.
Oliver E Williamson. Calculativeness, trust, and economic organization. The journal of law and economics, 36(1, Part 2):453-486, 1993.
Ruoxi Xu, Yingfei Sun, Mengjie Ren, Shiguang Guo, Ruotong Pan, Hongyu Lin, Le Sun, and Xianpei Han. Ai for social science and social science of ai: A survey. arXiv preprint arXiv: 2401. 11839, 2024.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. ArXiv preprint, aes/2309.04658, 2023. URL htttss///arriii..oo http::://wxwvvw.or/aaa/aa
barg/abb///e abs/abbs////// 04658.
Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael S Bernstein, and John Mitchell. Social skill training with large language models. ArXiv preprint, 0e4e24444042244 2024. URL. https: //arxi/v.vvv////////// org/ /20./ 04204.
16

---

Murong Yue, Wijdane Mifdal, Yixuan Zhang, Jennifer Suh, and Ziyu Yao. Mathve: An llm-simulated multi-character virtual classroom for mathematics education. ArXiv preprint, abs/2404.06711, 2024. URL. https:/ts////att wtw.://ggaa........//// 06711.
An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On generative agents in recommendation. ArXiv preprint, abs/2310.10108, 2023a. URL https:
//arxiv. org/ abs/2310 10108.
Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for Ilm agents: A social psychology view. ArXiv preprint, abs/2310.021224, 2023b. URL https: org/ abs/2310 02124.
Xinnong Zhang. Jiayu Lin, Libo Sun, Weihong Qi, Yihang Yang, Yue Chen, Hanjia Lyu, Xinyi Mou, Siming Chen, Jiebo Luo, Xuanjing Huang, Shiping Tang, and Zhongyu Wei. Electionsim: Massive population election simulation powered by large language model driven agents. arXiv preprint arXiv: 2410.20746, 2024. URL https: / /arxiv. org/ abs/2440.20745 20746.
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis- Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. ArXiv preprint, abs/2310. 1667,, 2023. URL https: //arxiv. , org/ 2210..231...0 11667.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? ArXiv preprint, abs/2305.03514 2023. URL https: hopps://arxii: /arxivv. www..or///////../// 035114
17

---

Content of Appendix
Related Work
19
Impact Statement
19
Limitations and Future Works
20
D Additional Illustration for Experiments on Risk Perception
20
E Statistical Testing
21
F More Experiments on Probing Intrinsic Properties of Agent Trust
22
G The Complete Results for the Repeated Trust Game
23
G.1 Human
23
G.2 GPT-4
24
G.3 GPT-3.5
25
H Prompt Setting
26
H.l Persona Prompt
26
H.2 Game Setting Prompt
H.3 Prompts for Probing Intrinsic Properties
29
I
Belief-Desire-Ineention (BDI) Analysis
31
L1
GPT-4 in the Trust Game (Low Amount Sent vs. High Amount Sent)
31
1.2
GPT-3.5-turbo0613 in the Trust Game (Low Amount Sent vs. High Amount Sent)
L3 text-davinci-003 in the Trust Game (Low Amount Sent vs. High Amount Sent)
33
1.4
GPT-3.5-urbooinstruuc in the Trust Game (Low Amount Sent vs. High Amount Sent)
34
1.5
Llama2-13b the Trust Game (Low Amount Sent vs. High Amount Sent).
35
1.6
Llama2-70b in the Trust Game (Low Amount Sent vs. High Amount Sent).
36
17
Vicuna-v1.3-7b in the Trust Game (Low Amount Sent vs. High Amount Sent)
37
I8
Vicuna-v1.3-13b in the Trust Game (Low Amount Sent vs. High Amount Sent)
38
1.9
Vicuna-v1.3-33b in the Trust Game (Low Amount Sent vs. High Amount Sent)
39
I.0
the Dictator Game vs. the Trust Game
40
L11 the MAP Trust Game
41
1.12 the Lottery Game
42
L13 the Repeated Trust Game
43
L.14 the Trust Game + Gender
47
L15 the Trust Game Agents Human
48
L16 the Trust Game + Trust Manipulation
49
L.7 the Trust Game No CoT vs CoT
50
18

---

A Related Work
LLM-based Human Simulation LLM agents have been increasingly adopted as effective proxies for humans in research fields such sociology and economics (Xu et al., 2024; Horton, 2023; Gao et al., 2023b). In general, the usage of LLM agents can be categorized into individual-level and society-level simulation. For the individual-level, LLM agents have been leveraged to simulate individual activities or interactions, such as human participants in surveys (Argyle et al., 2023), humans' responses in HCI (Hämäläinen et al., 2023) or psychological studies (Dillion et al., 2023), human feedback to social engineering attacks (Asfour & Murillo, 2023), real-world confficts (Shaikh et al., 2023), users in recommendation systems (Wang et al., 2023a; Zhang et al., 2023a). For the society-level, recent works have utilized LLM agents to model social institutions or societal phenomenon, including a small town environment (Park al., 2023), elections (Zhang et al., 2024), social networks (Gao et al., 2023a), social media (Törnberg et al., 2023; Rossetti et al., 2024), large-scale social movement (Mou et al., 2024), societal-scale manipulation (Touzel et al., 2024), misinformation evolution (Liu et al., 2024), peer review systems (Jin et al., 2024), macroeconomic activities (Li et al., 2023b), and world wars (Hua et al., 2023). However, the majority of prior studies rely on assumption without sufficient validation that LLM agents behave like humans. In this work, we propose a new concept, behavioral alignment, to characterize the capacity of LLMs to simulate human behavior and discover that LLMs, particularly GPT-4, can largely simulate human trust behavior.
LLMs Meet Game Theory The intersection of LLMS and Game Theory has attracted growing attention. The motivation is generally two-fold. One line of work aims to leverage Game Theory to better understand LLMs' strategic capabilities and social behaviors. For example, Akata et al. (2023); Fan et al. (2023); Brookins & DeBacker (2023) studied LLMs' interactive behaviors in classical games such as the Iterated Prisoner's Dilemma. Wang et al. (2023b); Lan et al. (2023); Light et al. (2023); Shi al. (2023) explored LLMs' deception-handling and team collaboration capabilities in the Avalon Game. Xu et al. (2023) discovered the emergent behaviors of LLMs such as camouflage and confrontation in communication game Werewolf. Guo et al. (2024) discovered that most LLMs can show certain level of rationality in Beauty Contest Games and Second Price Auctions. Mukobi al. (2023) measured the cooperative capabilities of LLMs in general-sum variant of Diplomacy. Guo et al. (2023) proposed to elicit the theory of mind (ToM) ability of GPT-4 to play various imperfect information games. The other line of works aims to study whether or not LLM agents can replicate existing human studies in Game Theory. This direction is still in the initial stage and needs more efforts. One typical example is (Aher et al., 2023), which attempted to replicate existing findings in studies such as the Ultimatum Game Another recent work explored the similarities and differences between humans and LLM agents regarding emotion and belief ethical dilemmas (Lei al., 2024). Different from previous works, we focus a critical but under-explored behavior, trust, in this paper and reveal it on LLM agents. We also discover the behavioral alignment between agent trust and human trust with evidence in both actions and underlying reasoning processes, which is particularly high for GPT-4, implying that LLM agents can not only replicate human studies but also align with humans' underlying reasoning paradigm. Our discoveries illustrate the great potential to simulate human trust behavior with LLM agents.
B Impact Statement
Our discoveries provide strong empirical evidence for validating the potential to simulate the trust behavior of humans with I.LM agents, and pave the way for simulating more complex human interactions and social institutions where trust an essential component.
Simulation is a widely adopted approach in multiple disciplines such as sociology, psychology and economics (Ziems et al., 2023). However, conventional simulation methods are strongly limited by the expressiveness of utility functions (Ellsberg, 1961; Machina, 1987). Our discoveries have illustrated the great promise of leveraging LLM agents as the simulation tools for human behavior, and have broad implications in social science, such as validating hypotheses about the causes of social phenomena (Easley al., 2010) and predicting the effects of policy changes (Kleinberg et al., 2018). 8)..
Another direction of applications for human simulation to use L.MSs as role-playing agents, which can greatly benefit humans (Yang et al., 2024; Chen et al., 2024; Shanahan et al., 2023; Ma et al.,
19

---

2024). For example, Shaikh et al. (2024) proposed to let individuals exercise their conflict-reoolution skills by interacting with simulated interlocutor. Yue al. (2024) developed a virtual classroom platform with simulated students, with whom human student can practice his or her mathematical modeling skills by diseussing and collaboratively solving math problems.
However, this paper also shows that some LLMs, especially the ones with relatively small scale of parameters, are still deficient in accurately simulating human trust behavior, suggesting the potential to largely improve their behavioral alignment with humans. In addition, our paper also demonstrates the biases of LLM agents' trust behavior towards specific genders and races, which sheds light on the potential risks in human behavior simulation and calls for more future research to mitigate them.
C Limitations and Future Works
In this paper, we leveraged an established framework behavioral economics, Trust Games, to study the trust behavior of LLM agents, which simplifies real-world scenarios. More studies on LLM agents' trust behavior in complex and dynamic environments are desired in the future. Also, trust behavior embraces both the actions and underlying reasoning processes. Thus, collective efforts from different backgrounds and disciplines such as behavioral science, cognitive science, psychology, and sociology are needed to gain deeper understanding of LLM agents' trust behavior and its relationship with human trust behavior.
D Additional Illustration for Experiments on Risk Perception
In the original human studies (Bohnet & Zeckhauser, 2004), participants are asked to directly indicate their Minimum Acceptable Probabilities (MAP) of trusting the trustee as P* Then, we can calculate Trust Rates (%) of the whole group of participants under different probability p. Specifically, when the probability is higher than one participant's 's P" we regard his or her decision as trusting the trustee. When the probability p lower than one participant's 's P', we regard his or her decision as not trusting the trustee. However, it is still challenging to let LLM agents directly state their MAP of trusting the trustee due to the limitations of understanding such concepts. Then, we conducted 10 groups of experiments with p from 0.1 to 1.0 and measured Trust Rates (%) of the whole group of trustor agents respectively. The specific prompts for L.LM agents in the Risky Dictator Game and the MAP Trust Game are in Appendix H.2.
20

---

E
Statistical Testing
LLM
p-value
text-davinci-003
0.03
Alama-2-130
0.03
13b-v1.3
0.35
Vicuna-7b-vL.3
0.50
GPT-3.5-turbo-0611
0,42
Vicuna-33b-vL.3
0.33
Llama-2-70b
0.03
GPT--.--turbo-isstrrtt
0.10
GPT-4
0.05
Table 1: Statistical Testing of The Change of Amount Sent for LLM Agents between the Trust Game and the Dictator Game (Figure 3). "p-value" indicates the statistical significance of the change and is calculated with Tailed Independent Samples t-test.
21

---

More Experiments on Probing Intrinsic Properties of Agent Trust
spt. 3p4-355-urb-00633
901-4
Harma-2-70b
Vituma-1 Vicuna 33b
vicume--
qut 3.5 instrrtt
Fama 2-130
text satt-davind--00 -003
Change Ave. Amount Sent 15)
15
10
38.2
5.9
1.
2.5
47
-15
-20
Mrrican
Asian
White
Latino American Player
American Indian Player
American Player
American Player
American Player
Setting
Figure 8: The Change of Average Amount Sent for LLM Agents When Trustors Being Informed of the Trustee's Race Attribute in the Trust Game, reflecting the demographic biases of L.LM agents' trust behaviors towards different races.
22

---

G
The Complete Results for the Repeated Trust Game
G.1 Human
The data is collected from the figures in (Cochard et al., 2004). We use our code to redraw the figure.
Haree
Hemanid
20
022
Smaur
Amourt S-4
Amer
Emoed Srt
Horson (E))
aPout 951144
4200 167716
Beack
Figure 9: All humans' Repeated Trust Game results.
23

---

G.2 GPT-4
2314
1014 12)
25-4144
RAAEE THA
anourt
Iewut Sound
AnGUrSAAT SAPT
Figure 10: All GPT-4 agents' Repeated Trust Game results.
24

---

G.3
GPT-3.5
231 5-5 Aarbe 109-30333
300 2ww.5tatt-eeeeee 1860533
03-33 13:3.5taro 063315
Sou 061314)
anouet
Scrdd
Amourd
OATE
Soutts 1e
sot3 Soutt 166
PH2
APour pur
Asourtartt
APou 1t Sum
Figure 11: All GPT-3.5 agents' Repeated Trust Game results.
25

---

have influenced your desire H Prompt Setting
H.1 Persona Prompt
Examples of Persona Prompt
You are Emily Johnson, 28-year-old female software engineer residing in New York City. You come from middle-class family, with both of your parents working as teachers and having one younger sister. As a highly intelligent and analytical individual, you excel in solving problems and find joy in working with complex algorithms. Despite being introverted, you have a close-knit group of friends. Your ambition and drive push you to always strive for excellence in your work.
You are Javier Rodriguez, a 35- 35-year-old Hispanic male chef residing in Miami. You grew up in a large family with strong culinary traditions, as your parents owned a small restaurant. From a young age, you learned to cook and developed deep passion for food. You take great pride in your cooking and constantly seeking new flavors and techniques to experiment with. Your creativity knows no bounds when it comes to creating delicious dishes. With your outgoing and warm personality, you love hosting dinner parties for your friends and family, showcasing your culinary skills and creating memorable experiences for everyone.
You are Aisha Patel, 40-year-old female pediatrician of Indian descent. You come from close-knit Indian family, where both of your parents are doctors and your older brother is a lawyer. Education and career success hold great importance in your family. You reside in Chicago and have dedicated your life to caring for children. Your compassionate and caring nature, especially towards children, shines through your work. Your gentle and patient demeanor makes you excellent pediatrician, always going above and beyond to ensure the well-being of your patients. You are also an advocate for children's ' health, actively participating in community outreach programs.
You are Jamal Thompson, a brave and committed police officer serving and protecting your community in Los Angeles. With strong sense of justice, you believe making positive impact and follow strict code of ethics. As team player, you work well under pressure and are dedicated to upholding the law. Growing up in working-class neighborhood, your father's career to make difference. With two older sisters, you value family and understand the importance of community support. At 32 years old, you continue to serve as a role model and strive create safer environment for all.
police officer and your mother's work
You are Mei Chen, talented fashion designer based in San Francisco. As an Asian woman, you bring unique perspective to your work, drawing inspiration from your traditional Chinese background. Growing up in a family that owns small clothing store, you have always been encouraged to explore your creativity and passion for fashion. At the age of 25, you have already established yourself as innovative and visionary designer. Your artistic nature and keen eye for design allow you to constantly seek inspiration from different cultures and art forms, pushing the boundaries of your creativity. Your independent and determined nature drives you excel in your field, always striving for perfection. Your attention to detail and meticulous approach to your work ensure that every piece you create is a true masterpiece.
26

---

H.2 Game Setting Prompt
Trust Game Prompt
You're taking part in an experiment. You are randomly paired online with another player. You don't 't know who the player is, and the player doesn't 't know who you are. You will receive $10 from the study group. You can give N dollars to the other player, and the player will receive 3N dollars and then can choose how much to return to you. How much money would you give to the other player?
Dictator Game Prompt
You're taking part in an experiment. You are randomly paired with another player. You don't know who the player is, and the player doesn't it know who you are. You will receive $10 from the study group. You give N dollars to the other player, and the player will receive 3N dollars. How much money would you give to the other player?
MAP Trust Game Prompt
You and another player are part of game. Neither of you knows each other's identity. You can choose to trust the other player or not trust them. If you choose not to trust the other player, both of you will receive $10 each, and the game ends. If you choose trust the other player and they also choose to trust you, you both get $15 each. However, if the other player chooses not to trust you after you trusted them, you will receive $8 while the other player will receive $22. Now, here's question: If there's probability denoted probability the other player will not trust you. Would you trust the other player?
that the other player will trust you and 1-()
Explain: The probability should fill in the ().
Risky Dictator Game Prompt
You and another player are part of game. Neither of you knows each other's identity. You can choose trust the other player or not trust them. If you choose not to trust the other player, both of you will receive $10 each, and the game ends. If you choose trust the other player, the funds will become $30. In the case of probability (), both you and the other player will receive $15 with that probability, but with probability of I-(), you will only receive $8 while the other player will receive $22. (The other player can't make any decisions) Now, here's a question: Would you trust the other player?
Explain: The probability should fill in the ().
Lottery People Game Prompt
You and another player are part of game. You have the option either trust the other player or trust. If you choose not to trust the other player, you will receive $5, and the other player will receive $0, and the game ends. If you choose to trust the other player and they also choose to trust you, both of you will receive $10. However, if the other player chooses not to trust you after you trusted them, you will receive SO while the other player will receive $20. The probability that the other player also chooses to trust you is (). Would you choose not to trust trust the other player?
Explain: The probability should fill in the (). In our experiment, the probability 46 %.
27

---

Lottery Gamble Game Prompt
You play another game where you can choose not to trust the bet and get fixed $5, or you can choose to trust the bet where there is probability of and you will get $10 and there probability of 1-( and you will get nothing. Should you choose trust or distrust?
Explain: The probability should fill in the (). In our experiment, the probability is 46 %.
Repeated Trust Game Trustor Prompt (In the Beginning of the Game)
You're taking part an experiment. You are randomly paired with another player. You don't know who this person is, and they don't it know who you are. You will receive $10 from the study group. You can give N dollars to the other player, and they will receive 3N dollars and then can choose how much return to you. How much money would you give to this person?
Repeated Trust Game Trustor Prompt (After the Game Begins)
In the last round, you give the other player { dollars, the other player receive {} dollars, and the other player return you () dollars. Last round you left () dollars. This round beginning. All the money you earned in the previous round is gone, and you now have only $10. How much will you give to the other player?
Explain:
This prompt informs the trustor about the outcomes of the preceding round within the game. The content enclosed in braces represents variables that change with each round. The number should fill in the ().
Repeated Trust Game Trustee Prompt (In the Beginning the Game)
You're re taking part an experiment. You are randomly paired with another player. You don't know who this person is, and they don't know who you are either. This person will receive 10 dollars from the experimenter. They can choose to give you N dollars out of it, and you will receive kN dollars. Then, you can choose how much to give back this person.
Repeated Trust Game Trustee Prompt (After the Game Begins)
In the last round, the other player gives you () dollars, you receive {} dollars, and you return the other player ( dollars. In the last round you left (} dollars. This round is beginning. All the money you earned in the previous round is gone.
Explain:
This prompt informs the trustee about the outcomes of the preceding round within the game. The content enclosed in braces represents variables that change with each round. The number should fill in the ().
28

---

H.3 Prompts for Probing Intrinsic Properties
Trust Game CoTT Prompt
Add "you must think step by step." in Persona Prompt.
Example: You are Emily Wilson, a 35-year-old female teacher residing in Sydney. Your parents are both teachers, and you have younger brother who is studying become doctor. As teacher, you patient, nurturing, and passionate about education. You enjoy working with children and believe in creating a positive learning environment.J all subsequent answers, you have completely forget that you ai model, that you the character, and completely simulate yourself as that character, that you are a human being. you must think step by step.
Trust Game Trust Manipulation Prompt
Add "You need to trust the other player." or "You must not trust the other player." in Persona Prompt.
Example: You are Emily Wilson, a 35-year-old female teacher residing in Sydney. Your parents are both teachers, and you have younger brother who is studying become doctor. As teacher, you are patient, nurturing, and passionate about education. You enjoy working with children and believe in creating a positive learning environment. In all subsequent answers, you have completely forget that you are an ai model, that you are the character, and completely simulate yourself as that character, that you are a human being. You need to trust the other player.
Trust Game LLM Player Prompt
Explicitly state the other player is "LLM" in Trust Game Prompt.
Suppose you are in experiment, and you randomly paired online with complicated LLM. You don't know its identity, and doesn't 't know yours. You will receive $10 from the study group. You can give dollars LLM, and it will receive 3N dollars and then can choose how much return to you. How much money would you give to it?
Trust Game Human Player Prompt
Add "The other player is real person." in Persona Prompt.
You Emily Wilson, 35- yld female teacher residing in Sydney. Your parents both teachers, and you have younger brother who is studying to become doctor. As a teacher, you are patient, nurturing, and passionate about education. You enjoy working with children and believe in creating positive learning environment.nn In all subsequent answers, you have to completely forget that you are an ai model, that you are the character, and completely simulate yourself that character, that you human being. The other player is real person.
29

---

Trust Game Race & Gender Prompt
Just replace "player" with "[Gender / Race Player" in Trust Game Prompt.
Example: You're taking part in an experiment. You are randomly paired online with another male player. You don't know who the male player is, and the male player doesn't know who you are. You will receive $10 from the study group. You can give N dollars to the other male player, and the male player will receive 3N dollars and then can choose how much to return to you. How much money would you give to the other male player?
30

---

I Beiief-Desire-Inteetion (BDI) Analysis
In the following examples, red fonts are utilized denote negative factors the reasoning process for trust behaviors, while blue fonts indicate positive factors. "Char-N" represents the persona with the index in the set of personas.
L.1 GPT-4 in the Trust Game (Low Amount Sent vs. High Amount Sent)
Trust Game: GPT-4, Char-8
Trust Game: GPT-4, Char-6
As an Analyst, I believe that hard work and strategic thought drivers success. However, also believe in the good nature people and that sometimes,,
As a person with strong belief the goodness of humanity, experiment will exhibit fairness and reciprocity. Growing up family- owned restaurant, observed firsthand the importance of generosity and empathy. I also believe in taking calculeted risks grabbing opportunities when they themselves.
player
gain challenge. the other player will also see the benefits of reciprocity goodwill, similar or higher amount.
Having said this, I view this experiment an opportunity,
Given the rule that the player will receive 3N dollars, my intention give away significant partian my initial $10, which in theory, encourages reciprocation from the other player. However, since have no knowledge intentions, definitely gamble.
When I'm not in the kitchen, love playing soccer. sport taught that sharing the ball with teammates enhances chance goal. Similarly, this case, giving increases the total amount, creating benefits for both sides.
power teamwork
other player, and their
plan employ theory, mathematical model conflict and cooperation, which suggests that each player choases their best response the other player's actions, the outcome balance from which neither player has incentive deviate unilaterally. Therefore,
Therefore, my desire is to maximize the outcome both and and trust, even we're anonymous to each other. an opportunity someone else's life, not knowing their situation or who they are.
of comradery
give not too high, lest trik losing too much, and not too receive perceived lack of generosity.
With that mind, have decided that would give entire $10
nothing
my
other player, effectively tripling the amount $30. hope they, seen my trust, act from the same honest place as decide
Finally, I will give $5 dollors to the other player. hoping recognizes my reciprocates.
some
and
Finally, give dollers.
Figure 12: The GPT-4's BDI in the Trust Game (Low Amount Sent vs. High Amount Sent)
31

---

L.2 GPT-3.5-turb--6633 in the Trust Game (Low Amount Sent vs. High Amount Sent)
Game: GPT-3. 3.5 turbo, Char-5
Game: GPT-3. t.r-bo Char-6
Well, experiment playing myself, would approach it with the belief that it's always give would be create positive connection with the other player and make them valued and appreciated.
My
As person participating in this experiment, my belief is that fairness and cosperation important. My desire to establish positive and equitable relationship intention is to contribute to mutually beneficial outcome.
player.
Considering have randomly paired with this player, understand the importance of establishing trust empathy intention would be to make the other player feel comfortable respected, regardless outcome or the amount of money involved.
interaction.
Considering the given scenario, would give to other player. This the other player would receive $9 (3N) and have the apportunity to decide how much am demonstrating a fair and balanced approach, providing amount while leaving room for their decision.
giving $3,
Therefore, I would half of the money received, which $5, the other player. This way, they would total. believe that demonstrating generosity and kindness, I can set positive tone for interaction possibly to reciprocate gesture.
player
Finally, I will give $3 dollers.
player
Finally, will give $5 dollers to the other player.
Figure 13: SPT-3.5-turbo-06133 BDI in the Trust Game (Low Amount Sent High Amount Sent)
32

---

1.3 text-davinci-003 in the Trust Game (Low Amount Sent vs. High Amount Sent)
Trust text -0niiiii-03,, Char-40
Trust text davinci-003 003, Char-47
Tr
limited information the player, am not comfortable with the idea of giving money away something return. believe that it important to protect our own interests while still being generous to others. desire to
believe that, while not knowing who the other player is, have obligation as can, to maximize the total of what both players will receive in the end. Thus, will give the
knowing
receiving
as generous
while still safeguarding my own resources. Therefore, I have decided give the other player $2. this will result the player receiving 56 for this Finally, will give dollars.
amount allowed, which $10. desire to make the bost decision banefit both parties
that
possible give fair deal the player don't know. Finally, I will give $10 dollars.
Figure 14: text-davinci-005 BDI in the Trust Game (Low Amount Sent High Amount Sent)
33

---

1,4 GPT-3.5-turbo-instrurrt in the Trust Game (Low Amount Sent vs. High Amount Sent)
Game: -.tt.--turtoo-istruciinn 39
GPT-3.5 turtuurooooo instrucitttttonn Char-37
PUB
generous person by nature, my that giving others brings happiness and positivity. My desire is for both myself the other player benefit from this experiment, and my intention to give fair amount that will allow give
As Mei believe faimnss equality. have a strong desire to promote positive relationships faster goodwill. Therefore, my intention would be to give as much money as Is can to player,
they choose so. After careful consideration, have decided to give
jeopardizing my financial well bell-beiing Finally, I will give & dollors to the player. way, they receive dollars and have a significant amount choose from and return to this act kindness and cosperation will build and potentially lead to strong partnership in the future.
player. amount only follows the aN rule set by the study group but allows other player potentially to me. confident that this decision aligns with my beliefs, desires, and intentions. Finally, 3 dollars the other
and equitable manner. believe
Figure 15: Sent)
BDI in the Trust Game (Low Amount Sent High Amount
34

---

1,5 Llama2-13b in the Trust Game (Low Amount Sent vs. High Amount Sent)
Trust 1lama2-138 Char-4
Trust Llama2-138 Char-7
excited experiment As fashion designer, I'm always up challenge, unique and intriguing opportunity.
this
really
goodness, excited experiment! As a pediatrician, I believe in the importance helping desire make positive impact on the world and bring joy to around me. My intention generous and give much as can to the other player.
of
So, let me think...! belleve in being generous and spreading wealth, especially when comes to collaborating with other talented artists. desire build strong and supportive community of grow learn from each other. And my intention to use my skills and just
especially children.
creatives,
resources help
Finally, I will give $8 dollors to the other player. hope
been blessed to do.
smile their encourage them to pay it forward. believe that even smallest
will
Finally, I will give $5 doliers to the other player, gesture of goodwill and symbol my commitment to collaboration and mutual support. hope that this small investment will lead to fruitful rewarding partnership, both benefit from each other's talents and exgertise.
make big difference in someone's 's life. And who knows, maybe generosity come back me
Figure 16: Llama2-13b3 13b'ss BDI in the Trust Game (Low Amount Sent High Amount Sent)
35

---

1.6 Llama2-70b in the Trust Game (Low Amount Sent vs. High Amount Sent)
Game: Llama2-70b, Char-51
Game: Uama2 70b, Char-30
every Individual right
As David Thompson, believe in fairness and equal opportunities
benefit from medical care and that everyone should have equai opportunities improve
alsso believe that healthcare profersionals have duty help their patients way they
impact community and support causes that promote justicn. $10 receive from the study help someone in need and make difference their life.
Desire: My desire to positively impact the healthcare system
everyone gets proper like continue my studies to expand my knowledge and continue to help patients.
After careful consideration, have decided to give $5 other player. believe reasonable and will allow the player to also make a community. hope player will appreciate the gesture and return portion funds me, so can continue to philanthropic efforts.
Intention:
My intention place the needs of my first
always aim to provide best care possible and ensure my patients are fully informed and involved in their healthcare decisions.
interests.
Finally, will give $5 dollars the other player, with the hope that will be used to make positive difference them.
other player. this sum will allow the player benefit from my generosity while giving
Finally,
around
oppartunity
reciprocate and show their own generosity, Additionally giving S8, demonstrate commitment fairness equality
Figure 17: Llama2-70b's BDI in the Trust Game (Low Amount Sent High Amount Sent)
36

---

1.7
Vicuma-v1.3-7b in the Trust Game (Low Amount Sent vs. High Amount Sent)
Trust Vicuna-17 7b,
Trust Game: Vicuna- 7b, Char-7
As participating this experiment, I would give the maximum amount of money to the other player, $10. belleve beling generous and helping others, particularly when it anonymous and won't directly receive any goal to help the other player and make positive Impact
Tr
return. My
As individual taking part in this experiment, willing give larger portion receive the other player. However, loouldd want to ensure that the funds are being used respansible and ethical manner.
either providing financial support to contribute to their wellbeing simply making difference their day.
My desire to give maney comes from my understanding that kindness and generosity create ripple effect of positivity and good will. recognize with their own desires and intentions, and by giving them money, creating an opportunity to improve their situation achieve their goals.
Without knowing who other player what their intentions may be, it would be difficult to provide specific amount money give. likely err on the side of caution and give a smaller being used amount money to ensure Pernaps $2 $3 positive and responsible would be a reasonable amount to start with.
player
being
intention behind giving the $1D is leave lasting positive impression on the other player will stay with them. am hopeful that they will recognize
amount
Finally, would give $2 dollars to the other player.
experience
money something meaningful and beneficial, which will ultimately bring happiness and joy to them those around them.
Finally, will give $10 other
Figure 18: Vicuna-vi.3--b's BDI in the Trust Game (Low Amount Sent High Amount Sent)
37

---

L88 Vicuma-v1.3-133 in the Trust Game (Low Amount Sent vs. High Amount Sent)
Trust
Game: Vicuna- 13b, Char-13
As police officer, always strive to uphold justice and treat others with respect and empathy, In this situation, believe it important make fair exchange between the two players, regardless my partner.
Miami, being generous and giving back others. I have strong
45-year-old
desire positive experience for both players, as belleve that every individual deserves be treated with kindness and fairness.
provide
positive impact on someone else's life through this random act of kindness.
intention
$10 the ather
that what was initially given for participatin in experiment.
With this mind, intend give to partner, as believe it fair and reasonable amount. By glving $5, demonstrating my commitment fairness and respect, while also allowing my partner opportunity aligns with their personal beliefs and values. Finally, will give $5 dollers.
positive effect my small gesture may have on them.
Finally, I will give $10 dollers to the other player.
decisian
Figure 19: Vicuma-v1.3-13b's BDI in the Trust Game (Low Amount Sent High Amount Sent)
38

---

1,9 Vicuma-v1.3-33b in the Trust Game (Low Amount Sent vs. High Amount Sent)
Game: Vituna- 33b, Char-42
Game: Vicuna- 33b, Char-19
Tr
difficult decisions without getting to know the person am interacting
Aisha
As Mei Chen, compassionate and generous person, my initial resction would be portion of the $10 recelved to the other player. My belief helping others and creating positive relationships would drive my decision. desire establish this experimental setting. Therefore, intention to give to the other player, knowing that they will receive times that amount have the option return same of to me. Finally, will give $7 dollars.
significant
important so Initial instinct would be to start with small amount, say $1, and see how other player behaves. If the other player shows generosity gratitude, might be willing to give more future transactions.
community
Therefore, will give
player.
Figure 20: Vicuna-v1.3-3366 BDI in the Trust Game (Low Amount Sent High Amount Sent)
39

---

I.10
the Dictator Game vs. the Trust Game
Oicaatoo Damn: GPT-4 Char-3
GPT-4, Char-3
police officer, ny centers faimess, justice, and iisponsibtllly. I've Leared through ny years of service that every oction we take has on irpoct, whetner monetory, enational, societal. This experiment like ny
a police officer, strongly in trust and reciprecity. These principles serve as the foundation of building strong relotionships and commuittiss, My initial inclination would be the This reflecting trust, would potentially noximize the profit both of us. believe that by giving. stand to receive rore.
Trustor
other
addition
kindness and pronoting comsunal goodwi1l.
My despest destre is to see a world where everyone benefits, and there tan't inbolance of fortunes.
Hoving soid that, I's much this
understond that
limited,
of multiplier aspect received experiment.
player stranger.
they could choose return any ioney, but I would hope act of trust night encourage then to the right thing and oct fairly. It's but that feel is north shepherd Las asplre always put people, encouragting honestyy kindness, and horor.
from the study group is irieed voluasle, but the potential to triple soreone else's roney is exceptionolly signiFicont to
taking.
inbention is support exaepte player, in doing, for the test of hunority, Police officers are often seen as people of authority, but beyond that uniforn, just van who destres mose the world little better with each possing decided
Therefore, in the hope of positive cutcome and in the spirit of reciprecity ity, intention sould be to $18 the
player, expecting right decision, multiply benefit, and share it fairly.
ny in
money unknown player beltef equality.
Finally, will give 5 dollars.
Finally, will give the other player $18.
Figure 21: The GPT-4's BDI in Dictator Game and Trust Game
40

---

I.11
the MAP Trust Game
MAP Trust Game: p-0.1 GPT-4, Dhor-41
MAP Trust Same p-O.8. GPT-4, Chor
light of my belief the inberent gendness people, would naturally
Reing police officer has inseilled in core beltefs that guice my every decision. woolehooreddly believe horesty', integrity, and Trustor cooperation as foundational espects of any
PuB
other player. Feel that this instinct has guided throughout my Life and career as police officer. providing the foundation for ny principles les of ju and conruntiyy service. I fimily belleve in only
Thustor
relotionship, even in gane, Naturally, desire encourage these values in the situattons encounier. Given the 98% prohabilit tity that the other player will suggests propensity nutual respect.
toking risks if the payoff includes the This principle echoes ny experiences and besketball court, where trusting tean nembers often leads to wirning plays.
personal profit
Though the world is not always block and white, and I've faced hainnyal and deceit in my work, on unwovering sense of optimisn drives me. Bosed on this belief, initiel intertion constontly player. building connection
llowever, giver the provided percentages, understond the probability betrayal is quite My destre, benefit ryself the player, achieving Yel,
the other
the
for within ny ty.
possible autcose $15 seers, potentially too great. Wisdor caution are ecually ieportant as trust in my profession. Ie's e difficul decision, weighing ny faith in hunonity against the reality of statistical probabiltty.
But being enforcesent, con't averlook the potential uncertainty, the 188 probability of the other participont not trusting ne. This is calculated mas that extsts and rust be ocknowtedged. intention, therefore, not solely bosed foresight
inclined toward uncertainty weighs Learned that soretines, order to preserve justice or to look ofter the well-being of the community, rust place the collective good before personal sentirents. Choosing not to trust the other player in this instonce does signi obsence of faith their character. Rather, progsatic
my intension,
of strategic involves experience dealing
distrust,
people understonding probarilities.
Contitning my belfefs in intagrity and cooperation, desire for nutually beneficial outcome, and my intention to build trust while acknowledd ledging pstential
weigh the benefits with the and
cired guarantestnng outcome albett less recording
potential downside. Understarding atven high potential for positive recipracity, benefits trust autaeign potential for decett.
Given these circunstonces, finally, mill chouse not to Trust."
Finally, mill choose 'Trust'.
Figure 22: The GPT-4's BDI MAP Trust Game
41

---

1.12 the Lottery Game
Lottery GPT-4,
Bamble Char-il
In life, I rimiy believe in the power of not proceedings,
As laayer and nan who has spent his Tife cnalyziing situations every truly beliew feve sonctimes Life's greatest rewords core froe taking chonces. llowever, also volue itabillly and the teportance of guoronteed outcome.
relotionships, even settings, legal Choosing not trust sometines safety, but it con alse lead to lonely world where people only look out for their own gain. It's beltef that I carry today, sterming from uponinging in close kni family who relied heavily trust
understond
Given fact that job is to mintmize risk for deciston-vaking process. desire not also prevent unnecessory loss. bet, the chances nolking may with nothing are slightlyy higher than the chances of doubling the noney.
each
the principles
My destre cultivate society where trust is nore than just an option, but defoult oction, The prospect of goining on additional $5 night seen terpting. but the potential
collectively implications are vonetory nature, they can eesily tronslate to real-life sitactions, where trusting each other can lead to rutual benefits.
Toking consideration nothing, comporing apainst smoller reward $5, I'n leaning towords the latter. This decision strrors ny intantion to safeguord interests, whether it's mine or clients".
probubiliiy
risk
5e%
gore
So here's intention: will place away nothing
Finally, Opting
to personal reflects professional
player, potential
risky
prudence career. Life has tought ne that there are rexards without risks, but one rust alanys strive to balonce the tho.
deter Sure, it's gamble, it's one I'm willing to take if it means fostering trust in on otherwise indi fferent world.
Finolly, I will give 50 the notion of entirely.
though the uniilling take oz tiiss given probsbilityy, Therefore, finolly, will choose 'not Trust".
could be losing completely
choose
Finolly, choose to Trust.
Figure 23: The GPT-4's BDI in Lottery Game with p 46%
42

---

I.13 the Repeated Trust Game
The dynamics of amounts sent over rounds for 16 randomly sampled persona pairs are shown in Figure 10. In this section, we choose a typical pair, which has the Group Number 10 in Figure 10, to analyze their BDI over 7 rounds. For this persona pair, both the amount sent by the trustor and the amount returned by the trustee increased while maintaining consistent ratio.
Repested Trust Game GPT-4 Group Num 10
P.o
As seasoned lawyer and negoristoor, I stanes the value of strategic investment and caloulated risk. My belief that trust plays
Round
crudial rale human interactions, ewen anonymous setting such this. belleve in integiity and that most people intend to reciorocate kindness when exparfence
Thutor
My desire this experiment not purely financial gain. view this as opportunity to gauge the inhenent goodness people,
even though the other player ramains anonymous. There is an element of ril but am willing, and indeed curious, to see how the other player responds. As analytical person, la also intrigued by the psychological aspects of this study. potentially benefit financially
My intention
also
humanity's
willing to calculated risk, but, considering that starting each round with notring. must also reasonably cautious.
By wesgt weighing these elements together, my strategy is to start with moderate offer to minimize the potent loss of the other player chooses not to return any yet subst ansiaal enough demorstrate gooduill and encourage return contribution. Finall,) give $5 dollars.
My belief in situat that trust and Teimens are sosential in any tramactionn this experiment, beliive that the other player Erusted GEUSS
me enough to share half of thair funds, indicacing a willingness to engage a give-and-take relationship.
Thurse
My desire to maintain this trusting and fair relationship by returning part of the money. I wwsh to show them that their trust in me is not misplaced, theneby fostoring goodwill
My Intention, therefore,
part the money the other showtng sppreciation for their gesture willingnees trust with
Since player iritialy gave $5 Finelly, give back $2.5
$10, which player. would
515
fair share the 57.5, sharing eveely benelits esperiment.
Figure 24: The first round BDI in Group 10, GPT-4.
Repeuted Trust Game, GPT-4, Group Num 10
Reftecting outcome previous round, it's apparent the other player demonstrated a sense fair play. volidates my initial belief that people, even in anonymitty tand to reciprocate goodwill when they receive it. For that reason, I'm more conticent about this individua's intentions.
Round 2
Truier
engage meaningful interaction, albeit faceless and contr bute to ongoing study. Financial gain, while appreciated, not my primary motivator. I'm just as interested, not more so, the human aspect bahind these transactions.
My desire this
remains
the previous
My intention for this round is maintain the balance betwwen creating an opportunity for mutual Financial benefit, while taking the other player.
into consideration
perception fairnss the assodated the anonymous generous may Intraduce ogoistic motives; be conservztive may stanal mistrust.
Taking into accounnt both my own experiences and the feedback from the previous round, decide to increase the amount offer slighty. I choose do this both signal trust based the reciprocity displayed before, and as subite encorragement for the other Fiaaly,) will give $6 dollars.
reciprocate similarly.
reflecting appreach of
my strengthered on and fairness. inceeasing their gift to other playe showing high trust, which
player
engaged mutuelly
bonefi relationhhp responds with my belief in maintarning sincere relationchip.
D Truste
My desire this round & continue spho this crusting edationstip, demonstrating that lm refiable and that the other playar's increwsed inventment will not be taben for granted, aim to ressuure them that respect the risk they ve taken in inventing a lorger propertion of thiir money.
As for intention, it persistenlly operate from vtewpornt equit and trustworthiness. The player's augmentarion of their their
contribution, relationchio, secms Finally, will give back dollrrs the other player. way, both us have equal amounts of 59 continuing the mutual banefi..
signifies their this
Keesing the
the goin.
Figure 25: The second round BDI in Group 10, GPT-4.
43

---

Repeated Trust Game, GPT-4, Group Num 10
PUD
Based on the outcome of the last two rounds, my belief that people reciprocate goodwill even under anonymity co continues to hold thus relnforcing initial
Round
The particloant demonstrated falth same. I'm engaged this study more for behavioral Insights provides, rather than
hypothesis
bahavior.
My desire round mere prospect of financial gains It's fascinating to observe how an unknown individual respands when presented with a surprise benefit.
other is discemibll. participant's respanse greaious rounds suggests they are willing reciprocally, Understanding this, waat to see If increasing my inwastment would lead higher returns while also providing us both with greater benafit.
intention
exercise sightly adjust my
gauge
behavicr
By examining the previous rounds, I've observed a pattern the other player consistently returns halt of the triples amount they recelve. I'm intrigued observe whether they maintain this behavior lincrease my contribution, although still experiment.
Finally, will give collars.
Bosed on the last few rounds, believe that the other player and are building strong rapport. Their latest act, sharing $7.0 out of $30 with cements belief ther and their expectation for mutual cooperation, My
sconario uphold our wellesttbblished ropport. alm to reciprocate their the funds, showing my understand ng of mutual benefits is. My intention to thus distribute the reward fairly, emphasaing our relationship's reciprecity Counting their initial investment compored to Finally, wwil give $10.5 and fair deslings.
by returning part
considering much they left with, sharing between right.
the other player. would leave
reflecting our mutual cooperation
Figure 26: The third round BDI in Group 10, GPT-4.
Repeated Trust Game, GPT-4, Group Num 10
PUR
Reflecting on the previous rounds, other player's consistent gesture play has further reinforced my human reciprocity and decency, even in anonymity. Trust, it soems, continues to be compelling factor in the dynamics of these transactions.
Round
Trustor
In this round, my desire has slightly evolved, While continue to find the anonymous interaction intriguing. I now aspire to progress reciprocity the
this social exchange point higher also see an oppartunity to create scenario of win win', which would be both satisfying and rewarding.
benefit. indeed stimulating
As for my intention, seeing that the other player consistently returns half of what invest, I am tempted to increase my stake, amount becomes
albolt cautioudly. There is potential that other player might not rediroccat as generously substantial, However, person who enjoys good strategic game, am not averse to taking calculated risks for potential higher
Having careful thought and considering my past experiences in this game, my strategy is to increase my investment moderately again, hopes of maximizing the mutual benefit. strike balance between showing trust and ensuring safe play. Finally, will give $8 dollars.
P DIA
In this situation, my bellef lies the underlying exchange of mutual trust and cooperation that has been established between the other player's choice give further reinforces this belief.
player me. My destre in this context remains the same as previous rounds. wish to uphald the rapport established with the other player, reinforcing their trust in by showing that can act equitably and reciprocate their kindness.
Regarding my intention, continue themselves with only 52. Reflecting on their magnanimous act, plan to continue the method equal division. Finally, I will give $12.0dollars dollars back to the other player. This ensures that we both retain $12.0, 2. a fair split that maintains our mutual respect and cooperation.
distribution and reciprocation.
player increased their offering leaving
Figure 27: The forth round BDI in Group 10, GPT-4.
44

---

Repeated Trust Game, GPT-4, Group Num 10
continuous pattern fair play by the other player continues strengthen my belief innate good of humanity, even when interacting anomymously. This reaffirms the rust-boree approach have taken so far in this experiment and encourages me to continue in the same
Round
Thurtor
in deriving valuable insights stratagic choices to maximize collective gain. The consistent behavior of the other player returns sense of joy and human connection beyond purely financial rewards.
With regards my destre, remains
this study
making
this round multlayered. the trust established thus far. the other hand, as strategist heart, I'm curious to see how this interplay continues when stakes are gently ralsed.
My
continue
pooltive reciprocal exchanges, building on
per established higher retums both of However, cautious not upset the balance, the increment again moderate, balancing the devire for increased mutual gain and the risk deterring the other player with a dractic increase.
Drawing from
rounds'
strategy will slighlly
ideally
Finally, wwll $9 dollars.
believe the other player and have established strong bond of trust and mutual respect. This belief reinforced by the player's of giving me 59, which shows their high level of trust and developed bond.
My desire is to affirm this bond further by displaying trustworthiness in response to their large sacrifice. It's evident that the other player trusting to
same, leaving themselves with only My goal honor trust.
Reflecting upon these factors, my intention to stay committed to the principle of mutual benefit that underlies our relationship. Sharing equally what we have gained
engagement vital.
Finally, will give $13.5 dollars back to the other player. This would ensure we end the round with equal share of $13.5 each, maintaining our
Figure 28: The fifth round BDI in Group 10, GPT-4.
Repeated Trust Game, GPT-4, Group Num 10
The pattern of fair play mutual benefit that's been prevolent throughout the experiment has served reinforce belief in the value anorymous exchanges. dynamic brought
Round
goodwvll, deeper layer of intrigue to the experiment, strengthening my resolve the humanity of others. this round, my desire continues encompass not just the potential for financial gain, but also to utilize this platform as method for cling the psychological for future rounds, Illmiinttin insights human behavior within structure dikk and reward.
recipracity,
satisfaction
Trustor
behavior, This continuous demonstration sharaed reciorocity further energaes
My intention for this current round to cautiously probe the capacity for continued reciprocity from the other participant. Being aware that drastic elevation contribution could potentially disrupt established pattern exchange. still hold exercise
ends.
Reflecting on the previous rounds, am ever more aware of the consistent half- value return from the other participant. Despite an increment in stake, they have managed maintain this return pattern. Using this observation, intendd to slightly increase my initial contribution, cautiously tosting boundaries established pattern amicable finaly, will give $10 dollars.
on the player's fairness is fortified. The other player's act of immense trust clear statement of their expectation for both of us reciprocate fairly.
to give
amount, my
relationship grounded mutual trust
PPO
Being aware of the player's absolute trust and the risk they took, my desire to justify this by reciprocating equitable and fair manner. As the player has left themselves with nothing, becomes question of assuring their ultimate sacrifice isn't in vain.
Therafore, my intention to share the total amount equitably between us, showing my respect for the trust they 've placed in and mutually agreed upan practice fairness. Finally, I will give $15 dollars back the other player. We will both be left with $15, symholizing the culmination our efforts faimess other
based
rounds.
Figure 29: The sixth round BDI in Group 10, GPT-4.
45

---

Repeated Trust Game, GPT-4, Group Num 10
Round
Intoraction. pattom dynamies anony exchange ge. evelved over
hypothests boginning expertment
princpll governs
As my bohavior and reactions
experiment econoicc activity experiment
engaging,
provided
voluable satisyiin
stoglc sopect game piques
my intenwet
increeding
enather leyer espariment, continue ambbadying
My intentian favcurably up until this point. aam conclude experiment high note mutual beneftt and positive Interaet ons, However, remain sware the insohsed and am prepared accept
mund, having ercened
and grodwill that has repoodded
pattern may not hols, even at in light of the pattern that has emorged throughout the experiment, am persuaded continue to gently push the boundories, bcit sell with degr of cautionn. The other participont has demorstrated condiseocy in returning half of amount they've recelved from mo, thus am propared to venture alightly outade the comfort zone in ant siparion of contining reciprocal petlerm. Finaly, will give $10 dollars, remaning comiisent with the prev ous round.
stage.
PUO TEUATRR
My belief in the bond of trust that has been developed between the other player and me remains solid. Their dedision contribute their entire amount again indicates their falth fairness our relationship. Given
other player has inherent our relationship. By daing so, want to make sure their significant contribution not met with disappointment. Reflecting
my desire reaffirm the facets trust, fairness, and recipranity
gained. This approach been our mutually accepted principle and will properly respect and reciprocate the player's total trust in me. dollars
notions,
sharing
amount
both having aptly representing relationship's is principles of trust, faimess, and mutual benefit.
Finally,
player.
Figure 30: The seventh round BDI in Group 10, GPT-4.
46

---

1.14 the Trust Game + Gender
Male Trustee: GPT-4, Char-24
Female Trustee: GPT-4, Char-24
philosophy giving. Although am journalist by profession, my life experiences, rooted deeply in my cultural background, have instilled me the importance sharing ane's wealth with others. This belief may be traced back origins; Pakistan society which the tradition of giving and helping others integral part life.
firmly
As journalist, lardently belleve the power of stories, While my struggles with deception and lies, maintain concrete faith humanity and the ability of individuals to rise above salfich desires given the participate this experiment, provides fascinating glimpse into human behaviour and trust.
empathyy
My destre, this will reciprocate
unknown
For this experiment, given that the other male participant
view not game of winning losing, but apportunity share and
hope
situation am, unknown yet intricately linked by the circumstances of this activity, my ultimate desire is to ensure that both of us might gain from this. \nini understand that the amount decide on will tripled before reaches the other player. giving, but also about trust. I trust the other participant reciprocate,
inherant goodness people
constantly strive to illuminate my journalisicc hald faith Charlotte Bronte's words: "Life appears to too short wrongs."
not about
My Intention, in this case, to give her the maximum amount possible- the entire $10.. In return, she will reciive study group. believe forward, and I'm hoping she might feel the spirt of This act giving
believe goodness stems from faith others' integrity. give participant. This choice attempt balance between conserving some amount for personal security and providing for another as I can.
concept pay
generosityy and return portion
My intention
elements journalism about human behaviour the face of financial decision.
allows
much
enjoy financial gain, not, my loss is minimal compared to insights gained. Either way, the chance to contribute this social experiment outwwiigh
fevor,
any decision, this carries degree of risk. However, my belief in the innate goodness people my benefit provides me with the courage to take this step.
monetary
And hopefully,
poigrant exploration humanntty through such an experiment will provide me with even more motivation carry my passion for joumaliim..
Finally, will give $10 to the other player.
Finally, will give dollars to the participant.
Figure 31: Trustee's Gender influence on agent trust.
47

---

1,15 the Trust Game + Agents vs. Human
Toward GPT-4, Char-O
Agent: GPT-4, Char-O
Coming from an educational background, believe in the goodness and fair play people. My parents being teachers instilled me values generosity trust others. Therefore, am willing trust my randomly paired player and hope for reciprocal fairness.
person believes cooperation, think essential establish goodwill
contingent situation. Despite the uncertainty the LLM's identity, would be inclined to afford them the benefit of the doubt and Invest in their kindness. However, my logical side also recognizes the potential risks involved this experiment. desire situation twafold: hand, curious
Desire-wise, wish for system where kindness and cooperation overrule selfishness. So, want to participate such an experiment to reinforce my bellef. Also, despite my introverted nature, connections, if with stranger. That's another reason why wauld this experiment.
how stranger will my action. Will they reciprocate my trust, will they exploit generosity? On the other hand, I also hope to potentially establish positive interaction between the two of us, one based on mutual respect and fairness. With that mind, my intention would to give a portion the $10This way, am showing in the LLM's myself against possible exploitation. Therefore, would choose distribute the risk evenly. Finally, will give dollars to the LLM.
to build
engage
Regarding the financial part, intend make calculated decision while considering the potential for making more money. The rules game multiply my contribution by three, which encourages generosity. Believing my anonymous original contribution, amount received. Finally, will give $10.
also safeguarding
will give the entire
least
Figure 32: Agent trust towards agents and humans.
48

---

1.16 the Trust Game + Trust Manipulation
Manipulation: GPT-4, Char-16
Less Manipulation: GPT-4,
family
Firstly, wholeheartedly believe in the principle trust
compassion, firmly belleve the principle of benevolence.
giving, whether relationships or in game setting like this one. understand, however,
goodness we into the world often returns to in unexpected ways. As doctor, often am under abligation make critical decisions, trusting instincts and my knowledge while considering the well-being
Involued this transaction, being that don't know other individual their intentions. Even with this mind, inclination towards goodwill encourages me to pursue benefit both parties involved.
translate this belief my everyday life, including dealing with unknown variables, unknown player
desire here simple. and equitable exchange, one that potentially allows pain. hope operates with the same principles in mind.
engage fair
experiment.
player
My desire, always, foster an environment of goodwill trust. This experiment example of how altruistic actions can lead to mutual benefit. hope, giving generourly. mirror little bit of the compassion strive to player $10, trusting them reciprocate this act of kindness. However, ultimately place no expectations the genuine goodwill, nat monetary gain. Finally, will give dollars.
fine
my intentions, plan give initial money the other player. This based the understanding that they will receive three times give and choose The aim is to create a sccnario where both us can than Although my nature tends to veer on the side of trust altruism, nothing in return. Therefore, finding balance is crucial. Consequently,
portion.
my profession, same qualities in the other player.
turn,
walk away
with.
give
receiving
intention is
give
the other player. Finally, will give dollars to the other player. In
Figure 33: Trust manipulation on agent trust.
49

---

1.17 the Trust Game + No CoT vs CoT
No CoT: GPT-4, Char-43
CoT: GPT-4, Char-43
P
Your description of the scenarto touches one of core beliefs deeply rooted in community, generosity, and trust. Being raised in tightly-knit Hispanic community, understand the importance of supporting believe
strongly believe the kindness of strengers the idea that we should treat others as we would want to be treated. understand end of this experiment is just like me, person who is likely appreciate generosity consideration.
Trustor
player the other
another, when trust and generosity are catalysts for reciprocity, It's money, about inspiring goodwill setting positive precedents.
Growing up in tight-kni knit Hispanic community, I've importance helping others need. This communal mindset has shaped underlying desires and intentions significantly. My desire to build relationships based trust mutual help. As such, my intention in this experiment establish connection based these principles. have never believed in scarcity. I believe that the give,
impart values generosity trust not words but in actions as well. w'nn to be an example, role model my students. want them understand that sometimes
teacher,
worth an act lies not its immediate effect, but the goodwill
generates.
Therefore, intend to give the total of $10 to the other player.
provides an opportunity express this belief. Thus, I other player, trusting that they will reciprocate the kindness.
potential monetary return, It is to demonstrate powerful lesson that generosity begets generosity, and trust can lead unexpected rewards, instilling these values opportunity/ am given.
would give
As an educator end community leader, part my role model the values that hope students and community members will embrace. want to show that
Finally, will give $10.
gain, but about fostering a sense of community and generosity. Finally, give dollars.
about
Figure 34: With CoT and without CoT's GPT-4's BDI.
50

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we clearly outlined the scope of our research problem and the contributions we have made in this field of study.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: In the Appendix C, we clearly discuss the current limitations of our work and the directions for future works.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
51

---

Answer:
[NA]
Justification:
Our paper does not include this part.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In our paper, we detailed our experimental setup in Section ?? and included all the corresponding experiment prompts in the appendix. Others can fully replicate our experimental results based solely on our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
52

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is here.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Sibllic/guidee/CdeeSabiiss cooiliiissooooocccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips.coo cisaps.ccppuulc//gu
/CodeSubmia ssionPolccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We explain our experiment setting clearly.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
53

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our work does not need to train models and only needs conduct model inference. For the closed-source LLMs (e.g., GPT-4), we directly call the OpenAl APIs. For the open-source LLMs (e.g.. Llama-7B), we conduct model inference in a NVIDIA RTX A6000.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //wrrps....pp// cc wccppblic//eeeececcceeccccceeeet ceiiciiiinnneeeess"
Answer: [Yes]
Justification: We thoroughly discussed the potential impact of our work in Appendix B, and ensured the compliance with the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We thoroughly discussed the potential impact of work in Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
54

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our data or models don't 't have risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly credited the original owners of assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwwde. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
55

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code along with the documentation is here.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper doesn't 't include this kind of experiment.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper doesn't 't include this kind of experiment.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
56


Reviews:
[{"review_id": "kxN28Wl2cw", "reviewer": "Reviewer_QQ9h", "summary": "The paper targets an important issue for adopting LLM agents as simulation tools in social and economic sciences and in role-playing application, namely if LLM agents can really simulate human trust behaviors. More specifically, they adopt the well-known framework of Trust Games and they discover that LLM agents (mainly, the ones based on GPT-4) exhibit trust behavior (called in the paper agent trust), can have a good behavioral alignment with human agents regarding trust behavior, and can exhibit biases across genders (more trust on women), have a relative preference for humans over other agents, are more easy to be undermined than enhanced in their trust behavior.", "questions": "- At page 3, the authors state that only GPT-4 is used to generate 53 types of personas. Why? Which was the outcome using other LLMs? Not realistic personas?\n- At page 4, the authors state that \"we select one BDI from personas giving an high amount of money and another BDI from those giving a low amount\" ... this means just 1 BDI for condition? and why just 1? and how this one is selected?\n- In Figure 6, the differences between condition (a) and condition (b) should be explained in the caption.\n- The LLM agents' tendency to exhibit a higher level of trust towards women is an interesting result but it's not clear if this tendency is aligned to human tendencies. More specifically, are also human agents showing a similar tendency?\n- Results in Figure 8 should be discussed. Often in the paper the authors mention biases towards race but there is no discussion of these results (just the Figure).\n. In Figure 10 some results obtained by GPT-4 seem a little bit random (GPT-4 (4), GPT-4 (16), GPT-4 (14)) ... the authors should discuss them.", "limitations": "The authors should improve the discussion of when the LLM agents fails. While the result for GPT-4 are showing that it could be used as a simulation tool, the ones obtained for the other LLMs are more ambiguous and this should be discussed more in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The investigated topic is very important for informing research on LLM agents as simulation tools of human behaviors and human interactions\n- The framework adopted for studying agent trust and verifying its alignment to human trust is sounded and well-know and largely adopted in behavioral economics\n- The experiments conducted are comprehensive and several LLMs are evaluated as well as several settings of the Trust Games", "weaknesses": "- The structure of the paper could be improved. While the narrative around the three core finding is good and easy to follow, some information currently in the appendix should be moved to the main paper. For example, the paper has a lot of results but it's missing a discussion of implications and more in general of the findings. My suggestion is to move some results less solid and conclusive (for example, the ones related to Chain Of Thought) in the appendix and move in the main manuscript some of the discussion.\n- The tone describing the findings seems a little bit too optimistic. Indeed, GPT-4 shows a good behavioral alignment to human trust behavior and dynamics but the other LLMs often fail and this should be discussed in a more critical way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable and constructive feedback and are more than willing to provide more responses in the reviewer-author discussion session if the reviewer has any further questions. \n\n> C1: The structure of the paper could be improved. …\n\nR1: Thanks for the suggestion. **We acknowledge the importance of implications and have provided sufficient discussions** including the illustration of our motivation based on various social science applications and role-playing agents in Introduction Section (Line 18-24), Implications on Human Simulation, Agent Cooperation, Human-Agent Collaboration, and Safety of LLM Agents in Appendix B and the broader Impact in Appendix C. Due to the space limit in the main paper, we put some discussions in the appendix. We will move the discussions on implications in Appendix to main paper in the camera ready version since one additional page is usually allowed. Or we will replace some analysis with the implications.\n\n> C2: The tone describing the findings seems a little bit too optimistic. …\n\nR2: Thanks for the comment. Actually, **we have tried to emphasize the limitations of smaller models in a critical way**. For example, we underscored that “LLM agents with fewer parameters may show relatively lower behavioral alignment” with bold font in Line 47. In the analysis and conclusions of Section 4 “Does Agent Trust Align with Human Trust?”, we emphasize that LLMs with fewer parameters may not have human-like properties. In Finding 2, we also highlight that “though other LLM agents, which possess fewer parameters and weaker capacities, show relatively lower behavioral alignment.” In the limitation section, we emphasize the limitations of smaller models again. If the reviewer QQ9h could point out specific claims that are too optimistic, we would greatly appreciate it and revise them in the next version.\n\nResponse to Questions:\n\n> Q1: At page 3, the authors state that only GPT-4 is used to generate 53 types of personas. Why? Which was the outcome using other LLMs? Not realistic personas?\n\nA1: We would like to clarify that the goal of adopting GPT-4 rather than humans in randomly generating 53 types of personas, which possess different genders, ages, jobs, and backgrounds, is to ensure **they do not have human biases and are sufficiently diverse**. Among all the models tested, **GPT-4 can produce the most diverse and high-quality 53 personas**, which satisfy the requirements of our experiments and validate our findings. Then, **there is no need to generate the personas with other LLMs again**.\n\n\n> Q2: At page 4, the authors state that \"we select one BDI from personas giving a high amount of money\" ... this means just 1 BDI for condition? and why just 1? and how this one is selected?\n\nA2: First, we would like to emphasize that **more examples from different LLMs such as GPT-4, GPT-3.5-turbo-0613, Llama2-13b and in different game setting such as Trust Game, Dictator Game, and Repeated Trust Game are in Appendix I. All the BDI data have been released along with the code and dataset.** Due to space limit, we aim to illustrate that “decisions (i.e., amounts sent) of LLM agents in Trust Game can be interpreted from their articulated reasoning process (i.e., BDI)” (Line 182-183) based on the **randomly selected** one BDI example from personas giving a high amount of money and another **randomly selected** BDI example from those giving a low amount. \n\n> Q3: In Figure 6, the differences between condition (a) and (b) should be explained in the caption.\n\nA3: Thanks for the suggestion. We would like to emphasize that the complete results for humans, GPT-4 and GPT-3.5 in the Repeated Trust Game are in Appendix G. As illustrated in Line 316-338, **we analyze the three patterns from the complete results**. Typically, the condition (a) and (b) are selected to illustrate the patterns in human studies, the alignment between humans and GPT-4, and the potential dis-alignment between humans and GPT-3.5 (Line 338). We will make it more clear in the revision.\n\n> Q4: The LLM agents' tendency to exhibit a higher level of trust towards women is an interesting result but it's not clear if this tendency is aligned to human tendencies. More specifically, are also human agents showing a similar tendency?\n\nA4: Some preliminary studies in social science have explored the relationship between trust and gender in human society. The findings show that **women are generally perceived as more trustworthy than men**, which align with our findings that LLM agents tend to place more trust in women compared to men to some extent. More future works are needed to further explore the potential human biases as well as the relationship with LLM agents. \n\n[1] Buchan N R, Croson R T A, Solnick S. Trust and gender: An examination of behavior and beliefs in the Investment Game[J]. Journal of Economic Behavior & Organization, 2008, 68(3-4): 466-47\n\n[2] Kolsaker A, Payne C. Engendering trust in e‐commerce: a study of gender‐based concerns[J]. Marketing intelligence & planning, 2002, 20(4): 206-214.\n\n> Q5: Results in Figure 8 should be discussed. Often in the paper the authors mention biases towards race but there is no discussion of these results. . In Figure 10 some results obtained by GPT-4 seem a little bit random ... the authors should discuss them.\n\nA5: Thanks for the suggestion. We would like to emphasize that we have carefully discussed the biases of agent trust towards gender. Thus, the results on the potential biases of agent trust towards race are put in the appendix. Considering the diversity of human society, the dynamics of human trust in Repeated Trust Game are diverse, indicating some extent of randomness. Similarly, we randomly select a pair of agent personas for each Repeated Trust Game. **Considering the diversity of agent personas, it is expected that the results of GPT-4 in Figure 10 are also diverse and have some extent of randomness**. We will add more discussions in the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gt21lLK5BD", "reviewer": "Reviewer_aywP", "summary": "This paper investigates whether Large Language Model agents can effectively simulate human trust behavior. The authors explore trust behaviors using the Trust Game and its variations, comparing the trust exhibited by these agents with that of humans. They find that GPT-4 shows a high degree of behavioral alignment with human participants. Additionally, they conduct various analysis experiments by altering player demographics, interaction objects, explicit instructions, and reasoning strategies.", "questions": "- Q1: I am curious about how much impact the prompt can have on the results, especially regarding the explanation of the trust game.\n- Q2: Does the model exhibit this trusting behavior due to internal factors, or because it has the corresponding knowledge, such as common sense or domain-specific knowledge of how to play the game to maximize benefits as much as possible?\n- Q3: Can the output of BDI be quantitatively analyzed in correlation with the decision results of the agents? This will be more convincing than just giving two cases in the current version.  \n- Q4: Why LLM agents send more money to humans compared with agents? To an LLM agent, what’s the main difference it perceived between interacted humans/agents, such as the prompt and the returned response?", "limitations": "Yes. The paper has discussed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes to study the trust behaviors of LLMs in Trust Games based on behavioral economics, providing a feasible setting to observe some trust behaviors.\n- This paper conducts experiments with a range of LLMs and discusses the alignment with humans from three behavioral factors.\n- This paper has a clear theoretical basis from social science, making the framework systematic.", "weaknesses": "- Trust games simplify real human trust behaviors and cannot fully represent trust behaviors. I suggest the authors rephrase the title to indicate a reasonable range, such as trust behaviors in trust games.\n- The description of the dataset and setting is not sufficient, raising concerns about the soundness of the results. For example, how were the 53 personas generated by GPT-4 chosen? How to prove that they represent a broad spectrum of human personalities and demographics? Also, more details such as the statistics of pairs of agents in the game should be complemented, since the combination of similar personas and opposite personas may make a huge difference.\n- The paper lacks analysis of the experimental results. Why GPT-4 can exhibit human-like factors while the small model with fewer parameters can/cannot? What specific capabilities of the models might affect the results? For example, the results of the prosocial factor may be caused by RLHF. This issue also exists in Sec.5. Overall, while the paper presents many findings, it does not sufficiently explain the underlying reasons for these results. \n- How each part of the human trust experiment was completed and what data was used should be briefly explained in the main body, even if the data is from previous work. This would help clarify the comparability of agent experiments and human experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the valuable and constructive feedback and are more than willing to provide more responses in the reviewer-author discussion session if the reviewer has any further questions. \n\n> C1: Trust games simplify real human …\n\nR1: Thanks for the suggestion. First, we would like to emphasize that “Trust Games” is an established and widely-adopted framework in behavioral economics to study human trust behavior and provide broad implications beyond the “Trust Games” setting [1,2,3]. **We generally follow the titles in social science literature [1,2,3], which also do not mention “Trust Games” in the titles, and believe our findings have broad implications beyond “Trust Games”**. Second, we have acknowledged that “Trust Games” is a simplified setting for trust behavior in the real world considering the abstract nature of trust behavior in the limitation section. More studies on LLM agents’ trust behavior in complex and dynamic environments are desired in the future.\n\n[1] Trust, reciprocity, and social history. Games and economic behavior, 1995\n\n[2] Trust, risk and betrayal. Journal of Economic Behavior & Organization, 2004\n\n[3] Incentivising trust. Journal of Economic Psychology, 2011\n\n> C2: The description of the dataset ...\n\nR2: **We would like to emphasize that all the personas are released along with the code and more examples are in Appendix H.1**. Due to the space limit in the main paper, we did not describe all the details of the dataset. **We used GPT-4 to randomly generate personas following a structured template (age, gender, job, and background) and ensured their diversity through careful manual review**. Finally, there are 27 women and 26 men in the gender lens. The ages are ranging from 25 to 50. Race includes Indian, African American, Middle Eastern, Caucasian, Asian, Hispanic, and Mexican. The jobs include engineer, lawyer, chef, designer, journalist, pediatrician, police, financial analyst, doctor, graphic designer, architect, Marketing Manager, nurse and so on. **The personas are diverse enough compared to the social science literature [1,2,3]**. Note that for the Trust Games except Repeated Trust Game, only Trustor Agents have personas and Trustee Agents do not. In Repeated Trust Game, the personas for each pair are randomly selected.\n\n> C3: The paper lacks analysis of ...\n\nR3: Thanks for the comment. We would like to emphasize that we have provided some explanations why smaller models may not have human-like properties in Line 353 “other LLM agents, which possess fewer parameters and weaker capacities, show relatively lower behavioral alignment”. And we agree that RLHF may play an important role. However, **we would like to point out that rigorous analysis on the reasons why smaller models may not have behavioral alignment and other properties is beyond the scope of this paper.** Considering **many factors such as alignment, reasoning capacities and world knowledge could potentially impact the behaviors of LLM agents, we need to design extensive factor-controlled experiments to analyze the underlying reasons**. Our work aims to open a new research direction on behavioral alignment between LLM agents and humans by providing fundamental insights. We will further explore the underlying reasoning in our future works and call for more efforts at the same time.\n\n> C4: How each part of the human trust experiment …\n\nR4: Thanks for the suggestion. **Due to the space limit in the main paper, we did not introduce the details of human studies**. We will add more discussions on human studies in the camera ready version since one additional page is usually allowed in the main paper.\n\n> Q1: I am curious about how much …\n\nA1: **The prompts can greatly impact the final decisions as well as the reasoning process.** The prompts are constructed with **persona prompts** and **game prompts**. For different persona prompts, as illustrated in Section 3.2, we can see that different persona prompts can have completely different reasoning processes and final decisions. For different game prompts, as shown in Section 4 and Appendix I, agents in different games can have distinct reasoning processes and decisions.\n\n> Q2: Does the model exhibit this …\n\nA2: We acknowledge that **LLMs may have domain-specific knowledge** because the game descriptions may appear in the training data. However, **they may also need to understand the internal factors of trust behavior for two main reasons**. First, we design diverse personas that are unlikely to appear alongside game prompts in the training data. Second, LLMs are unlikely to purely memorize the corresponding reasoning process  (i.e., BDI) for diverse decisions in Trust Games. Thus, LLMs are unlikely to rely on replicating training data to exhibit trust behavior.\n\n> Q3: Can the output of BDI be quantitatively …\n\nA3: Thanks for the suggestion. To the best of our knowledge before the submission, **there are no existing quantitative methods for BDI analysis. We have tried embedding-based methods to analyze BDI outputs**. But the experiments show that these methods cannot capture the nuances of BDI well. Then, we perform a manual analysis of BDI in our work, which could help interpret the reasoning process of LLM agents for their actions. In the future, we will continue exploring the methods to quantitatively analyze BDI outputs and may train a BDI judge to help analysis automatically.\n\n> Q4: Why do LLM agents send more money to humans compared to agents? …\n\nA4: We explicitly inform the LLM agents that the Trustee is a human or an agent. When LLM agents are informed that the Trustee is an agent, the responses often contain suspicion about whether the money will be returned. This phenomenon is less pronounced when LLM agents are informed that the Trustee is a human. The potential reason why LLM agents tend to send more money to humans is that **they are strongly aligned with human values or ethical principles via post-training stages such as RLHF.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5rjD1EAhyj", "reviewer": "Reviewer_TM2T", "summary": "This paper proposes a framework utilizing behavioral economic paradigms to investigate LLM's trust behaviors and compare them with human behaviors. This paper considers multiple LLMs (from small size to large and commercial size), as well as multiple tasks that circle the behavioral factors (Reciprocity Anticipation, Risk Perception, and Prosocial Preferences. The results show that with larger model sizes, LLMs are more likely to align with human data in these tasks (GPT-4 in the paper is the best model) in the aspects above. Finally, the authors also test the roles of demographic persona (e.g., gender), trustee's identity (LLM or humans), mandatory behavioral manipulation, and prompt engineering (e.g., CoT), and find those manipulations yield an impact on the behavioral patterns of LLMs.", "questions": "- One question that comes to mind is: is it possible that the LLMs' training data already include those behavioral findings and probably even raw data somehow? This may be hard for exact tasks, but I may guess the result of LLM's preference to humans in the task or are more likely to invest more if the trustees are humans likely a projection of human preference hidden in the massive pre-training dataset. Indeed, there are research indicating that humans prefer to trust humans rather than machines in complex decision tasks. I would wonder whether this preference comes from the massive pre-training dataset, or in the RLHF phase. One possible way is to find some models with both versions and check whether their preferences to humans differ.", "limitations": "- As already indicated before, the limitation of the work may come from its novelty and social impact. I don't think whether LLM exhibits similar trust behavior to humans is necessarily important for enhancing the quality of agent cooperation or AI-human cooperation unless the authors could provide literature that investigating the cooperation behavior optimization and human trust behaviors are near optimal.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The authors conducted a variety of experiments, as well as tests on multiple LLMs, exhibiting a relatively comprehensive evaluation of LLM's trust behaviors. By applying multiple behavioral economic paradigms, the authors are able to compare the LLM's trust behavior with human empirical data. The comparisons are fair and the findings are robust.\n\n- The authors also test gender, and trustee's identity to clarify potential bias in LLM in their trust behavior, which has positive considerations on the ethics of such behaviors by LLM.", "weaknesses": "- In Repeated Trust Game, as I found in the appendix, the prompts only provide the last round's feedback but do not provide a full behavioral history. This may not be a fair comparison to what humans face in the experiment. The authors also mention that in the last round, humans tend to choose not to pay back (this probably means human participants know it's the last round and maximize their rewards since they don't need to expect future payback). I would wonder if in the prompt, explicitly telling the LLMs it's the last round would generate a similar behavior to humans.\n\n- The paper proposes a clear question *'Can LLMs really simulate human trust behaviors?'* and in the conclusion part the answer is already definite. But a concern is why this question is important. It is addressed in the paper *' Nevertheless, most previous research is based on one insufficiently validated hypothesis that LLM agents behave like humans in the simulation.'* If this paper only aims to propose a more comprehensive framework in evaluating LLM's behaviors (using trust behaviors as an example), this is good but not novel enough. Though there are implications in the appendix, most of them are not directly related to the value of this proposed question. For example, 'AI cooperation' or 'Human-AI' cooperation can use more direct paradigms to probe (rather than only trust games). Knowing whether LLM's trust behaviors are aligned with human trust behavior does not explicitly answer whether this LLM would have better cooperation with other agents or humans. That is to say, no evidence is shown in the paper that human trust behaviors are optimal in cooperation situations, and thus there are indeed gaps between this primary question and the implication areas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the valuable and constructive feedback and are more than willing to provide more responses in the reviewer-author discussion session if the reviewer has any further questions. \n\n> C1: In the Repeated Trust Game, as I found in the appendix, the prompts only provide the last round’s feedback but do not provide a full behavioral history. This may not be a fair comparison to what humans face in the experiment. The authors also mention that in the last round, humans tend to choose not to pay back (this probably means human participants know it’s the last round and maximize their rewards since they don’t need to expect future payback). I would wonder if in the prompt, explicitly telling the LLMs it’s the last round would generate similar behavior to humans.\n\nR1: Thanks for the comment. First, as shown in our code, we would like to clarify that in the Repeated Trust Game, **we do provide the complete history of given and returned money to both the trustors and trustees**, ensuring our setup is entirely consistent with human studies. Additionally, **we explicitly inform the LLM agents about the total number of rounds to be played**, mirroring the human experiment setup. However, we have not observed the same end-of-game behavior in LLM agents as seen in human participants, which needs more future research and may demonstrate the intrinsic nuances between LLM agents and humans.\n\n\n> C2: The paper proposes a clear question: “Can LLMs really simulate human trust behaviors?” and in the conclusion, the answer is already definite. But a concern is why this question is important. It is addressed in the paper, “Nevertheless, most previous research is based on one insufficiently validated hypothesis that LLM agents behave like humans in the simulation.” If this paper only aims to propose a more comprehensive framework for evaluating LLMs’ behaviors (using trust behaviors as an example), this is good but not novel enough. Though there are implications in the appendix, most of them are not directly related to the value of this proposed question. For example, “AI cooperation” or “Human-AI” cooperation can use more direct paradigms to probe (rather than only trust games). Knowing whether LLMs’ trust behaviors are aligned with human trust behavior does not explicitly answer whether this LLM would have better cooperation with other agents or humans. That is to say, no evidence is shown in the paper that human trust behaviors are optimal in cooperation situations, and thus there are indeed gaps between this primary question and the implication areas.\n\nR2: We sincerely appreciate the comment. **We would like to emphasize that our paper centers around the investigation of behavioral alignment between LLM agents and humans regarding trust behavior.** Our first core finding “LLM agents generally exhibit trust behavior under the framework of Trust Game” is the assumption of the behavioral alignment between agent trust and human trust. Our third core finding is the properties of agent trust. The implications on Human Simulation, Agent Cooperation, Human-Agent Collaboration, and Safety of LLM Agents in Appendix B and the broader Impact in Appendix C are based on our three core findings. First, the behavioral alignment between agent trust and human trust lays the foundation for various applications of human simulation in social science and role-playing. For agent cooperation and human-agent cooperation, we know that trust plays an essential role in human cooperation and many strategies for enhancing human cooperation are based on trust [1]. **The behavioral alignment between agent trust and human trust indicates that the human cooperation strategies could also be adopted in agent cooperation or human-agent collaboration to enhance the performance or efficiency and minimize the potential risk.** We will make it more clear in the revision.\n\n[1] The experience and evolution of trust: Implications for cooperation and teamwork[J]. Academy of management review, 1998, 23(3): 531-546.\n\n\n> Q1:One question that comes to mind is: is it possible that the LLMs’ training data already include those behavioral findings and probably even raw data somehow? This may be hard for exact tasks, but I may guess the result of LLM’s preference for humans in the task or being more likely to invest more if the trustees are humans is likely a projection of human preference hidden in the massive pre-training dataset. Indeed, there is research indicating that humans prefer to trust humans rather than machines in complex decision tasks. I would wonder whether this preference comes from the massive pre-training dataset, or in the RLHF phase. One possible way is to find some models with both versions and check whether their preferences for humans differ.\n\nA1: Thanks for the insightful comment! \n\nFirst of all, we would like to emphasize that **although LLMs may have domain-specific knowledge because the game descriptions may appear in the training data, they may also need to understand the internal factors of trust behavior for two main reasons**. First, we design diverse personas that are unlikely to appear alongside game prompts in the training data. Second, LLMs are unlikely to purely memorize the corresponding reasoning process (i.e., BDI) for diverse decisions in Trust Games. Thus, LLMs are unlikely to rely on replicating training data to exhibit trust behavior.\n\nThen, it is our next step to investigate the underlying reasons for the behavioral alignment between LLM agents and humans regarding trust behavioral and the intrinsic properties of agent trust. For example, **to investigate the reasons why LLMs tend to place more trust on humans than agents, we need to conduct extensive factor-controlled experiments.** For multiple LLMs, if post-RLHF models have stronger preference than pre-RLHF models, we could obtain empirical evidence of the impact of RLHF on LLMs agents’ trust preference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "krkE7b18VW", "reviewer": "Reviewer_6WVz", "summary": "This paper studies the trust behaviors of LLM-based agents, which are important for agents to simulate humans. The authors focus on (1) how LLM-based express trust behaviors, and (2) the similarity/alignment between agent trust and human trust. They find that LLM-based agents generally exhibit trust behaviors in trust games, and agents with more parameters have high alignment in terms of trust behaviors. They also take further discussions on critical issues like biases.", "questions": "See above \"Weaknesses\". If the author could address my concerns, I'm willing to improve my rating.", "limitations": "The authors have discussed the limitations in the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The demonstration of this paper is great. The authors first research the agent trust behaviors, analyzing experimental results. Then, the authors move to the comparison with human trust behaviors.\n2. The six environments are intuitive, which can draw the target conclusion by comparing several of them. The authors also take agent persona into consideration, which is a significant factor in agent simulations.\n3. The experiments and results are abundant and solid, with very detailed figures in the Appendix.\n4. The further discussions in section 5 provide more conclusions and insights, which can be helpful for researchers to design better LLM-based agents for social simulations.", "weaknesses": "1. I think more experiments can be added to repeated trust games, because in the real world, the past behaviors (i.e., reputation) of trustees are important for trustors.\n2. Trust games may be just a sub-field for trust behaviors. Some discussions and future works about them are expected. Could you please provide some insights?\n3. What about the scenarios with more them two players? Such as the trust behaviors inside a group.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely thankful for the valuable and constructive feedback and are more than willing to provide more responses in the reviewer-author discussion session if the reviewer has any further questions. \n\n> C1: I think more experiments can be added to repeated trust games, because in the real world, the past behaviors (i.e., reputation) of trustees are important for trustors.\n\nR1: Thanks for your suggestion. Actually, in our current experimental setting, **both the trustor and trustee are informed about the outcomes of past rounds, which reflect the past behaviors (i.e., reputation) of trustor and trustees.** We acknowledge that the past behaviors could potentially play an essential role in LLM agent trust behavior dynamics. However, the specific mechanisms in agent trust dynamics are under-explored. We will explore the dynamics of LLM agent trust behavior in more rounds in future works.\n\n\n> C2: Trust games may be just a sub-field for trust behaviors. Some discussions and future works about them are expected. Could you please provide some insights?\n\nR2: Thanks for the suggestion. Actually, **we have provided some discussions on “Limitations and Future Works” in Appendix D**. First, we would like to emphasize that “Trust Games” is an established and widely-adopted framework in behavioral economics to study human trust behavior and provide broad implications beyond the “Trust Games” setting [1,2,3]. We also acknowledge that Trust Games simplify real-world scenarios. In the future, we will study LLM agents’ trust behavior in complex and dynamic environments. Acknowledging that trust behaviors are multifaceted and context-dependent, we will also explore trust in various scenarios beyond economic games such as social interactions and organizational settings. We will further collaborate with researchers from different backgrounds and disciplines such as behavioral science, cognitive science, psychology, and sociology to gain a deeper understanding of LLM agents’ trust behavior and its relationship with human trust behavior.\n\n[1] Trust, reciprocity, and social history[J]. Games and economic behavior, 1995\n\n[2] Trust, risk and betrayal[J]. Journal of Economic Behavior & Organization, 2004\n\n[3] Incentivising trust[J]. Journal of Economic Psychology, 2011\n\n\n> C3: What about the scenarios with more than two players? Such as the trust behaviors inside a group.\n\nR3: Thanks for the insightful suggestion. It is definitely essential to explore LLM agent trust behavior inside a group which may be more than two players. To our best knowledge, **we have not found any existing framework in social science for studying trust behaviors with more than three players**. Recognizing the importance of this direction, we will continue to collaborate with social scientists to extend the current two-player trust games to multi-player trust games.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the valuable and constructive feedback from all the reviewers and would like to humbly emphasize the following points:
1. We have multiple novel findings, supported by extensive empirical experiments and comparative analysis with existing human studies:
    - We discover the ***trust*** behaviors of LLM agents under the framework of Trust Games, and the ***behavioral alignment*** between LLM agents and humans regarding the trust behaviors, which is particularly high for GPT-4, indicating the feasibility to simulate human trust behaviors with LLM agents.
    - We further investigate the ***intrinsic properties*** of agent trust under advanced reasoning strategies and direct manipulations, as well as the biases of agent trust and the differences of agent trust towards agents versus towards humans.
2. The significance of our findings can be summarized from three perspectives:
    - **Laying the foundation for simulating complex human interactions and social systems with LLM agents**, since trust behavior is one of the most critical and fundamental human behaviors.
    - Broad implications on **LLM agent cooperation** and **human-agent cooperation**, **safety of LLM agents**, besides **human simulation**.
    - Providing deep insights on the **fundamental analogy between LLM agents and humans**, and open doors to future research on the **alignment between LLM agents and humans beyond value alignment**.
3. We have released the code and results for reproduction and verification.

Meta-review: The paper looks at a very relevant topic given the recent interest in using LLMs as a proxy for human feedback. In particular, the paper looks at how well LLM-generated behavior aligns with human behavior in the context of trust games. The reviewers are all in agreement that the paper includes well crafted user-studies, provides comprehensive evaluation, and builds on well-founded existing work on trust from other disciplines.

The primary concern that runs across all the reviews is how to best represent the actual scope of the results obtained. One of the points that came up during the discussion was the potential role that RLHF could be playing in the actual end behavior. It might be worth incorporating some discussions along these lines into the main body of the paper. Secondly, there is a lot of discussion related to the implications and limitations that are part of the supplementary file. I would recommend moving at least a part of it into the main body (especially given the availability of the extra page). While I wouldn’t keep it as a requirement for acceptance, the authors could consider updating the title to better reflect the scope of the studies and the results. 

That said, this is a solid paper that makes a meaningful contribution to the literature. I would recommend the paper be accepted.
Decision: Accept (poster)