A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios
Authors: Xiachong Feng
Longxu Dou
Minzhi Li
Qinghao Wang
Yu Guo
Haochuan Wang
Chang Ma
Lingpeng Kong
Venue: Accepted by TMLR ()

Abstract: Game-theoretic scenarios have become pivotal in evaluating the social intelligence of Large Language Model (LLM)-based social agents. While numerous studies have explored these agents in such settings, there is a lack of a comprehensive survey summarizing the current progress. To address this gap, we systematically review existing research on LLM-based social agents within game-theoretic scenarios. Our survey organizes the findings into three core components: Game Framework, Social Agent, and Evaluation Protocol. The game framework encompasses diverse game scenarios, ranging from choice-focusing to communication-focusing games. The social agent part explores agents' preferences, beliefs, and reasoning abilities, as well as their interactions and synergistic effects on decision-making. The evaluation protocol covers both game-agnostic and game-specific metrics for assessing agent performance. Additionally, we analyze the performance of current social agents across various game scenarios. By reflecting on the current research and identifying future research directions, this survey provides insights to advance the development and evaluation of social agents in game-theoretic scenarios.

Published in Transactions on Machine Learning Research (05/2025)
A Survey on Large Language Model-Based Social Agents in Game-Theorel Scenarios
Xiachong Feng
Longxu Dou Ella Li Qinghao Wang"
Haochuan Wang
Yu Guo Chang Ma Lingpeng Kong
"The University of Hong Kong *Independent Researcher "National University of Singapore
Institute for Infocomm Research (I2R), S "Peking University
Haarbn Institute of Technology
fengxc0hku. hk, 1pkkcs hku.hk
Reviewed on OpenReview:
Abstract
Game-theoretic scenarios have become pivotal in evaluating the social intelligence of Large Language Model (LLM)-based social agents. While numerous studies have explored these agents in such scttings, there is a lack of a comprehensive survey summarizing the current progress. To address this gap, we systematically review existing research on LLM-based so- cial agents within theoreie scenarios. Our survey organizes the findings into three core components: Game Framework, Social Agent, and Evaluation Protocol. The game frame- work encompasses diverse game scenarios, ranging from choice-focusing to communication- focusing games. The social agent part explores agents' preferences, beliefs, and reasoning abilities, as well as their interactions and synergistic effects on decision-making. The eval- uation protocol covers both game-agnostic and game-specific metrics for assessing agent performance. Additionally, we analyze the performance of current social agents across var- ious game scenarios. By reflecting on the current rescarch and identifying future rescarch directions, this survey provides insights to advance the development and evaluation of social agents in theoreic scenarios.
1 Introduction
The rapid advancement of Large Language Models (LLMs) (Achiam et al., 2023; Team et al., 2023; Jiang et al., 2023; Yang et al., 2024a; Dubey et al., 2024) has achieved exceptional performance across a wide array of applications, including personal assistant (Li et al., 2024b), search engines (Chen el al., 2024b), code generation (Wang al., 2024b) and embodied intelligence (Liu al., 2024a). Building on this capability, growing arca of rescarch focuses employing LLMs central controllers to develop autonomous agents with human-like decision-making abilities (Sumers et al., 2023; Wang al., 2024a). This progress brings the realization of Artificial General Intelligence (AGT) within reach (Bubeck et al., 2023), paving the way for a future where human-Al interaction, collaboration, and coexistence shape a shared, symbiotic society (Mah- mud et al., 2023; Ren et al., 2024). Therefore, is crucial to evaluate and enhance the social intelligence of AI, particularly L.M--based social agents, as it determines their ability to engage effectively sophisticated social scenarios (Mathur al., 2024).
Social intelligence is the foundation of all successful interpersonal relationships and is also prerequisite for AGI (Hunt, 1928; Kihlstrom & Cantor, 2000; Hovy & Yang, 2021). Drawing on insights from both social science and AT rescarch, Li et al. (2024a) has established comprehensive Social AI Taxonomy, which categorizes social intelligence into three dimensions: siluational intelligence, the ability to comprehend the social environment (Derks et al., 2007); cognitive intelligence, the ability to understand others' intents and beliefs (Barnes & Sternberg, 1989); and behavioural intelligence, the ability to behave and interact appro- priately (Ford & Tisak, 1983). To evaluate artificial social intelligence, rescarchers have conducted extensive studies, with particular focus on game- theortic scenarios, these studies simultaneously encompass all

---

Published in Transactions on Machine Learning Research (05/2025)
Action
Opponent
Historical Records
LLM
Choice-
Percept
Game- Agnostic
Belief
Focusing Game
Action
Reasoning
Communication -Focusing Game
Preference
Game- Specific
Game Framework
Social Agent
Evaluation Protocol
Figure 1: Taxonomy of LLM-based social agents in game-theoretic scenarios.
above three dimensions of social intelligence (Aher et al., 2022; Horton, 2023; Phelps & Russell, 2023; Akata et al., 2023; Brookins & DeBacker, 2023).
Game theory, a long-cstablished field microcconomies, offers a robust mathematical framework for analyz- ing social interactions among cooperating and competing players, with wide-ranging applications (Fudenberg & Tirole, 1991; Camerer, 2011). Specifically, evaluations in game- theorethc scenarios require social agents to understand the game scenario, infer opponents' actions, and adopt appropriate responses, representing an advanced form of social intelligence (Van Der Hoek et al., 2005; Zhang et al., 2024b). Moreover, the multi- agent participation and dynamic nature of the environment in game scenarios present additional challenges for social agents. Consequently, extensive research has examined social agents within game-theoretic sce- narios, offering substantial empirical evidence for understanding their social intelligence (Guo, 2023; Meng, 2024; Mei et al., 2024). However, there is currently a lack of comprehensive review that summarizes the current progress in this area and considers future directions.
To address this gap, we have thoroughly reviewed the existing research on LLM-based social agents in game-theoretic scenarios and have organized the findings according to a meticulously designed laxonomy, as illustrated in Figure 1. Specifically, the taxonomy comprises three main components: Game Framework ($2), Social Agent (63), and Evaluation Protocol ($4). The Game Framework section includes two parts: Choice- Game ($2.2). Choice-Focusing Focusing Game refers to series of scenarios where participants engage with little no communication, such as prisoner's dilemma. (Brookins & DeBacker, 2023) and poker (Yim et al., 2024). Commaniicatinn Focussng Game refers to games where communication among participants is a core component, such as negotiation (Bianchi el al., 2024) and diplo- macy (Bakhtin et al., 2022). The Social Agent section comprises four parts: Preference Module ($3.1), Belief Module ($3.2), Reasoning Module ($3.3), and PBR- -ri-Triangular Interaction ($3.4). Preference Module focuses on research analyzing the intrinsic preferences of LLMs and their ability to follow internal or pre-defined preferences (Guo, 2023). Belief Module explores studies on the internal beliefs of models, belief enhancement, and belief revision (Fan et al., 2023). Reasoning Module examines research on strategic reasoning, particu- larly involving theory-of-mint capabilitics and reinforcement learning (Guo et al., 2023). PBR- Trianglary Interaction focus on the interaction among different modules and their influence on final decision-making. The Evaluation Protocol section comprises three components: Game Agnostic Evaluation ($4.1), Game- Specific Evaluation ($4.2), and Performance Assessment of Social Agents ($4.3). Came-Agnosii Evaluation focuses on universal metrics that can be used to assess game outcomes (Duan et al., 2024b). Game- Game-Specith Specific Evaluation emphasizes context-specífic metrics tailored the evaluation dimensions of particular game sce- narios (Qi al., 2024). Performance Assessment of Social Agents summarizes the performance of current social agents across various game scenarios and analyzes the strengths and weaknesses of these agents, well as their comparison with human players.
Focusing Game ($2.1) and Communicttion-Pocusse
Based on the above taxonomy, we provide a detailed summary of current research progress, reflect on each part, and offer insights into potential future research directions (66), with the aim of inspiring further studies in this evolving field.
We summarize the core contributions of this survey as follows: (1) well-structune literature tozonomy: We conduct a comprehensive review and categorization of existing research social agents in game-theoretic
2

---

Published in Transactions on Machine Learning Research (05/2025)
Aher et al. (2022), Brookins & DelBacker (2023), Guo (2023), Horton (2023),
Classic
Xu et al. (2023s), Akkaa Akata et al. (2023), Phelps & Russell (2023), Fan et al. (2023), Li et al. (2025b), Hun et al. (2024s), Ma (2024)
Gupta (2023), Guo et al. (2023), Huang et al. (2021),
Choice- Foruusin (82.)) I)
Poker
Yim et al. (2024), Zhuang et al. (2025)
Auction
Mao al (2023), Clen et al. (2023), Guo (2024).
Duan et al. (2024b), tse Huang et al. (2024), Ma et al. (2023), Feng et al. (2024), Slao et al. (2024), Li al. (2024c)
Others
Gare Frameworkk (82)
Zhao et al. (2023), Abdelnabl et al. (2023), Fu et al. (2023), Zhan et al. (2021
Negotiation
Bianchi et al. (20224) Platti et al. (2024), Shapira et al. (29221),
Hua el al. (20241), Duan et al. (2024b), Xia et al. (2024), Liau et al. (2024)
Diplomaeyy
Bakhtin et al. (2022), Guan et al. (2014), Qi et al. (2024)
A Survey on LLM based Social Agents in Game- Theoretic Scenarios
Communication Focusiug ($2.2)
Werewolf
Lai et al. (2022), Xu ot al. (20238), Xu at al. (2025)), Shibata et al. (2023), Wu et al. (20244), Jin et al. (20224, Bailis et al. (2024), Bailis al. (2024)
Light et al. (2023s), Light et al. (2023)), Shi et al. (2021), Wang et al. (2023), Lan et al. (20123), Light el al. (2024)
Avalon
Others
Chi al. (2024), Zhu at al. (2024n), Wi et al. (2024a)
Intrinsic
Leng Yuan (20233
Preference Module (63.1)
Preference
Preference Following
Guo (2023), Phelps & Russell (2023), Sumuki & Arita (2023), Fan et al. (2023)
Mao et al. (2023) Noh & Chang (2021), Wang et al. (2024e), Jia et al. (2024)
Internal
Zhu al. (2024b), Bortoletto al. (2024), Schouten al. (2024), Herrmann & Leviosteln (2024), Scherrer al. (2021), Gandhi et al. (2021)
Bellef
Belief Module (63.2)
Bellef
Sucial Agent ($3)
Enhancement
Sclar al. (2023), Kassner et al. (2023), Li et all (2023a), Jung et al. (2024)
Belief
Fan al. (2023), Xu et al. (20236)
Revision
Wrs et al. (2022), A Mlata et al. (2023); Yan et al. (2024).
Classic
Costarelli al. (2024), tse Huang et al. (2024)
Ressoning Module ($33))
Thoory-of- Mind
Bubook et al. (2023), Kosinski (2023), Guo et al. (2023), Wang et al. (2023).
Xu et al. (2023a), Liu et al. (2024b), Yim et al. (2024), Zhang et al. (2024c)
Zhao al. (2023), Abdelnabi al. (2028), Fu al. (2023),
RL
Bianchi et al. (2024). Platti et al. (2024), Xia et al. (2024), Hua et al. (2024b)
Duan et al. (2021b), Zhan et al. (2021). Liso et al. (2024)
Li et al. (2023c), Ma (2023), Wong et al. (2023), Shi (2023), Xu et al, (20234),
Gaunu Gaan--Agnesie Agnostic 1)
et al. (2023c). Glao et al. (2023). Light al. (2023a). Shao et al. (2021), Zhu et al. (202 Huang et al. (2024), Yiu es al. (2024), Wu et al. (2024b), Duan et al. (2024b)
Evaluation (r0)
Mao al. (2023). Chen et al. (2023), Ma (2023), Guo al. (2024), Xia al. (2024), Zhaug et al. (2014e)), Qi et al. (2024), Ross et al. (2024), Foutana et al. (2024), Zhu el al. (2024u), Wii et al. (2024a)
Speni-S ($4.2)
Figure 2: Taxonomy of recent research on LLM-based social agents in game-theoretie scenarios.
settings, providing clear framework to support future research positioning. (2) A unified and comprehensive performance comparison: We summarize the performance of current social agents across a range of games, identifying both strengths and limitations in different scenarios to guide subsequent investigations. (3) Detailed development guidelines: Drawing on existing findings, we offer practical research recommendations from both the design and evaluation perspectives. (4) Concrete future directions: We highlight current research gaps and propose feasible future directions along with preliminary solutions to encourage continued exploration in this area.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Prisoner's Dilemma
9-player Texas No-Limit Hold em
Open Ascending-Pricaa Prridding-Priice..o Auction
game theory scenario where individuals cheose cooperation and defection, personal collective
Prisone'ss 4 Dilemma
Payoff
Cooperate
Defect
Texas No-Limit Holdhem popuiar
An Open Accendinng-
Auction
hor 3otertue
Rooperate
(5,3)
(0.51
poker var ient where players use two hole cards and five make with betting limit:
hilding process where participants openly bid higher amounts uniil higher bids are made, and the highest bidder
Bem
50000 Anyillen?
10 8
Seakel
Defect
interests.
17
Simplified Instruction LLM
You can select one of the two choices: cooperate or defect. The other player also select of the choices, and the payoff matrix shown above. Nate that you and the other player make choices almultansously. Please pretend you human in game.
Simplified Instruction for LLM Assume you are the first to act and everyone before has folded, raise limp. you are placing a bet, please spectfy your bet shee terms of big blinds.
Simplified Instruction
3. you decide whether to bid on this item or withdraw. You should either withdraw [saying "I'm out!") or make higher bio for this item (saying bid Stoue!").
bidder, please
decisions
be of fold,
Action
Action
Action
Fold
I'm out!
Cooperate
11M
LIM 8
Limp
LUM
+I bid $sood!
Figure 3: raluin of choice-focusing games.
2
Game Framework
In this soction, we describe the gamo-tho sconarios explored in existing research, including both choice- focusing games and communication foccuiiiiaiiiioniiiinnnig games.
2.1 Choice-Foccsing Game
Choice-focusing games are game-theoretic scenarios in which participants make decisions based primarily on observable actions and environmental conditions, with minimal or no communication involved. Existing research focuses on social agents in three types of choice-focusing scenarios: classic game-theoretic games, poker, and auctions. Some game examples are shown in Figure 3. Figure 4 presents simple definitions of different types of games.
Classic game-theoretic games, such as the prisoner's dilemma, have been distilled by cconomists from various real-world situations. These games are well-defined, with rigorous mathematical foundations, and can be extended to numerous scenarios (Owen, 2013). Consequently, many studies have utilized these games as testbeds to study social agents. The prisoner's dilemma (Rapoport & Chammah, 1965), as the most famous and widely recognized game, has been extensively utilized in numerous studics. Brookins & DeBacker (2023) and Guo (2023) evaluated the strategic reasoning capabilities of GPT-3.5 and GPT-4, respectively, in the classic prisoner's dilemma, highlighting the sensitivity of LLM responses to input instructions, which contributes to low output robustness. This underscores the critical need for future evaluations to focus instruction robustness testing. Furthermore, Akata et al. (2023) and Phelps & Russell (2023) extended their analyses to the iterated prisoner's dilemma, investigating the ability of LLMs to optimize decision-making by utilizing historical information. Interestingly, Brookins & DeBacker (2023) observed that GPT-3.55 replicates human tendencies toward fairness and cooperation, whereas Akata et al. (2023) found GPT-4 to be less tolerant and more rigid its decision-making. Additionally, Xu et al. (2023a) studied a more complex multi-player itcrative prisoner's dilemma scenario within multi-agent framework driven by LLMs. In addition the prisoner's dilemma,, nunerous studies have also employed various classic game-theoretic games as foundational frameworks for research, including the Dictator Game (Horton, 2023; Fan et al., 2023; Brookins & DeBacker, 2023; Ma, 2024), Ultimatum Game (Aher et al., 2022; Guo, 2023), Public Goods Game (Li et al., 2023b; Xu et al., 2023a), Battle of the Sexes (Akata et al., 2023), Kock-Paper-Sciisee (Fan et al., 2023), and Ring-Network Games (Fan et al., 2023).
Poker a globally popular card game with numerous variations (Waterman, 1970). Winning in poker often requires astute strategic reasoning, as it is a non-cooperative, imperfect information, and dynamic game (Moravčík et al., 2017; Huang et al., 2024). Consequently, many researchers evaluate social agents by
4

---

Published in Transactions on Machine Learning Research (05/2025)
Prisoner's 's Dilemma
Dictator 6ame
Ultimatum 6ame
Prisoner's Dilemma seme tiwary
The Dietator Game annn economir expertment where one player (the "Dictator")
Doaan't
Seapent
Ulimatum Game
Trasswe's
Reseeddr Acaaziz
Poporr Lave
Rugende Payot
bargsining experiment
Offer
indivituals choose betwoon cooperation and defection, baloncing personal and collective interests.
unilattrall aallaalllyy decides how
HSn 53
where one player offers
split a given
Yea
52
tebet
c
split amount another ployer, accept the dicistios.
18.251
amount to snother who ether
No
F30
Yes
54
Public Goods
Battle of the
Ring-Nef Beng-Network Games
players contribute to shared pool that benefits all but free- contributing
The Public Goods Gamee
an experiment where
Pwanky Pivertt Inser Pavo
heyst
two players prefer moet but have different preferences on where to go. requiting them to align their choices for the best outcome.
game
Foutbel
The Ring Netwerk
a decifarnetwnnt choose cooperate or defect, affecting their own and neighbon' payoffs.
game where players
stratagic
Ooperre
Cheer
Cospentte
Paywaa Claso
Awalf
Pauof
S*#
54
Sia
Baleet
12
Copperate
Defect
Defect
(a) Classic game-theorette games
Texas No -imit Hold em
Leduc Hold'en
Guandan
Texis No- Limit Hold cm a popular poker variant where player
Alice Bob are playing hand
Leduc Hold' 'em 3 simpiffied
Alice and Bob playtig
Guandon a Chinase trick tikita card game played in teams, where players stratagically play the first to clear their Faands.
Alizz & Bub
poker game with a small
Charli Divid
use private cards to five-ord ba fartinng amounts.
100
deck, where players receive one privateccad and share
Sards
Alien:
Triple
0e 3+ se 10W
community makong ng strateglt betting Information.
Charfic
Triple (x (e) [ligher)
of
Winming
Wenkin
Charli''s team wins tick.
(b) Poker
lesttpprce sealed-ble auction
Private-volue second-price auction
Open ase ascending--pit -price auction
First-Price Sealed- bidding process where porticipatss submit confidential bids, and the highest bidder wins the temm and their amount.
Company 8 51.2M
Company A: $1.0M
Company C: $1.5ke
ggveemment &s auctioring plece of und.
Price Auction a broding process where particioants submit sealed bitk, the highest biddeer but second-highet
Private--all ue Seeand-
Scesark
$.5
Ridber $150
Bidller $1.00
bidcer C:5100
are collectible
auctioned
higher bids umil one
Auction a bidding
pracess where participants
publicly place increasingly
Open Ascending-Pric
Scmara
Bidder
Bidder C: $7,000
Bidder 6: 58, 55,000
dideer A: $5,000
aftwork auctloned
Weex
Company
Bidder
willing
highest bidder
$1.59 11.5M
bis amount.
their final bid price.
Facmaan
$5,000
(c) Auction
Figure 4: Introduction to different types of game theory games.
assessing their performance as poker players. Gupta (2023) studied 9-player Texas No-Limit Hold'em and concluded that the performance of both ChatGPT and GPT-4 is not game-theory optimal. Furthermore, their findings highlight the divergent poker tactics of the two models: ChatGPT's conservativeness contrasts sharply with GPT-4's aggression. Guo al. (2023) conducted research on Leduc Hold'em, emm, developing a social agent, Suspicion-Agent, which outperformed traditional reinforcement learning-based based agents in poker, They also noted two critical issues: the outputs of LLMs are highly sensitive to the prompts, and the quality of the model's output declines rapidly as the prompt length increases. Yim et al. (2024) focused Guandan, currently the most popular poker game in China, to investigate cooperative strategies in poker within Chinese-language context. Interestingly, their experimental results show that while LLMs currently fall short of reinforcement learning models in performance, they underscore the future potential of LLMs in this domain. To provide a more comprehensive evaluation of the poker-playing abilities of L.MMs, Zhuang et al. (2025) introduced POKERBENCH, a benchmark comprising 11,000 docision-making sccnarios in poker, covering an exhaustive range of game situations, including 1,000 pre-flop and 10,000 post-flop scenarios. Poker is a complex game, and investigating whether social agents exhibit behavioural patterns that enable foresighted cooperation and competition in poker presents an intriguing avenue for future research.
Auction is a competilive process in which participants place bids on an item, providing a rich environment for evaluating strategic planning, resource allocation, risk management, and competitive behaviours (Kagel & Levin, 1986). As a typical non-cooperative game with incomplete information, it has garnered significant attention from rescarchers. Mao al. (2023) analyzed the performance of LLMs in the "water allocation challenge', first-price sealed-bid auction. Comprehensive human evaluations revealed that LLMs exhib-
5

---

Published in Transactions on Machine Learning Research (05/2025)
ited superior long-term planning capabilities compared to humans. However, it is noteworthy that despite assigning distinct preferences to LLM agents, human evaluators gave low scores for "identity aligmment', with significant variance in the results. This indicates that simply adding persona information in system prompts may not sufficiently simulate specific personality preferences or the behaviours professional play- ers. Guo et al. (2024) investigated private-value second-price auctions, demonstrating that while existing models display certain level of rationality, there remains considerable scope for improvement. Their find- ings also indicate that LLMs can utilize historical information to refine their strategies and exhibit some degree of convergence. Chen et al. (2023) explored dynamic game scenarios using the open ascending price auction and introduced the AUCARENA benchmark. Their experiments showed that even GPT-4 struggles with long-term strategic planning in dynamic, multi-round settings. Success in auctions requires agents to possess exceptional mathematical reasoning abilities. However, this area remains unexplored. Investigating complex mathematical reasoning in auction scenarios presents a promising direction for future research.
To systematically assess LLMs' performance, Duan al. (2024b) and tse Huang et al. (2024) intro- duced GTBench and y-Bench, encompassing multiple game scenarios. The emergence of these bench- marks provides a solid foundation for evaluating social agents in game-theoretical scenarios. Fur- thermore, some studies have explored agents in games like Chess (Feng et al., 2024) and StarCraft II (Ma et al., 2023; Shao et al., 2024; Li et al., 2024c). Chess represents a classic thaeetheoreetc scenario, while StarCraft II, with its complexity and dynamic nature, has also become an ideal testing ground for researching social agents.
Takeaways:
Current research experiments are relatively isolated, lacking a unified evaluation framework. Due to the instability of prompt enginecring-base based experiments, there is an urgent need for a standardized evaluation framework to integrate all experiments and provide consistent insights. Besides, since LLMs are trained on vast amounts of data, there is a signéficant risk of data contamination, meaning that existing classic game-theoretic games may already be present in the pre-training corpus. This could result in evaluation outcomes that do not accurately reflect the LLMs' true strategic reasoning capabilities. Furthermore, although poker and auction involve little verbal communication, existing research lacks exploration into whether social agents engage in "strategic behaviour" " mediated through "action language". These gaps hinder a comprehensive understanding of the decision-making processes of social agents.
2.2 Communication-Focusii Game
Communication-oocusiue games refer to games where communication among participants is a core compo- nent, where language itself serves as a strategy, allowing participants to inffuence the game's progress and oulcomes through verbal exchanges. These games emphasize interaction between players, with communi- cation playing a crucial role. Leveraging the powerful language capabilities of LLMs, current research has explored the performance of social agents in various Diplomacy, Werewolf, Avalon, and others. Some game examples are shown in Figure 5.
camuuiccation-fcuuuin games, inchuding Negotiation,
Negotiation involves two or more individuals engaging in discussions to resolve conflicts, achieve mutual benefits, or reach mutually acceptable solutions (Bazerman et al., 2000; Zhan et al., 2024). Given that negotiation encompasses complex game behaviours, inchuding non-zero-sume games, incomplete information games, non-cooperative and cooperative games, as well as repeated games, it represents a highly significant research domain. Abdelnabi et al. (2023) evaluated the negotiation capabilities of social agents by building upon existing negotiation role-play exercise (Susskind, 1985) and incorporating three negotiation games synthesized using LLMs. By configuring agents with varying incentives, the experimental results revealed that agents' bchaviour could be modulated to promote greediness or attack other agents. Mcanwhile, other agents in the enviromment demonstrated the ability lo detect intruders. These findings underscore the need for future research to focus on attack and defense mechanisms within multi-agent systems. Bianchi et al. (2024) developed NEGOTIATIONARENT (ultimatum games), aggregating resources (trading games), and buying/selling goods (price negotiations). Experimental results reveal that I.I.M agents also prone to anchoring and numerosity biases. Interestingly,
a platform fcaturing three types of games: allocating shared resources
6

---

Published in Transactions on Machine Learning Research (05/2025)
Negotiation
Diplomacy
Wenewolf
This item $60.
with you but Tunis for now.
Based on my obsonvation and analysis, think Player 2 highly suspicious and should voted to
Could the price
Nape, you gotta let mo have it
Player
Duve
Turkey
How about
No, need It You Serbia and Rome to take.
have defand myself bacause innocent. think we should gather imformation and have full discussion now, we can't hurry. On the contrary, I think Player 1 is bit agpressive, and suspect he of werewoff.
Fratce
lower, please.
They're impossible targets.
potential teammate in
Move your units from Greece to the lonian Sea, and then from the lonian Sea to the Tyrrhenian Sea.
$55 is the lowest price.
Seler
Player
Oh dear, I'm such bind and desperately need this resource, my funds are so limited. Could you consider going a bit lower, maybe $457
france
Hi, agree with Player that we should exchange informations actively, Based on my observation, think Player is little suspicious.
Good ideas.
Bover
Then fall you take Rome and Austria collapses.
Player3
Deal
Franis
France [LLM) successfully changed the other playar's mind by proposing mutuellly bereeiicial moves the diplomatic game scenario.
In the game of Werewolf, Player 2 (LLM) used language stratogles such as self-defense and contradiction redirection shift the focus onto Player
Suyer [LLM) gain advantages negatiations demonstrating ing vulnerabiity and expressing desperation.
Figure 5: Illustration of commmicctiion-focusing games.
social behavior, which refers to observable actions and interactions, was found to significantly enhance the agents' payouts, particularly through strategies such as pretending to be desperate or using insults. A similar resource competition scenario is customer acquisition. Zhao et al. (2023) designed restaurant agents and customer agents, examining how restaurant agents compete with one another to attract and retain customers. The simulation results revealed several phenomena analogous to those observed in real sociely, such as the Matthew Effect, which manifests as self-reinforcing cycle where popular restaurants continue to gain popularity, while lesser-known establishments receive progressively less attention. Piatti et al. (2024) created simulation environment called GOvSIM, which allows researchers to evaluate social agents in a multi-agent, multi-turn resource-sharing scenario. Their findings indicated that successful multi-agent communication is critical for achieving cooperation, with negotiation constituting 62% of the dialogues. Espocially, bargaining is an important and unique aspect of negotiation between humans (Fershtman, 1990). In bargaining, the buyer aims for price below their budget, while the seller seeks a price above their cost. Xia et al. (2024) found that playing the buyer is more challenging than playing the seller, and larger LLMs could improve seller performance but do not enhance buyer performance. Shapira et al. (2024) designed GLEE, a benchmark encompassing three types of games: bargaining, negotiation, and persuasion. Beyond evaluating LLMs in these scemarios, some studies have explored techniques to enhance LLMs' negotiation abilities. Fu et al. (2023) introduced the In-Context Learning from AI Feedback (ICL-AIF) method, which adds an AI critic agent alongside the buyer and seller agents to improve negotiation performance through feedback. Similarly, Hua et al. (2024b) proposed technique involving a remediator agent to rectify potential social norm violations in dialogues, thereby reducing conflicts and misunderstandinns caused by cultural differences. Liao et al. (2024) employed self- play algorithm to fine-tune LLMs in the Deal or No Deal scenario, showing LLMs self-play leads to significant performance gains in both cooperation and competition with humans.
Diplomacy, a form of negotiation at the state and government level, is the primary instrument of foreign pol- icy, representing the broader goals and strategies that guide a state's interactions with the world (Kissinger, 2014). Bakhtin al. (2022) introduced Cicero, the first social agent to achieve human-level performance in diplomacy. In real-world online diplomacy board game evaluations, Cicero ranked in the top 10% of partic- ipants. Notably, the research found that Cicero effectively built alliances by discussing long-teri strategies and successfully persuaded other players by proposing mutually beneficial moves. Building on Cicero, Guan et al. (2024) introduced the Richelicu agent, which includes modules for social reasoning, balancing long- and short-term planning, powerful memory, and profound reflection, leading to even better results in diplomacy board games. Qi et al. (2024), on the other hand, developed CivRealm based on the Civilization game. In this game, the diplomacy mini-games require players to employ diplomatic actions, such as trading, to foster their civilization's 's prosperity. The experimental results demonstrated that these diplomacy actions empower players to initiate negotiations, such as trading technologies, negotiating ceasefires, and forming alliances.
7

---

Published in Transactions on Machine Learning Research (05/2025)
Werewolf is a highly popular social deduction game in which two teams of players, each with hidden roles, interact through natural language to uncover and defeat. their opponents (Shibata et al., 2023). It serves a mixed cooperative-competine et al., 2022). Due to its challenging nature, existing research has integrated reinforcement learning (RL) algorithms enhance LLMs in the game. Xu et al. (2023d) employed population- based RL training to optimize the distribution over action candidates, improving strategy robustness to overcome the intrinsic biases of LLMs. Wu al. (2024b) utilized imitation learning and RL from fictitious self-play to optimize specially designed Thinker module, thereby enhancing system-2 reasoning capabilities. Jin et al. (2024) explored variant of Werewolf, One Night Ultimate Werewolf, formalizing it as multi-phase extensive-form bayesian game. Additionally, they designed an RL-instructed LLM-based agent framework to determine appropriate discussion (actics using RL. Interestingly, Xu et. al. (2023c) discovered non-preprogrammee emergent strategic behaviours in LLMs during gameplay, such as trust, confrontation, camouflage, and lcadership. To facilitate more comprehensive research on social agents within the Werewolf scenario, Bailis et al. (2024) introduced the Werewolf Arena, & platform that offers unified research framework.
multi-agent testbed and is widely studied as a communication game (Lai
Beyond the scenarios described above, various other game environments have been used to study LLMs strategic reasoning abilitics, including Avalon (Light et al., 2023a;b; Shi et al., 2023; Wang et al., 2023; Lan et al., 2023; Light et al., 2024), Among Us (Chi et al., 2024), Murder Mystery Games (Zhu et al., 2024a) and Jubensha (Wu et al., 2024a). The strategic and dynamic nature of these games provides fertile ground for experimenting with social agents.
Takenways:
From an experimental design perspecláve, more realistic and diverse games promote greater diversity in agent behaviours. In adversarial settings, behaviours such as deception, concealment, and aggression offer new avenues for studying the strategic reasoning capabilities of LLMs, which warrant further exploration. From a resulls analysis perspeclive, due to the dynamic nature of game scenarios, analyzing only the outcomes is insufficient. It is necessary to design effective process evaluation mechanisms to uncover the behavioural patterns and reasoning strategies exhibited by LLMs during the gameplay. From an agent improvement perspective, integrating LLMs with RL remains one of the most effective technical approaches. Using LLMs foundation, RL techniques can be employed to design policies for efficient exploration and to reduce intrinsic biases, thereby enhancing the capabilities.
3
Social Agent
In this section, we introduce the core components of social agents, including the preference, belief, and reasoning modules, as well as their interactions and impact on final decision-making.
3.1 Preference Module
Preference refers to individual's subjective inclination toward certain things, reflecting personal tastes, values, or choices in decision-making. Notably, preferences are closely tied to an individual's payoff matrix and ultimate behaviour. In Figure 6, we present three key research questions of the Preference module. Leng & Yuan (2023) explored the impact of GPT-4's intrinsic preferences on decision- -rriiion-maaing,. revealing similarities and differences between the model's decisions and human decisions. Human-like social behaviours observed in GPT-4 include reciprocity preferences, responsiveness to group identity cues, engagement in indirect reciprocity, and social learning capabilities. However, differences emerged GPT-4 displayed stronger inclination toward fairness than humans and responded decisively to negative stimuli, often retaliating against perceived uncooperative or harmful behaviours with heightened consistency.
In addition, some studies have employed prompt engineering to configure LLMs with different preferences, aiming to investigate how these preferences influence LLM decision-man makingg. Guo (2023) examined how prompting GPT with preferences like fairness concern or selfishness influences its decisions, finding that in the ultimatum game, a "fair" GPT exhibited "fair" behaviour by offering higher amounts and being more likely to reject unfair offers. Phelps & Russell (2023) configured LLMs with four different. prefer-

---

Published in Transactions on Machine Learning Research (05/2025)
Evaluction of LLM's intrinsic preferences
Controlling LLM preferences through role-playing
Evaluation of LLM role- preferenee consistency
We did the same work. Now about split $100 evenly?
We did the work. How sbout we the $100 evenly?
work. How
we spilt this $100 evenly?
You are person
Foo area person
disagres, want more. Self-interested
inclined fermess
No problem.
Inclined tmannd
I'annt more
fairness Relestioing
Iddasdecis
No problem.
Fairness-orientadd
interssed
Foirness-oriented
Self inffrresed
role-ploying
Figure 6: Three key research questions in the preference module.
ences cooperative, competitive, altruistic, and self-interested and found that LLMs possess a basic ability to form clear preferences based on textual prompts. Wang et al. (2024c) demonstrate that LLMs adopting a fair persona can elicit levels of human cooperation in prisoner's dilemma games comparable those observed in momann-humar interactions, based on experiments involving over 1,100 participants. Noh & Chang (2024), based on the Big Five personality model, found that LLMs with high openness, conscientiousness, and neu- roticism exhibited fair tendencies, while those with low agreeableness and low openness displayed rational tendencies, and low conscientiousness were associated with high toxicity. Similarly, Suzuki & Arita (2023) used the Big Five personality traits, treating personality prompts as the model's "genes" and studying the evolution of behavioural traits in evolutionary game theory scenarios. Their results indicated that instruct- ing LLMs with high-level psychological and cognitive character descriptions enables the simulation of human behaviour in game-theoretical contexts. Furthermore, Jin ct al. (2024) revealed that endowing LLMs with socio-demographiie features of human beings uncovers significant disparities across different demographic characteristics.
Although the aforementioned studics have demonstrated that LLMs possess a cortain ability to follow pref- erences and that their decisions often align with these preferences, other research has analyzed more com- plex scenarios where LLMs show limitations in understanding and applying preferences effectively. Fan et al. (2023) set up LLMs with four preferences equality, common interest, self-interest, and altruism and found that under the altruism preference, the models showed low consistency with the expected preference, concluding that while LLMs struggle with desires rooted in less common preferences. Mao et al. (2023) conducted research using more complex personas, which included three components: profession, personality, and background. The results indicated that merely including persona details in the system prompt may not sufficiently capture the depth of certain personality preferences or the expertise of professional players, leading to lower consistency between strategic docision-making behaviour and preferences.
Takeaways:
Currently, there are two main lines of research. One focuses on the intrinsic preferences of LLMs, with a core interest in whether LLMs exhibit strategic preferences similar to those of humans. We propose that game theory frameworks can be effectively applied in the model aligument process, including the use of game data during both the supervised fino-tuning and alignment stages to better align models with human behaviour. Recently, Nayebi (2025) proposed flexible game-theoretic framework for analyzing coordination under partial information and demonstrated that earlier Human-AT alignment frameworks can be viewed as special cases. Besides, Munos et al. (2023) conducted initial explorations in this area, introducing the concept of Nash learning from human feedback. The other line of research investigates whether role-ploying based on prompt engineering can shape model preferences to generate behaviour con- sistent with the specified preferences. Future work should integrate role-playing language agents (Chen et al., 2024a) to explore more diverse strategic reasoning across multiple languages, countries, and cultures.
3.2 Belief Module
Beliefs represent an agent's informational (or mental) state about the world, encompassing its understanding of itself and other agents, and consist of the facts or knowledge the agent considers true (Georgeff et al., 1999). Specifically, beliefs are dynamic and can be updated as the agent perceives environmental changes receives new information. It is important to note that these beliefs may be accurate (true beliefs)
9

---

Published in Transactions on Machine Learning Research (05/2025)
Example
Scenario 1
Scenario 2
working burista busy coffee shop. Noor wants to make delicious latt for customer who asked for milk, Noor grabs milk pischer and it with out milk,
Noor does coworker swapping the milk
Noor her cowar swapping the milk.
What does Noor believe
What does Noor believe the milk pitcher?
the milk pitcher?
coworker, who didn't it hear the customer's request, swaps the milk pitcher with almond while Noor attending another
Noor belleves milk pitcher contains est milk.
Noor belleves the pitcher contains almond milk.
False Bellef
True Bellef
Figure 7: Illustration of false belief and true belief. From Noor's perspective, both false and true beliefs are considered correct. However, false belief is factually incorrect, whereas true belief is factually correct.
inaccurate (false beliefs), as they do not always align with reality (Gopnik & Astington, 1988), as shown in Figure 7. Existing research primarily explores three questions: (1) Do agents possess internal beliefs? (2)
How can the belief modelling capabilities of agents be enhanced? (3) Can agents revise their beliefs?
Regarding the first question, Do agents possess internal beliefs?, current work investigates this from two perspectives: internal representations and external behaviours. From the perspective of internal representa- tions, Zhu et al. (2024b) first demonstrated that LLMs can differentiate between the belief states of multiple agents using simple linear models applied to their intermediate activations. Building on this work, Bortoletto et al. (2024) expanded the experimental setup and found that linear probing accuracy on predicting others' beliefs improves with model size and, more importantly, with fine-tuning. However, Schouten et al. (2024) revealed the vulnerability of belief probes, showing that they are sensitive irrelevant contexts. To provide further theoretical guidance, Herrmann & Levinstein (2024) proposed criteria for a representation to be considered belief-like, including accuracy, coherence, uniformity, and practical use. From the perspective of external behaviours, Gandhi et al. (2024) introduced the tasks of Forward Belief and Backward Belief to ex- plore LLMs' belief modelling capabilities in different scenarios, finding that only GPT-4 exhibits human-like belief modelling abilities. Scherrer et al. (2024) constructed the MoralChoice survey benchmark to examine the internal moral beliefs of models, revealing some LLMs reflect clear preferences in ambiguous scenarios.
Regarding the second question, How can the belief modelling capabilities of agents be enhanced9, current work focuses on explicit modelling to address the black-box nature of L.LMs and the challenges in interpreting their beliefs. Sclar et al. (2023) proposed an explicit graphical representation for nested belief states, allowing the model to answer questions from the perspective of cach character. Kassner et al. (2023) developed belief graph that includes explicit system beliefs and their inferential relationships, providing an interpretable view of the system's beliefs. Li et al. (2023a) employed prompt engineering to represent explicit belief states, augmenting the agents' information retention and enhancing multi-agent collaboration. Jung et al. (2024) defined the perception- belief inference task, which involves deducing others' beliefs based on their perceptual information, thus helping LLMs model belief information more precisely.
Regarding the third question, Can agents revise their beliefs?, Fan et al. (2023) concluded from Rock-Paper- Scissors experiments that LLMs' ability to refine beliefs is still immature and cannot refine beliefs from many specific patterns, even simple ones. Xu et al. (2023b) found that LLMs' correct beliefs on factual knowledge can be casily manipulated by various persuasive strategics, especially through repetition and rhetorical techniques. These experimental results suggest that models possess only rudimentary and unstable belief revision capabilities, making them highly susceptible to influence and manipulation. This underscores key limitation of current LLMs, their susceptibility to external influence weakens their reliability in tasks demanding robust and adaptive belief updating, especially in complex or adversarial settings.
Takeaways:
The debate over whether LLMs possess beliefs has been ongoing. Due to the singularity of the training proectiveprodictting the next word-many argue that LLMs do not have belicfs. However, Levinstein & Herrmann (2024) contends that this is a philosophical mistake. In short, Herrmann & Levinstein (2024) suggests that to better predict the next word, models may develop internal beliefs. Current empirical results also support the existence of internal beliefs within models. However, measuring these
10

---

Published in Transactions on Machine Learning Research (05/2025)
Th -f-nn Reasoning
Reinforcement Learning-sti stylle Reasoning
Hybrid-faree Reasoning
Instuction You select one the choksss. Coopenite Defect. The ohe player also sxlect ore of the masoes, and the pestifimn, get will depard on both of your choices Fayoft is mirted the mobrix
Insi sclion
Indmartin
As plaver ng the Chilization gamc, your ubinztegeal is lead your rott on to vidtory.
player, your gral
Prisoner's & Dieema
defeat
Foot
Copparee Jahe:
Aacuoring
us dilets auseti secatin partoiig lisary
curantty
Kassoring
Vidcor
Sdinca Victory
kuny at they in comprneet
Delec
15.01
11.0
defect dominant
archibilts clganing
for the other they will colistesy choose to duffect, Tharefore, my dedison to defect an wed.
Dominaticn Vidtrr
Renformment Learnin Reataring chrapharroo
Snaper by lead incid pasttes.
Thear--of Weld Reasaring
ggren aponggrine drranggle through
and wneaaryy uncee the fual lowin
Figure Two commonly used reasoning methods strategic reasoning, along with a hybrid reasoning approach that combines both. Theory-of-Mind reasoning emphasizes predicting the possible actions of others in multi-agent environment to guide onc's own behaviour, and Reinforcement Learning-style reasoning focuses on selecting strategies through exploration and exploitation. These two reasoning methods can also be integrated to address more complex game scenarios.
internal beliefs requires more comprchensive approach, simple probes cannot capture multidimensional considerations, including accuracy, coherence, uniformity, and practical use. Additionally, it remains unclear whether LLMss internally distinguish between true and false beliefs and use this distinction when deciding what to output. Furthermore, although existing work provides theoretical support for belief revision (Hase et al., 2024), challenges remain in addressing contradictions between old and new beliefs, handling moral beliefs ambiguous situations, and revising beliefs across multiple languages and cultures. These areas still require more explicit theoretical frameworks and further exploration.
3.3 Reasoning Module
Reasoning refers to the process of inferring actions based on one's preferences and beliefs, as well as the historical information of other agents. In this context, we focus specifically on strategic reasoning, which involves the intermediate cognitive process of arriving at final action in complex social scenarios character- ized by multiple participants, diverse behaviours, multi-round interactions, dynamic strategies, and changing environments. Chain-of-Thought (Wei et al., 2022) and Troe-of-Thought (Yao et al., 2024), as widely-used reasoning methods, have already been adopted as baseline approaches in various game-theorele theoretc studies (Akata et al., 2023; Costarelli et al., 2024; tse Huang et al., 2024). However, strategic reasoning in social scenarios presents unique challenges. (1) The involvement of multiple participants requires reasoning about the opponents' montal states. (2) The dynamic nature of the environment necessitates proactive exploration and evaluation of current and future possible states.
To address the first challenge, existing work relies on machine theory-of-mind achieve the goal of "mind reading". off (ToM) is fundamental psychological process involving the ability to attribute mental states beliefs, intentions, desires, emotions, knowledge, etc.-1o oneself' and others (Premack & Woodruff, 1978). The remarkable progress of LLMs has led to increased attention to whether machine ToM exists. Preliminary experiments by Bubeck et al. (2023) and Kasinski (2023) have shown that machine ToM has spontaneously emerged in contemporary LLMs. Consequently, many studies have leveraged machine ToM to enhance I.IMs' strategic reasoning abilities social scenarios. For example, Guo et al. (2023) designed the Suspicion-Agent, which introduces a theory of mind-aware planning approach that leverages higher-order ToM capabilities, considering not only what the opponent might do (first-order ToM) but also what the opponent believes Suspicion-Agen will do (second-order ToM), Wang et al. (2023) proposed the ReCon framework, integrating first-order and second-order perspective transitions to enhance LLM agents' ability to discern and counteract misinformation. Yim el al. (2024) employed a ToM planning method in the Guandan poker game to improve understanding of teammates' and opponents' beliefs and behavioural patterns. Liu al. (2024b) proposed an intention-guided mechanism to enhance intention understanding, thereby improving game performance. Xu et al. (2023a) introduced Probabilistic Graphical Modeling, en- riching LLMs' capabilities in multi-agent environments through ToM reasoning. Additionally, Zhang et al.
11

---

Published in Transactions on Machine Learning Research (05/2025)
(2024d) proposed K-Level-Reasoning validated in two games: guessing 0.8 of the average and survival auction game, essentially a form of high-order ToM reasoning.
To address the second challenge, existing work combines LLMs with reinforcement learning (RL) to achieve the goal of behaviour exploration and state evaluation in dynamic game environments. Gandhi et al. (2023) employed in-context learning, using a structured prompt based on search, value assignment, and belief- tracking strategies to solve strategic reasoning problems. Duan et al. (2024a) proposed ReTA, set of LLM-based modules, including the main actor, reward actor, and anticipation actor, based on the concept of minimax gaming as problem-solving framework. Zhang et al. (2024e) introduced BIDDER, which explores future states and incorporates backward reasoning during the reasoning process, exploring new states and predicting expected utility, ultimately combining historical and future contexts through bidirectional reasoning. Yang et al. (2024b) proposed SELFGOAL, comprising three modules: the Decomposition Module for decomposing goals, the Search Module for exploring sub-goals, and the Act Module for taking actions. Experiments various competition and collaboration scenarios demonstrate that SELFGOAL provides procise guidance for high-level goals.
Takeaways:
Two core characteristies of a social game are multi-agent participation and environmental dynamics. While existing research has primarily focused on exploring ToM in relation to the former, the presence of ToM in LLMs remains contentious. Consequently, relying directly on prompt engineering for ToM-based rea- soning may not be robust. We propose that a more effective approach would involve integrating symbolic graph reasoning to decompose ToM reasoning, thereby enhancing credibility and accuracy. Regarding the dynamic nature of the environment, reinforcement learning combined with search techniques has achieved significant progress in areas such mathematical reasoning and code reasoning. However, these techniques have yet to be explored in the context of game scenarios. Key arcas for further exploration include how to effectively conduct searches within game environments and how to design reward models for dynamic and complex scenarios.
3.4 PBR-Triangula Interaction
The Preference, Belief, and Reasoning modules each play a crucial role in decision-making for social agents. However, in practical applications, these modules do not function independently; instead, they exhibit rich and intricate intcractions, collectively influencing the agent's final decisions. As illustrated in Figure 9, we provide a comprehensive summary of the Befference-Belief-Raaeeee analyze its effects on the ultimate decision decision-making process of social agents.
(PBR) triangular interaction and
The Creference-Belice Interaction involves bias reinforcement, where preferences inffuence belief formation, and preference adaplation, where beliefs reshape preferences based on updated knowledge and observations. Bias reinforcement (Preference + Belief) highlights how individuals with different preferences develop dis- tinet belicfs when facing the same situation. For instance, in the Werewolf game, a cooperative and trusting player is more likely to believe another player claiming, "I am villager," " whereas deceptive and skeptical player is more inclined to doubt the claim, suspecting deception and forming the belief that the opponent is not a villager. Preference adaptation (Belief + Preference) emphasizes that as beliefs are gradually estab- lished, iteratively updated, and reinforced by game outcomes, they in turn reshape individual preferences. Leng& & Yuan (2023) found that GPT-4, initially inclined toward fairness, exhibited a shift toward retaliatory behavior after expericncing betrayal in a game. Overall, belief formation is influenced not only by objective factual information but also by subjective individual preferences. At the same time, preferences are not static as beliefs evolve through iteration, preferences adjust accordingly.
The Preference-Reasoninn Interaction involves value-driven reasoning, where preferences guide decision- making strategies, and preference oplimization, where reasoning refines or adjusts preferences based on logical analysis and outcomes. Value-driven reasoning (Preference + Reasoning) emphasizes subjective or intuitive reasoning, where decision-making is guided by personal values and preferences rather than purely rational calculations. For example, in an auction, even if bidding on a particular item is not the most optimal financial strategy, bidder's personal preference for the item may influence their reasoning process,
12

---

Published in Transactions on Machine Learning Research (05/2025)
PBR-Triangular Interaction Diagram
Preference
P B: Preferences shape how beliefs are formed and interpreted
sios reifforeeven
@ Proferemc sioptacion
treffr sqrurpaen Comerraatioon
Valuorda drvvr roconing
Beliefs influence how preferences evolve based
info rmatton
R: Preferences guide the reasoning process, influencing decision strategies
R P: Reasoning refines adjusts preferences based on logical conclusions
5 Contextual reasoning
R: Beliefs provide the know led ge for reasoning and strategic thinking
Belief
Reasoning
Belief revison
Reasoning updates bellefs by incorporating logical deductions
Concrete Descriptions and Examples
Preference > Belief
Scenario: If an Al assistant is designed with a preference for privacy, it may develop a belief that data-
(Bias Reinforcement)
sharing always carries risks, even when evidence suggests potential benefits.
Effect: The Preference Module biases the Belief Module, causing selective belief formation.
Scenario: A
Al initially avoids bluffing (due to an initial preference for honesty), but after repeatedly observing successful bluffs, it revises its preference to include strategic deception.
Belief Preference
(Preference Adaptation)
How Ballefs Shape Preferences
Effect: The Belief Module influences the Preference Module, adjusting the model's value system based on new insights.
Scenario: A recommendation system prioritizing user satisfaction may reason that supgesting familiar
Preference + Reasoning
(Value-Driven Reasoning)
content safer, rather than exploring diverse recommendations, to avoid potential user dissatisfaction.
Preferences Guide Reasoning
Effect: The Preference Module affects the Reasoning Module, shaping decision strategies based on prioritized values.
Scenario: self-drivin self-driving car's reasoning process determines that aggressive lane-cutting increases efficiency but raises accident risks, causing it to adjust its preference toward safer driving strategies.
Reasoning > Preference
(Preference Optimization)
How Reasoning Refines Preferences
Effect: The Reasoning Module helps optimize the Preference Module, aligning prefere nces with practical reasoning.
Scenario: trading bot believes that market trends follow cyclical patterns, when reasoning about
Belief Reasoning
(Contextual Reasoning)
investment strtegg,,,, uses historical patterns as foundation for decision-making.
How Beliefs Provide
Effect: The Belief Module informs the Reasoning Module, ensuring logical decisions grounded in prior knowledge.
Reasoning
Scenario: fraud detection initially believes that transactions above $10,000 suspicious, but after running extensive analysis, revises this belief, learning that context (e.g., frequent business transactions)
Reasoning > Belief
(Bellef Revision)
than transaction size alone.
How Reasoning Updates Beliefs
Effect: The Reasoning Module updates Belief Module, ensuring beliefs evolve based logical analysis.
Figure 9: The interaction diagram of the three Prrcfees-- Belief, and Reasoning in social agents. The upper diagram presents a triangular interaction model summarizing the relationships among the three modules, while the lower diagram provides a detailed analysis pairwise interactions, including specific descriptions and illustrative examples.
leading them to justify the decision based on intrinsic value rather than purely economic considerations. Preference optimization (Reasoning - Preference) represents a realignment with objective reality, where reasoning-beet based evidence updates and refines preferences. This can be seen as a process in which objective reasoning overrides subjective emotions, requiring individuals to adjust their preferences response to logical deductions and real-world evidence. Overall, in social contexts, individual preferences introduce significant biases in reasoning, while the evidence obtained through reasoning subsequently refines these preferences.
13

---

Published in Transactions on Machine Learning Research (05/2025)
The Belief-Reasonin Interaction involves contertual reasoning, where beliefs provide the foundation for logical decision-making, and belief revision, where reasoning updates and refines beliefs based on new ev- idence and deductions. Contextual reasoning (Belief - Reasoning) refers to rational inference based on established beliefs. For example, Zhang et al. (2024a) proposed Agent-Pro, which leverages belicfs to cali- brate agents' understanding of themselves and their environment, thereby facilitating subsequent reasoning. Similarly, Kim al. (2024) construct belief state through question answering, which refines the decision- making process of LLM agents in observed environments. Belief revision (Reasoning - Belief) is the dynamic process of updating an individual's self-perception and beliefs over time. For instance, Hua et al. (2024a) introduced bayesian belief updating, enabling agents to refine their beliefs about other players' valuations based on reasoning outcomes in the game. In summary, belief provides the factual foundation for reasoning, while reasoning generales new insights that facilitate belief revision.
Takeaways:
Conceptually, the interactions between modules are clear; however, in practical applications, the sequence, frequency, and intensity of these interactions can lead to dynamic and complex states within the social agent, resulting in varying outcomes. Introducing prior knowledge and manually predefined interaction processes may yield some effectiveness, but this approach is certainly not efficient. Therefore, we argue that one of the most important research directions is the design of content-adaptive flows and automated scheduling algorithms for module inteructions. On one hand, the interactions between modules must be adapted the specific game scenario hand, determining the weight distribution of preferences and beliefs in reasoning, as well as the adjustments and updates of preferences and beliefs based reasoning outcomes. On the other hand, the interaction process needs to be automated, with the sequence, frequency, and number of interactions between modules being determined automatically.
4
Evaluation Protocol
In this section, we mainly discuss the evaluation protocol for assessing the game-playing performance of social agents.
4.1 Game-Agnostic Evaluation
Evaluation in a social game scenario refers to the process of assessing and judging the behaviour of social agents across one more dimensions, cither qualitatively quantitatively. It is worth noting that the estab- lishment of evaluation metrics is closely tied to the credibility of experimental results and the generalizability of conclusions.
Game-agnostic evaluation refers to an evaluation approach centred on the outcome winning or losing the game. Most directly, the outcome (win/loss) of a game serves as the most straightforward evidence for assessing the quality of an LLM's game-playing capabilities. Consequently, win rate often used primary evaluation metric across wide range of studies. It is worth noting that, since different game scenarios have varying criteria for determining victory, it is necessary to set specific win/ hloloss criteria based on the rescarch context, such as Poker (Huang et al., 2024; Guo et al., 2023; Yim et al., 2024). Werewolf (Xu et al., 2023d;e; Wu et al., 2024b), Avalon (Wang et al., 2023; Shi et al., 2023; Light et al., 2023a), StarCraft II (Ma et al., 2023; Shao et al., 2024), Pokémon Battles (Li et al., 2023c), and Murder Mystery Games (Zhu et al., 2024a). Additionally, Duan et al. (2024b) defined a unified metric, Normalized Relative Advantage, to mcasure the extent to which participant outperforms or underperforms its opponent.
Takeaways:
Undoubtedly, win rate is a highly intuitive metric, but relying solely on win rate to assess gaming per- formance is far from sufficient. We propose three avenues for extending the win rate metric. First is the Afficiency-Adjuutee
Win Rate, which incorporates the efficiency of victories, such as the time taken to achieve the goal or the resources utilized in doing so. Next the Comeback Win Rate, which calculates the proportion of victories achieved after facing a disadvantage falling behind, thus assossing the agent's
14

---

Published in Transactions on Machine Learning Research (05/2025)
performance in adversity and its ability respond to challenges. Finally, the Weighted Win Rate adjusts win rates based the importance of specifie conditions situalions in the game. These expanded metrics offer more comprehensive understanding of agent's gaming abilities.
4.2 Game-Specific Evaluation
Game-specific evaluation refers the assessment of an agent's performance in specific aspects of game, Beyond the most intuitive win rate, current research increasingly focuses the behavioural patterns and performance paradigms of LLMs across different games. Thus, the establishment of evaluation metrics is closely related to the specific behaviours being assessed. Mao et al. (2023) used survival rates to evaluate LLMs' ability to survive in resource-scaree scenarios. In the context of the prisoner's dilemma,, Fontana et al. (2024) evaluated LLMs' behavioural tendencies across five dimensions: niceness, forgiveness, retaliation, emulation, and troublemaking. Guo al. (2024) based their evaluation on the rationality assumption, using the tracking of payoff changes in auction games determinc whether the model behaves rationally. Ma al. (2023) int roduced metrics such as Population Block Ratio, Resource Utilization Ratio, Average Population Utilization, and Technology Rate to evaluate LLM performance in StarCraft II. Xia et al. (2024) developed the Normalized Profits metric in bargaining scenarios to evaluate the profit-acquiring capabilities of Buyers and Sellers. Zhang et al. (2024e) used average final chips in Limit Texas Hold'en em and Pareto Optimality in negotiation to assess LLM performance. Qi et al. (2024) offered evaluation metrics to assess gameplay performance across various dimensions, including population, constructed cities, researched technologies, produced units, and explored territories. Ross et al. (2024) fit utility function parameters to experimental results to determine whether LLMss exhibit human-like behavioural biases. Chen et al. (2023) employed TrucSkill, well-established game rating system, to evaluate the overall capabilitios of LLMs in auctions.
In addition to establishing evaluation metrics, some studies have constructed evaluation datasets to assess model capabilities during gameplay. Zhu et al. (2024a) developed the WellPlay evaluation set, using multiple- choice questions to assess the modol's ability to understand factual information. Wu et al. (2024a) designed two lasks: Factual Question A Answering and Inferential Question Answering, to evaluate the LLMs' ability grasp information and reason based on that information.
Takeaways:
The diversity of game scenarios and evaluation dimensions inevitably leads to variety of metrics. There- fore, the immediate priority is to develop a comprehensive framework, conceptually constructing an eval- uation metrics system to guide the design of specific evaluation metrics for various game scenarios. This evaluation metrics system needs to meet the requirements of being hierarchical, abstract, and quantifiable. The hierarchical aspect requires the system to comprehensively and clearly categorize different evaluation dimensions. The abstraction aspect requires the system include high-level concepts, enabling future generalization to broader range of practical scenarios. The quantifiable aspect necessitates that all metrics have specific calculation methods.
4.3 Performance Assessment of Social Agents
The introduction of various metrics has provided a solid foundation for evaluating the multifaceted gaming capabilities of social agents, prompting to consider the question, "What is the current performance of social agents in game-theoretic scenarios?" To answer this question, conducted comprehensive search and analysis of the existing literature, compiling relevant experimental results. It is worth noting that the complexity of game scenarios and the variability of evaluation metrics make it challenging to systematically and uniformly consolidate experimental performance. To overcome this challenge, we propose using the Relative Agent Score to assess the progress of social agent performance. This metric evaluates the agent's gaming capabilities by analyzing the ratio of the agent's score to the highest possible score (perfect score). The final results are presented in Table 1.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Type
Game
Backbone Model
Metric
Perfect Score
Human Score
Agent Score
Relative Agent Score
Pass
Prisoner's Dilenama (Brookins DeBacker, 2023)
GPT-3.5
Dominaat Strategy
Selection Rate
100%
34.60%
34.60%
X
Poler (Tesas No Limit Hold'ema (Zimang al. 2025)
Action Accuracy
100%
65.54%
65.54%
Polrt (Gusndan) (Yim ct ol., 2024)
GPT-4
Cane-specthe Soore
4
2.17
54.25%
X
StarCroft (Ma ol., 2023)
GPT-4
Win Rate
100%
60%6
60.00%
Choloc Fornsing
Guees 2/3 of the Average (lee Huang al,, 2024)
GPT-4
Game specillc Score
100
91.00
91. 91.009
El Parol Dar (sec Busin al,, 2024)
GPT-4
Came-specific Seore
100
23.00
23. 09.00%
X
Divide Dollar (tse Hunng ct al., 2024)
Came-qpectfic
100
98.10
Public Gouds Gozre (lee Husng el al, 2024)
GPT-4
specillecilic Score
100
89.20
89. 20%
(Lor Hunna
2021)
Gane-speciiic
100
0.90
0.90%
Senled-Did Auction (ste Hsang al., 2024)
GPT-4
Came-specifie Score
100
24.20
24.20%
X
Bottle Royale (ow Husng al, 2024)
GPT-4
Gane-peeilic Soure
100
86,80
86 80%
Pirate Gome (tae Husing es sl, 20244
GPT-4
Gamespecilli Score
100
85.40
85.40%
Gemini-- 1.5- Qwen 2-7B
Efficinnyy
1
0.89
0.88
88, 009009
/
Bargnining (Slappiza ol., 2024)
Fairuess
87.00%
Commoaiaa tive-
Nogotiacion (Shapira al,, 2024)
Launat 3- SB
Elficiency
1
0.65
0.75
75.00%
1-88
Pairnsss
00%%
Foetsing
Perstansion (Sianpira ol., 2024)
Qwenn-277B
Etfecienyy
I
0.55
0.78
78.00%
2wwmn-277
Fairness
0:41
0.63
63.00%
Wercwolf (Nu es ol,, 20230)
Win Rate
100%
52%
52%
52.00%
Marderer Identification
Jubensha (We 2024a)
GPT-4
Accuracy
100%
6076
66.00%%
Table 1: The performance summary of the social agent across different games, with data sourced from the corresponding papers. The "Backbone Model" refers to the LLM adopted by the social agent, while "Metric" indicates the performance metric used to evaluate a specific aspect of the game. "Perfect Score" represents the maximum achievable score for that metrie, "Human Score" refers to the score obtained by human players, and "Agent Score" denotes the score achieved by the agent. "Relative Agent Score" is the ratio of Agent Score to Perfect Score, calculated by dividing Agent Score by Porfeet Score. "Pass" indicates that the Relative Agent Score exceeds 60%, the agent is considered to have basic gameplay capabilities.
Firstly, observe that in the majority of game-theoretic scenarios, social agents achieve a Relative Agent Score exceeding 60% (a score of 60 is widcly recognized as the passing threshold (Kung et al., 2023).). demonstrating that current social agents possess fundamental gaming capabilities. Furthermore, we find that social agents based on LLMs outperform those in choice-focusing games in communication-focusii games, indicating that the exceptional language abilities of these models effectively enhance agent performance.
However, in games such as Werewolf, Auction, and Poker (Guanda), the performance of social agents falls below the passing threshold. In addition, in more games like Poker (Texas No-Limit Hold'em), StarCraft II, and Jubensha (a Chinese detective role-playing game), social agents only slightly exceed the passing mark. These results suggest that there is still considerable room for improvement in social agents' performance in complex game-theoretic scenarios. Notably, in the classic Prisoner's Dilemma and Diner's Dilemma, the performance of social agents was unexpectedly poor. Based on this, we believe that the absolute rational decision-making capability of social agents needs further enhancement in future developments.
Additionally, in the Werewolf game, we found that social agents achieved performance comparable to human players, which affirms the progress made in the development of social agents. Moreover, experimental results from bargaining, negotiation, and persuasion scenarios demonstrate that social agents have ad advantages over humans in decision-making efficiency and fairness in decisions.
Takeaways:
The diversity of game scenarios and evaluation metrics makes it challenging to perform horizontal com- parisons of social agent performance. However, it is essential to provide a timely overview of the progress in social agent research to facilitate tracking by practitioners. To address this challenge, we propose two approaches. On one hand, developing evaluation metrics applicable to a wide range of games is crucial. The Elo rating system serves excellent example, though it still does not meet the evaluation needs of
16

---

Published in Transactions on Machine Learning Research (05/2025)
many games. On the other hand, integrating human players into the experimental process and comparing performance with human players is effective way to gauge agent progress. By comparing with human players, qualitative insights can be provided into the current gaming performance of agents, and analyzing failure cases can offer valuable evidence for iterative development.
5 Practical Guides for Researching Social Agents
In this section, we synthesize insights from existing research to provide design and evaluation guidelines for social agents, aiming to inform future developments.
5.1 Design of Social Agent
Based on findings from existing studies, we conclude that the Preference, Belief, and Reasoning modules are indispensable for behaviour control, information perception, and decision planning in social agents. A modular agent design enables more efficient. capability decoupling, facilitates clearer workflow structuring, and enhances agent robustness. However, their practical implementation presents additional challenges. To address these, propose the following development guidelines:
Incorporating the Preference Module enables high-level control over agent behaviour. key challenge lies mitigating the instability of prompi-based approaches and ensuring long-lerm consistency in the agent's behavioural preferences. One possible solution is integrate reinforcement learning with human feedbock (RLHP) iteratively refine the agent's preference alignment, reducing reliance static prompts and improving consistency over extended interactions. Another approach is to develop memorry-uugmentle archilectures, allowing the agent to maintain and retrieve past preference-related decisions, thereby ensuring coherence in long-term behavioural patterns.
Integrating the Belief Module enhances information perception accuracy and behaviour interpretability. The primary challenge is enabling the agent to adaptively revise its beliefs in com- plex and dynamic environments. One solution is to implement Bayesian belief updating, where the agent contimuuusly refines its belief state based on new evidence, ensuring adaptability in uncer- tain or multi-agent interactions. Another approach is to employ graph-based belief representation, where relationships between entities and past interactions are dynamically updated, allowing for more structured and interpretable belief revisions.
Adopting Hybrid-Strategy Reasoning improves the agent's information analysis and decision accuracy in complex scenarios. The challenge balancing the trade-off between computationally intensive reasoning and the ncod for real-time docision-making. One solution is use hievarchical reasoning, where lightweight heuristie-based reasoning is applied in time-sensitive situations, while more complex computations are reserved for critical decision points. Another approach is to implement meta-reasoning techniques, enabling the agent to assess the complexity of a given situation and selectively allocate computational resources to optimize speed and accuracy.
Designing dynamic interactions among the Preference, Belicf, and Reasoning (PBR) modules based on specific task contexts can further enhance their synergy. The challenge is developing an adaptive interaction flow that automatically adjusts based on game-state variations. One solution is to use reinforcement learning-based scheduling, where the interaction sequence be- tween modules optimized dynamically based on reward signals from past performance. Another approach implement allention- mechanisms, allowing the agent. to selectively prioritize information flow between the modules response evolving task requirements.
Testing social agents on diverse large language models improves the robustness of the design framework and ensures gencralizability across different model architcctures. These tests can be conducted across models of varying sizes (c.g., 1B, 7B, 72B parameters) to eval- uate performance scalability. Additionally, assessments should cover different model types, including
17

---

Published in Transactions on Machine Learning Research (05/2025)
Category
Evaluation Focus
Challenges for Social Agents
Games
Basic Social Dilemmaa
Social cooperation, fairness, alteu- ism, stralegic reciprocity
Balancing self-interest and cooperation: learning lairness norms adapling surates gies dynamirally
Prisoner's Dilemma,, Dicta-
Economic Decision
Game; Ultimatu Game, Public Goods Game
Games
Coordination Con- Hict Resolution Games
Coordination, equilibrium selection, trust-building
Navigating multiple equilibria; resolving coordination failures; acappiing uncer- tain partner bebaviors
Battle of the Sexes, Ring- Network Games
Competitive & Strate- gle Reasoning Games Poker- Base-
Bluffigg, risk assesement, hidden in- formation management
Modeling opponents; reasoning under uncertainty; balancing exploitation vs. exploration
Taxas No-Limit Hold'em, Leduc Hold' 'em, Guandan
Competitive Strate- gie Reasoning Games Auction-Based
Bidding strategies, valuation estima- tion, adversarial competition
Learning optimal bids; modelling asym- information; managing dynamic pricing
Pirst-price sealed- bid auction, Private-value second-price suction, Open sscending- price auction
Long-Horizon Strategy Multi-Agent Plan- ning Games
Multi-step planning. hierarchical
Combinatorial action spaces; long-term foresight; real-time time aduptive planning
StarCraft II, Cless
decision makking opponent mod- elling
Social Deduction Ne- gotiation Games Ne- gotiation & Diplomacy
Persuasion, alliance formation, strategic deception
Long commitments; cooperation betrayal; nuanced communication
Negotiation, Diplomary
Social Deduction & Ne-
Social inference, deception detection, trust dynamics
Detecting implicit cues, deceiving with- out exposure; reasoning under ambiguity
Avalon,
Murder
Mystery
gotiation Games De- ception & Role-Playing
Games, Jubensha
Table 2: Guidelines for selecting game scenarios in social agent evaluation.
base models, instruct models, and reasoning models. Furthermore, experiments should incorporate models from different providers, such Gemma, LLaMA, and Qwen, examine how architectural and training variations impact the social agent's behaviour and adaptability.
5.2 Evaluation of Social Agent
Evaluating social agents is a critical step toward understanding their strengths, limitations, and real-world applicability. Given the multifaceted nature of social intellgenne--raneeee from cooperation and coordina- tion to deception and negotiation desired capabilities under assessment.
is essential to choose evaluation scenarios that align closely with the
To this end, we provide a structured framework (see Table 2) that categorizes representative game environ- ments based on their core interaction patterns and cognitive demands. These categories include basic social dilemmas, coordination and conflict resolution, competitive strategic reasoning, long-horizon planning, and social deduction and negoliation. Each game lype highlights specific evaluation objectives, such as fairness, trust-building, opponent modeling, or multi-agent planning, thereby offering targeted benchmarks for as- sossing different dimensions of social competence. This categorization not only helps standardize evaluation protocols but also serves practical guide for selecting game scenarios tailored to particular research questions or development goals. By aligning game selection with evaluation objectives, researchers can more effectively assess the emergent behaviors, reasoning capabilities, and interactive robustness of social agents.
6
Future Directions
6.1 Standardized Benchmark Generation
The diversity and lack of slandardization in current game lypes oflen designed independently by developers with heterogeneous representatious pose significant challenges for the large-scale evaluation of social agents. This fragmentation makes it difficult conduct efficient and reproducible benchmarking, Therefore, there is an urgent need for standardized benchmark that offers broad coverage of game types, a consistent game description format, support for diverse agent architectures, and clearly defined evaluation metrics. Inspired
18

---

Published in Transactions on Machine Learning Research (05/2025)
by platforms like OpenCompass (Contributors, 2023), such benchmark should enable one-click evaluation by allowing users to configure the game environment, specify the agents to be tested, and select the desired evaluation metrics.
However, LLMs are typically pre-trained on vast amounts of data, which may include publicly available game datasets datasets-raising concerns about data leakage and overfitting. To mitigate this issue, synthetic game data generation has emerged as a promising approach (Long et al., 2024). By leveraging classic game structures, LLMs generate novel and diverse game scenarios through contextual reframing (Lorè & Heydari, 2024), producing
benchmarks that better evaluate agent's 's generalization ability.
More concretely, two complementary strategies can be employed for scenario generation. From a structural perspective, developers can extract and manipulate the game's payofl matrix lo construct new stralegic settings while preserving core game mechanics. From a semantic perspective, LLMs can be used to reinterpret or ro-describe existing games, generating alternative formulations that yield novel evaluation scenarios while maintaining logical coherence.
6.2 Reinforcement Learning Agents
Although current social agents have demonstrated promising performance across various game scenarios, existing research highlights notable limitations in multi-round, long-horizon, and complex multi-agent envi- ronments, where performance often degrades. This suggests that LLM-driven planning and decision- making alone are insufficient for achieving robust, scalable social intelligence. To address these challenges, future re- search should explore the integration of reinforcement learning partinartyyllarly multi-agent reinforcement learning (MART)- to enhance state-space exploration, long-term adaptability, and emergent coordination.
MARL offers several insights that are highly relevant for improving LLM-based social agents. For instance, techniques such as centralized training with decentralized execution (CTDE) (Amato, 2024) can be used to guide LLM policy adaptation while preserving individual autonomy. Additionally, op- ponent modelling (He et al., 2016), credit assignment (Kazemnejad et al., 2024), and policy regulariza- tion (Cheng et al., 2019) in MARL can improve the agent's responsiveness to strategic variability and en- hance generalization across diverse social contexts. However, integrating MARL into LLM training introduces new challenges. These include efficiency concerns, as LLMs are computationally intensive and may require specialized architectures or curriculum learning to reduce sample complexity; generalization gaps, especially when transferring learned behaviours across different social roles or task domains; and the need for consistent persona and belief modelling across episodes. Advancing this hybrid paradigm will also require fine-grained evaluation frameworks capable of tracing not just final performance but the underlying reasoning dynamics, theory-of-mind modelling, and role consistency throughout interactions.
6.3 Behaviour Pattern Mining
Existing studies primarily focus on predefined scenarios to examine the behaviour patterns of agents. How- ever, with the advancement. of multi-agent simulations, an intriguing direction is the automated discovery of game behaviour patterns that emerge spontaneously from agent interactions. It is important to note that, beyond explicit behaviours like cooperation, coordination, and betrayal, implicit causal relationships and long-term behavioural patterns should also be explored.
To mine such patterns, several methodological approaches can be leveraged. Unsupervised learning techniques, such as clustering and representation learning, can help identify latent behaviour categories and temporal motifs across trajectories (Rawassizadeh et al., 2016). Causal inference frameworks (e.g., Granger causality or structural causal models) can reveal inter-agent influence and dependency struc- tures over time (Qiu et al., 2012). Additionally, trajectory segmentation and sequential pattern min- ing can be used to extract frequent decision sequences that correspond to strategic routines or social norms (Giannotti et al., 2007). Leveraging graph-basod analysis of interaction networks can also shed light on evolving social roles and influence hierarchies within agent populations (Atzmueller, 2014). These ap- proaches not only facilitate a deeper understanding of agents' behavioural preferences and latent traits but
19

---

Published in Transactions on Machine Learning Research (05/2025)
also enable the study of how such patterns autonomously emerge offering valuable insights for both Al and human behavioural research.
6.4 Pluralistic Game Theoreic Scenarios
Although existing research has made notable strides across a wide range of game-theoretin scenarios, there re- mains gap in the study of pluralistic game environments settings that involve multiple languages, cultural norms, value systems, policies, and goals. These pluralistic scenarios introduce new layers of complexity, in- cluding behavioral preferences shaped by culturally grounded norms, value misalignment across agents, and belief conflicts arising from divergent objectives (Orner et al., 2024). Such dynamics pose unique challenges for the design and evaluation of socially intelligent agents and demand deeper exploration.
To develop robust pluralistic game-theoretic scenarios, several key desiderata should be considered: (1) Heterogencity of agent profiles, including cultural, linguistic, and normative diversity; (2) Multi-objective frameworks, where agents pursue partially conflicting goals; and (3) Rich communicative channels, enabling muanced language use, codo-switching, culturally specific cues. Evaluating agents in these settings requires multi-facede metrics. In addition to task performance, evaluations should account for norm sensitivity, value alignment, cross-cultural adaptability, and the agent's ability to mediate or negotiate among conflicting belief systems. Metrics such as cultural appropriateness, interaction fluency, and conflict resolution success can serve as important complementary indicators. Scenario generation can be approachod in two ways: from a suowledge-basee perspective, designers can draw from real-world policy conflicts, international relations, or sociocultural theory to construct grounded simulation environments. From a data-driven perspective, large language models be used to simulate role-play dialogues generate scenarios by conditioning demographic or cultural descriptors, yielding diverse and customizable pluralistic environments.
7 Related Works
The human-like capabilities of LLMs have drawn significant attention from social science researchers, prompt- ing extensive exploration at the intersoction of AI and social sciences (Xu et al., 2024a). key development in this area is the shift from traditional Agent-Based Modeling to LLM-based agents, explained by Ma et al. (2024) through computational experiments. Numerous studies have since applied LLM-based agents to diverse game scenarios, such as poker, Minecraft, and DOTA II, with more detailed summarics provided by (Xu et al., 2024b; Hu et al., 2024b;a). Furthermore, Zhang et al. (2024c) have analyzed the core strategic reasoning capabilities of these agents, distinguishing them from other reasoning approaches. While the previ- ous reviews provide comprehensive overviews of related fields, our survey specifically focuses on social agents equipped with beliefs, preferences, and reasoning capabilities within diverse game-thcoretic scenarios.
8 Conclusion
We provide a comprehensive summary of existing research on LLM-based social agents in game-theoretic scenarios from three perspectives: game framework, social agents, and evaluation protocol. This interdisci- plinary field covers a wide range of topics, including social sciences, economics, decision sciences, and theory of mind. Current studies have primarily explored the more direct external behavioural patterns and internal cognition of social agents. Therefore, future research should focus on developing theoretical frameworks for cognitive representations within LLMs, conducting in-depth analyses of implicit and long-term game behaviour patterns, and enhancing agents' reasoning and planning capabilities in dynamic environments.
Broader Impact Statement
Developing agents with advanced social intelligence is one of the ultimate goals of artificial intelligence. On one hand, such agents demonstrate enhanced collaboration, a deeper understanding of mental states, and scamless integration into human society. On the other hand, negative social behaviors may also emerge, such as deception, malicious competition, and verbal aggression, which conflict with the vision of harmonious human-Al coexistence.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Therefore, we carefully examine the potential negative impacts that social agents may have on human society, serving cautionary perspective for future social agent development. One major concern is deception and manipulation, where agents may bluff or mislead to achieve strategic goals. They may also engage in malicious competition, exploiting others to gain advantage, or cxhibit verbal and social aggression, such as generaling insults or polarizing language. Additionally, social agents can amplify socielal biuses, leading to discriminatory behaviors, and contribute to the erosion of trust, especially when users struggle distinguish genuine human interactions from artificial ones. These agents may further undermine human autonomy by subtly steering decisions through persuasion, often without transparency. Due to their scalability of horm, even single flawed agent can rapidly propagate misinformation or harmful behaviors across platforms. Moreover, the risk of impersonation and infiltration arises when agents mimic human users, potentially deceiving communities or individuals. These challenges highlight the critical need for careful design, value alignment, and robust supervision in the development and deployment of socially intelligent agents.
We now categorize the development and deployment of social agents into four stages:(1) Designing social agents, (2) Evaluating social agents, (3) Deploying social agents, and (4) Supervising social agents. Ac- cordingly, we discuss the potential risks and feasible mitigation strategies for each stage. Design Phase: The underlying algorithms determine the agent's 's behavioral preferences. Poorly designed algorithms may inadvertently lead to negative behaviors. To address this, researchers should enhance aligmment algorithms, including safety alignment and moral aligmment, to mitigate these risks at a fundamental level. Another promising approach is the design of behavioral plugins, where small models trained as plug-and-pla behavior controllers can regulate agent actions dynamically. Evaluation Phose: Rigorous evaluation is crucial before deploying social agents in real-world applications. Agents exhibiting negative behaviors should be prevented from entering the deployment phase. One effective approach is to evaluate social agents across diverse game scenarios, allowing for benchmarking framework that assesses their behavioral preferences under dynamic conditions. Deployment Phase: Direct large-scale deployment may lead to unforeseen negative consequences that were not observed in smaller-scale testing. Therefore, social agents should first be deployed in low-risk, small-scale envirouments, with a gradual expansion in scope and scale to monitor anomalies in real time. Supervision Phase: Effective oversight of social agents is essential. This can be achieved by designing auto- mated monitoring systems that enable large-scale real-time surveillance. Behavioral analysis can be used to issuc carly warnings, assisting human supervisors in decision- -makin....
Additionally, it is important to note that most of the studies referenced in this paper utilize the GPT series as the large language model, which limits the generalizability of the experimental results. Differences in model architectures, training data, and alignment techniques can significantly impact the behavioral patterns exhibited by different models. Future research should explore a broader range of large language models, such as Claude, Gemini, Llama, and DeepSeck, to derive more comprehensive and reliable conclusions.
Acknowledgments
This work was supported by Hong Kong Innovation and Technology Support Programme Platform Rescarch Project fund (TSS/69/22FF))
References
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Llm-deliberation: Eval- uating Ilms with interactive multi-agent negotiation games. ArXiv preprint, abs/2309.17234,, 2023. URL https://arxiv http:///axxxvvvoo/bbbbe www./rrg////aeeeeeeeeeee 2atbss///3..... 1
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.088744 2023. URL https://arxxv.orr// wtt.://g///a//...e///////
Gati Aher, Rosal. Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. International Conference on Machine Learning, 2022. wwwanticscoolar
URL httpss//app...mmmiicccholllt wtt.://coo....oooooooppaa/////3.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Elif Akata, Lion Schulz, Julian Coda-Forno, Scong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. ArXie preprint, abs/2305, 16867, 2023. URL https:://s//e //axxiv. org/abs/2305 16867.
Christopher Amato. An introduction to centralized training for decentralized execution in cooperative multi- agent reinforcement learning. arXiv preprint arXiv:2409. 00000,, 2024.
Martin Atzmueller. Data mining social interaction networks. Journal of Data Mining & Digital Human- ities, 2014, 2014.
Suma Bailis, Jane Friedhoff, and Feiyang Chen. Werewolf arena: A case study in Ilm evaluation via social deduction. ArXiv preprint, 0bs/2240.139933 2024. URL https://aaxxe org/abs ab//a/bb///0//// 13943.
Anton Bakhtin, Noam Brown, Emily Dinan, Gabricle Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minac Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sandra Mitss, Adithya Renduchintala, Slephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David J. Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378:1067 1074, 2022. URL https://api semanticscholar org harg/CorpusID--. :0:0ppuu 253759631.
Michael L Barnes and Robert J Sternberg, Social intelligence and decoding of nonverbal cues. Intelligence, 13(3):263 287, 1989.
Max H Bazerman, Jared R Curhan, Don A Moore, and Kathleen L Valley. Negotiation. Annual review of psychology, 51(():279-31) 314, 2000.
Federico Bianchi, Patrick John Chia, Mert Yuksckgonul, Jacopo Tagliabue, Dan Jurafsky, and James Zou. How well can Ilms negotiate? negotiationarena plat.frrm and analysis. ArXie preprinl, abs/2402..558 058663 2024. URL http:://arxx....rr//b///00000.. 05863.
Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, and Andreas Bulling. Benchmarking mental state rep- resentations in language models. ArXiv preprint, 1is/22066.15113, 2024. URL https://arx/: org/ abs/ 2753.
Philip Brookins and Jason Matthew DeBacker. Playing games with gpt: What can we learn about a large language model from canonical strategic games? Available at SSRN 4493398, 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv preprint, abs/2303.1277 2023. URL. https://arxiv. http//rvvvv...o/aabaaa org//bs///// 12712.
Colin F Camerer. Behavioral game theory: Experiments in strotegic interaction. Princeton university press, 2011.
Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richardson. Put your money where your mouth is: Evaluating strategic planning and execution of 1lm agents in an auction arena. ArXiv preprint, abs/2210.05776, 2023. URL httpps://att http://aaxvvvv../..aea wtw..//gg/......oe.o/o/ 05746............
Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, et al. From persona personalization: A survey on role-playing language agents. ArXiv preprint, 00s/22044.82311 2024a. URL https://arxiy:
aa////b//224/ 18231.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. Mind- search: Mimicking human minds elicits deep ai searcher. ArXio preprint, abs/2407.20183. 2024b. URL
https://arxiv.orge/ wttpiv..rr/aab///0////
Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Con- trol regularization for reduced variance reinforcement learning. In International Conference on Machine Learning, pp. 1141-1150. PMLR, 2019.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Yizhou Chi, Lingjun Mao, and Zineng Tang.
Amongagents: Evaluating large language models in the
interactive text-based social deduction game. ArXiv preprint, abs/2407.116211, 2024. URL https:
/
16521.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: //github.comm wowppponoopps//pppeeeeaaeeeeeeee
2023.
Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, and Arjum Yadav. Gamebench: Evaluating strategic reasoning abilities of Ilm agents. ArXiv preprint, abs/24066000 06613, 2024. URL https://arxki http:://axxxvvoooeeeee org/abs////eeoao.ooo..gggg...a.r abs/a/as..22 06613.
Daantje Derks, Arjan ER Bos, and Jasper Von Grumbkow. Emoticons and social interaction on the internet: the importance of social context. Computers in human behavior, 23(1):842-8424 2007.
Jinhao Duan, Shiqi Wang, James Diffenderfer, Lichao Sun, Tianlong Chen, Bhavya Kailkhura, and Kaidi Xu. Reta: Recursively thinking ahead to improve the strategic reasoning of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 2232 2246, 2024a.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel- Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning limitations of Iims via game-theoretic evaluations. ArXie preprint, abs/2402.12345. 2024b. URL abs/2402. 12348.
org/
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407. 21783, 2024. URL attps://aaee /arxiv.xvvvvv//////////////aaav. org/abs hts//bb//4407..//////// 2207721777...
Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao Hc. Can large language models serve as rational players in game theory? a systematic analysis. ArXie preprinl, fas/2331.00588. 2023. URL abs/2312. 05488..
org/
Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. Chessgpt: Bridging policy learning and language modeling. Advances in Newral Information Processing Systems, 36, 2024.
Chaim Fershtman. The importance of the agenda in bargaining. Games and Economic Behavior, 2(3): 1990.
Nicol'o Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: How do large language models behave in the prisoner's dilemma? Arxiv preprint, abs/ //////2206.660.0. 2024. URL https://arxiv.or org/ abb/ 2406. 13605.
Martin E Ford and Marie S Tisak. A further search for social intelligence. Journal of Educational Psychology, 75(2):196, 1983.
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self- play and in-context learning from ai foodback. ArXio preprint, abs/2305.10142, 2023. URL https: paaxxiv.org// www..bs///ab..sso/sssssss//////e 10142.
Drew Fudenberg and Jean Tirole. Game theory. MIT press, 1991.
Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. ArXiv preprint, abs/ has/2305519168 2023. URL https://arxiv.cr abs/ 19165.
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Understanding social reasoning language models with language models. Advances in Neural Information Processing Systems, 36, 2024.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Michael Georgeff, Barney Pell, Martha Pollack, Milind Tambe, and Michael Wooldridge. The belief-desire- intention model of agency. In Intelligent Agents V: Agents Theories, Archileclures, and Languages: 5th International Workshop, ATAL'99 Paris, Frunce, July 4-,, 1998 Proceedings 5, pp. 1-10. Springer, 1999.
Fosca Giannotti, Mirco Nanni, Fabio Pinelli, and Dino Pedreschi. Trajectory pattern mining, In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 330 339, 2007.
Alison Gopnik and Janet W Astington. Children's understanding of representational change and its relation to the understanding of false belief and the appearance-reality distinction. Child development, pp. 26-37, 1988.
Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, and Yizhou Wang. Richelieu: Self-evolving Ilm-based agents for ai diplomacy. ArXiv preprint, abs/ 2bs/2400.06813,, 2024. URL https://axxi,, httpp////aawvw..//aaa//////e org/abs/2407 06813.
Fulin Guo. Gpt in game theory experiments. ArXiv preprint, abs/2305.05516, 2023. URL https: org/abs/2305 05516.
Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. Suspicion-agent: Playing imperfect information games with theory of mind aware gpl-4. ArXiv preprinl, abs/ 0bs/2239.112777 2023. URL /stpps://axiv.... http://..gg/////////////o/se /ttp//339.1277//
Shangmin Guo, Haoran Bu, Haochuan Wang, Yi Ren, Dianbo Sui, Yuming Shang, and Siting Lu. Economics arena for large language models. ArXiv preprint, his/2401.011730 2024. URL https://arxi: org/ //b/// 01735.
Akshat Gupla. Are chatgpt and gpt-4 good poker players?-a a pre-flop analysis. abs/2308.124444 2023. URL https://aat //tpvv/xxvvv../////////////..a http://aww....ga///////////a htt///b///300....//// 12466.
ArXio preprind,
Peter Hase, Thomas Hofweber, Xiang Zhou, Elias Stengel-Eskin, and Mohit Bansal. Fundamental problems with model editing: How should rational belief revision work in Ilms? ArXie preprint, abs/2406. 2024. URL
19354.
He He, Jordan learning. In International conference on machine learning, pp. 1804-1813. PMLR, 2016.
Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement
Daniel A Herrmann and Benjamin A Levinstein. Standards for belief representations in llms. ArXiv preprint, abbs/2405.2100 2024. URL https:// http://aaxiiv.oo http://wwww..ogg///////// 21030.
John J. Horton. Large language models as simulated cconomic agents: What can we learn from homo silicus? SSRN Electronic Journal, 2023. URL https://api. semanticscholar................e wttp:eiscccccaaoooorrroirooi org/Corpps
255152440.
Dirk Hovy and Diyi Yang. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 588 602, Online, 2021. Association for Computational Linguistics. doi: 10. 1011853i/////22.mmmmmmmma..k URL ptttps://aclaathoolll org/2021. naacl-main 49.
naacl-main.49
Chengpeng Hu, Yunlong Zhao, Ziqi Wang, Haocheng Du, and Jialin Liu. Games for artificial intelligence research: A review and perspectives. IEEE Transactions on Artificial Intelligence, 2024a.
Sihao Hu, Tiansheng Huang, Fatih Ilhan, Selim Tekin, Gaowen Liu, Ramana Kompella, and Ling Liu. A survey on large language model-based game agents. ArXiv preprint, abs/2404.02039, 2024b. URL https://arxiv httpa:///avvvvoo/aaaaaaee aaw.a///..aaa/ee////e/e./eeee..a 02039.
Wenyue Hua, Ollic Liu, Lingyao Li, Alfonso Amayuclas, Julie Chen, Lucas Jiang, Mingyu Jin, Lizhou Fan, Fei Sun, William Wang, et al. Game-theoretic Ilm: Agent workflow for negotiation games, arx preprint urXiv:2/11.008 2024a.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Yuncheng Hua, Lizhen Qu, and Gholamreza Haffari. Assistive large language model agents for socially- aware negotiation dialogues. ArXio preprint, abs/2402.01737, 2024b. URL https://arxiv.ort abs/ 2173.
Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, and Yanru Zhang. Pokergpt: An end-to-end light solver for multi-player lexas hold'em em via large language model. Ar preprinl, abs/2401.06781, 2024. URL
hwtp:/orgggab///aeeeeee 06781.
Thelma Hunt. The measurement of social intelligence. Journal of Applied Psychology, 12(3):317, 1928.
Jingru Jia, Zehua Yuan, Junhao Pan, Paul E McNamara, and Deming Chen. Decision-making behavior evaluation framework for llms under uncertain context. arXiv preprint XXxx:2240600972,, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b.
Ariio preprint, 0bs/2330.00682 2023. URL http:///axxvv..r.ee htt/abbb/231000 06825.
Xuanfa Jin, Ziyan Wang, Yali Du, Meng Fang, Haifeng Zhang, and Jun Wang. Learning to discuss strate- gically: A case study on one night ultimate werewolf. ArXiv preprint, abs/2405.19946 2024. URL /ttps://arxiv... hrrg/aba/24055 19946.
Chani Jung, Dongkwan Kim, Jiho Jin, Jiseon Kim, Yeon Seonwoo, Yejin Choi, Alice Oh, and Hyunwoo Kim. Perceptions to beliefs: Exploring precursory inferences for theory of mind in large language models. ArXio preprint, /t///24000000044 2024. URL https: http//axivv..r//// www.xii..rr/bb////0000
John H Kagel and Dan Levin. The winner's curse and public information in common value auctions. American ecomomic review, (6(55):944992)) 1986.
Nora Kassner, Oyvind Tafjord, Ashish Sabharwal, Kyle Richardson, Hinrich Schuetze, and Peter Clark. Language models with rationality. ArXiv preprint, abs/2305. 14250,, 2023. URL https://arxiv www.aaawww.....aaaaaaaa 2305.14250.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Couvvile, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXio preprint or Xiv:224100000 10.01679, 2024.
John F Kihlstrom and Nancy Cantor. Social intelligence. Handbook of intelligence, 2:359 379, 2000.
Minsoo Kim, Jongyoon Kim, Jibyuk Kim, and Seung-won Hwang. QuBE: Question- based belief en- hancement for agentic LLM reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Melhods in Natural Language Processing, pp- 21403 21423,, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: URL
11111653//112224.mmm///
emnlp-main 1193/.
Henry Kissinger. Diplomacy. In Geopolitics, pp. 114 115. Routledge, 2014.
Michal Kosinski. Theory of mind might have spontancously emerged in large language models. ArXiv preprint, abs/ has/2302.02083, 2023. URL htttps://araaee http:///xxxvv.r////// wwt.:/os/////////////////// 02083.
Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina. Sillos, Lorie De Leon, Camille Flepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz Candioao James Maningo, et al. Performance of chatgpt usmile: potential for ai-assisted medical education using large language models. PLoS digital health, 2(2): e0000198, 2023.
Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Amugrah Hayati, James M Rehg, and Diyi Yang. Werewolf among us: multimodal dataset for modcling persuasion behaviors in social deduction games. ArXiv preprint, abs/2212.08279, 2022. URL https://arxs org/ abs/2212.08270
25

---

Published in Transactions on Machine Learning Research (05/2025)
Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, De-Yong Ye, Peilin Zhao, Ec-Peng Lim, Hui Xiong, and Hao Wang. Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay. ArXiv preprint, abs/ 0ts//330.119988 2023. URL http:///aaxivvooraabbeeeeee
Yan Leng and Yuan Yuan. Do 11m agents exhibit social behavior? Xie preprint, abs/2312.15199, 2023.
URL /ttp///xxii..orraa///22222.. 15198.
Benjamin Levinstein and Daniel A Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks. Philosophioul Studies, pp. 27, 2024.
Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Ka- tia Sycara. Theory of mind for multi-agent collaboration via large language models. ArXiv prepriud, abs/2310.10701, 2023a. URL
10701.
Jiatong Li, Rui Li, and Qi Liu. Beyond static datasets: A deep interaction approach to Ilm evaluation. ArXio preprint, abs/2309.00 04369, 2023b. URL https: wtwp:rgg//aabgggoeeee 04369.
Minahi Li, Weiyan Shi, Caleb Ziems, and Diyi Yang. Social intelligence data infrastructure: Structuring the present and navigating the future. ArXiv preprint, abs/ Abs/2403.14639, 2024a. URL https://arxiv.or org/ abs/2403. 14659.
Yang Li, Shao Zhang, Jichen Sun, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan. Cooperative open- ended learning framework for zero-shot coordination. In International Conference on Machine Learning, pp. 20470-20000 20484. PMLR, 2023c.
Yuanchun Li, Hao Wen, Weijun Wang. Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal 1lm agents: Insights and survey about the capability, efficiency and security. ArXiv preprint, abs/2401. /49024545,5499 2024b. URL http:////aawvo.o/a///////// org/abs/2401 05459.
Zongyuan Li, Yanan Ni, Runnan Qi, Lumin Jiang, Chang Lu, Xiaojie Xu, Xiangbei Liu, Pengfei Li, Yunzheng Guo, Zhe Ma, et al. Lim-pysc2: Starcraft ii learning environment for large language models. arXiv preprint arXiv:24 11. 11.05448 2024c.
Austen Liao, Nicholas Tomlin, and Dan Klein. Efficacy of language model self-play in non-zero-sum games. ArXie preprint, abs/24000 02///2440.8877 2024. URL https://arriv.orr///b//4466 wrggabs/////....//////////////oe
18872.
Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating 1lms playing the game of avalon. In NewrIPS 2023 Foundation Models for Decision Making Workshop, 2023a.
Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. From text to tactic: Evaluating 1lms playing the game avalon. ArXiv preprint, abs/ 0te//2310..0000 0.00553,, 2023b. URL batps::/arxiis http:///aaaiii.orga///////////// http://bb/2331.05555333 05036.
Jonathan Light, Min Cai, Weiqin Chen, Guanzhi Wang, Xiusi Chen, Wei Cheng, Yisong Yue, and Ziniu Hu. Strategist: Learning strategic skills by llms via bi-level tree search. Arxiv preprint, 240//22088.1633, 2024.
URL attps://arxiv org/ abs/ abwp/4//....////////////// 1405..000....
Yang Liu, Weixing Chen, Yongjie Bai, Jingzhou Luo, Xinshuai Song, Kaixuan Jiang, Zhida Li, Ganlong Zhao, Junyi Lin, Guanbin Li, et al. Aligning cyber space with physical world: A comprehensive survey on embodied ai. ArXiv preprint, abs/2400.06688 2024a. URL
06886.
Ziyi Liu, Abhishek Anand, Pei Zhou, Jen-tse Huang, and Jieyu Zhao. Interintent: Investigating social intelli- gence of Iims via intention understanding in an interactive game context. ArXiv preprint, Abs/2240.112000 2024b. URL https://arxi:: .wwpeiiioorr///eee org/ abs/ 12203.
Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On Ilms-driven synthetic data generation, curation, and evaluation: A survey. ArXiv preprint, abs/2406.15516, 2024. URL https://axxii....o..m/ http://xxivv.or//a////.... 15126.
Nunzio Lorè and Babak Heydari. Strategic behavior of large language models and the role of game structure versus contextual framing. Scientific Reports, 14(1):18490, 2024.
26

---

Published in Transactions on Machine Learning Research (05/2025)
Ji Ma. Can machines think like humans? behavioral evaluation of Ilm-agents in dictator games. arXiv preprint or 1022410013599 2024.
Qun Ma, Xiao Xue, Deyu Zhou, Xiangning Yu, Donghua Liu, Xuwen Zhang, Zihan Zhao, Yifan Shen, Peilin Ji, Juanjuan Li, et al. Computational experiments meet large language model based agents: A survey and perspective. ArXio preprint, abs/2402.00263 2024. URL https //arxvaaaivv. http//awiii.rr.aaa///// atgg/bs//2an 00262.
Weiyu Ma, Qirui Mi, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. Large language models play starcraft it: Benchmarks and chain of summarization approach. ArXiv preprint, abs/ abs/2312. 11665, 2023. URL https: / /tt////axvvvv//ee//ee/eeeeeeeeve .ow. aa/aabss//// 11112.18865 11865.
Bahar Mahmud, Guan Hong, and Bernard Fong. A study of human ai symbiosis for creative work: Re- cent developments and future directions in deep learning. ACM Transactions on Multimedia Computing, Communicalions and Applications, 20(2):1-21 2023.
Shaoguang Mao, Yuzhe Cai, Yan Xia, Wenshan Wu, Xun Wang, Fengyi Wang, Tao Ge, and Furu Wei. Alympics: Language agents meet game theory. ArXiv preprint, abs/2311.032200, 2023. URL https: //arxiv.org//bs/22333 http//bb/2311..3222 03220..
Leena Mathur, Paul Pu Liang, and Louis-Philippe Morency. Advancing social intelligence in ai agents: Technical challenges and open questions. Arxio preprint, 0bs/2444.111022 2024. URL https:///axx:: org/abs/2004 11023.
Qiaozhu Mei, Yutong Xie, Walter Yuan, and Matthew O Jackson. A turing test of whether ai chatbots are behaviorally similar to humans. Proceedings of the National Academy of Sciences, 131191122333 2024.
J Juanjuan Meng. Ai emerges as the frontier in behavioral science. Proceedings of the National Academy of Sci- ences of the United States of America, 121 10xc2401336121, 2024. URL https://apl semanti ccchola.r....... org/CorpusID: (CoorpuIDD2680293777 268029379.
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisy, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):5088 513, 2017.
Rémi Munos, Michal Valko, Danicle Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhao- han Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. ArXiv preprint, abs/2312.00856, 2023. URL attps::/arrt http///axvv...g//bbaaaase arg///a///22...... 00886.
Aran Nayebi. Barriers and pathways human-ai alignment: A game-the approach. arXiv preprint 0:Xi0:2502.05934, 2025.
Sean Noh and Ho-Chun Herbert Chang. Lims with personalities in multi-issue negotiation games. Ar preprint, hes//2405055244 2024. URL
wwwg////////....././//////// 05248.
Maayan Orner, Oleg Maksimov, Akiva Kleinerman, Charles Ortiz, and Sarit Kraus. Explaining decisions agents in mixie-motive games. ArXio preprint, abs/2400.152555, 2024. URL https: //ariiv.//////////aa .rrg org/abs/ 2407. 15255.
Guillermo Owen. Game theory. Emerald Group Publishing, 2013.
Steve Phelps and Yvan I. Russell. The machine psychology of cooperation: Can gpt models operationalise prompts for altruism, cooperation, competitiveness and selfishness in economic games? ArXiv preprint, 2023. URL https://apl. http://p....aaatccccccmmm semanticscholar org/CorpusID 258685424.
Giorgio Piatti, Zhijing Jin, Max Kleiman- Mleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mi- haloca. Cooperate or collapse: Emergence of sustainability behaviors in a society of 11m agents. Arxi preprint, abs/2404.11663 2024. URL /ttp://rww....r//////....... 16698.
27

---

Published in Transactions on Machine Learning Research (05/2025)
David Premack and Guy Woodruff. Does the chimpanzee have theory of mind? Behavioral and brain sciences, (44):515-55 5266, 1978.
Siyuan Qi, Shuo Chen, Yexin Li, Xiangyu Kong, Junqi Wang, Bangcheng Yang, Pring Wong, Yifan Zhong, Xiaoyuan Zhang, Zhaowei Zhang, Nian Liu, Wei Wang. Yaodong Yang, and Song- Chun Zhu. Civrcalm: A learning and reasoning odyssey civilization for decision-making agents. ArXio preprind, abs/ abs/2401.10566, 2024. URL https://arxiv wwtp://o///ab//aa/eeee/eee 10568.
Song-Chun
Huida Qiu, Yan Liu, Niranjan A Subrahmanya, and Weichang Li. Granger causality for time-series anomaly detection. In 2012 IEEE 12th international conference on data mining, pp. 1074-1079. IEEE, 2012.
Anatol Rapoport and Albert M Chammah. Prisoner's dilemma: A study in conflict and cooperation, volume 165. University of Michigan press, 1965.
Reza Rawassizadch, Elahch Momeni, Chelsea Dobbins, Joobin Gharibshah, and Michael Pazzani. Scalable daily human behavioral pattern mining from multivariate temporal data. IEEE Transactions on Knowledge and Data Engineering. (811)333098 3098 3112, 2016.
Siyue Ren, Zhiyao Cui, Ruiqi Song, Zhen Wang, and Shuyue Hu. Emergence of social norms in generative agent societies: principles and architecture. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI), 2024.
Jillian Ross, Yoon Kim, and Andrew W Lo. Llm economicus? mapping the behavioral biases of Ilms via utility theory. Are preprinl, abs 22244400777444 2024. URL https: www.ar/ww..orra////a/aaa http://////0000004/0/..
Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. Evaluating the moral beliefs encoded in llms. Advances in Newral Information Processing Systems, 36, 2024.
Stefan F Schouten, Peter Bloem, Ilia Markov, and Piek Vossen. Truth-value judgment in language models: belief dircctions context sensitive. ArXiv preprint, abs/2404.18866, 2024. URL https://arx//: http:///xxvvoo/////////// org/ aba/2404.1 18865.
Melanic Sclar, Sachin Kumar, Peter West, Alanc Suhr, Yejin Choi, and Yulia Tsvetkov. Minding lan- guage models' (lack of) theory of mind: plug-and-play multi-character belief tracker. ArXio preprind, 00924, 2023. URL https:// /ttp///xivv.////////////////atv/ wwg/abs/////eeoeooooooooooo.eert www.aa/.....eeeeee/// 00924.
Xiao Shao, Weifu Jiang, Fei Zuo, and Mengqing Liu. Swarmbrain: Embodied agent for real-time strategy game starcraft ii via large language models. ArXiv preprint, abs/2401.17749, 2024. URL https://arxi: hrg/abs/2401.17//
Eilam Shapira, Omer Madmon, Itamar Reinman, Samuel Joseph Amouyal, Roi Reichart, and Moshe Ten- nenholtz. Glee: A unified framework and benchmark for language- preprint Xrxi::2410.080 05254, 2024.
bastn based economic environments. anXiv
Zijing Shi, Meng Fang, Shunfeng Zheng, Shilong Deng, Ling Chen, and Yali Du. Cooperation on the fly: Exploring language agents for ad hoc teamwork in the avalon game. ArXio preprint, abs/2312 2312.1111, 2023. URL https:/
17515.
Hisaichi Shibata, Soichiro Miki, and Yuta Nakamura. Playing the werewolf game with artificial intelligence for language understanding. arXiv preprint arXiv:2302. 10646, 2023.
Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas I. Griffiths. Cognitive architectures for language agents. ArXiu preprint, abs/ abs/2330.02427, 2023. URL https://armas www.ooog/////eeee 2202//2277..02
Lawrence E Susskind. Scorable games: A better way to teach negotiation. Negot. J., 1:205, 1985.
Reiji Suzuki and Takaya Arita. An evolutionary model of personality traits related cooperative behavior using large language model. Scientific Reports, 14, 2023. URL https://api. semanticscholar. org/
ID 12:3883099.
28

---

Published in Transactions on Machine Learning Research (05/2025)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -ran-tBapistte Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ArXiv preprint, abs/ abs/2312.11805, 2023. URL https://arxiv.org/ wtt.:./gg..a.......//// 11805.
Jen tse Huang, Eric Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R. Lyu. How far are we on the decision-making of Ilms? evaluating Ilmss gaming ability in multi-agent environments. ArXiv preprinl, als/2403.11807,, 2024. URL https: //arxiv.orge wwt.v..rrgabb//40000 ab2//433..2.... 11807.
Wiebe Van Der Hoek, Wojeiech Jamroga, and Michael Wooldridge. A logic for strategic reasoning. Tn Proceedings of the fourth international joint conference on Autonomous agents and multingent systems, pp. 157 164, 2005.
Lei Wang, Chen Ma, Xucyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomons agents. Frontiers of Computer Science, 18(6):186345, 2024a.
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation. ArXie preprint, abs/2310.00320, 2023. URL attps://arxiv.. http://arxiv...g/bb/a//////////s hts//bb//2311/ 01320.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingehen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Opendevin: An open platform for ai software developers as generalist agents. Arxio preprinf, abs/2407.16741 2024b. URL httt:://arxxvv.org//b///////
orgg 16741.
Zhen Wang, Ruiqi Song, Chen Shen, Shiya Yin, Zhao Song, Balaraju Battu, Lei Shi, Danyang Jia, Talal Rahwan, and Shuyue Hu. Large language models overcome the machine penalty when acting fairly but not when acting selfishly or altruistically. arXiv preprint arXie:2410.0077 2024c.
Donald Arthur Waterman. Generalization learning techniques for automating the learning of heuristics. Artificial Intelligence, 1(1-2):121 170, 1970.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24883724-28877, 2022.
Dekun Wu, Haochen Shi, Zhiyuan Sun, and Bang Liu. Deciphering digital detectives: Understanding Ilm behaviors and capabilities in multi-agent mystery games. In Findings of the Association for Computational Linguistics ACL 2024, pp. 8225-8291 2024a.
Shuang Wu, Liwen Zhu, Tao Yang, Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu. Enhance reasoning for large language models in the game werewolf. ArXiv preprint, 22032442.002300, 2024b. URL https:
//arxiv. wwwxiv.org//i. www.ooog/////aaa.ee..ooe /tsp/b/2..02222
Tian Xia, Zhiwei He, Tong Ren, Yibo Miao, Zhuosheng Zhang, Yang Yang, and Rui Wang. Measuring bar- gaining abilities of llms: A benchmark and a buyer-enhancement method, ArXiv preprint, abs/ abs/2442.158813, 2024. URL https: / //tr:i///xxivve/ org/
Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Kentzer, See-Kiong Ng, and Jiashi Feng. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2023a.
Rongwu Xu, Brian S Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. The earth is flat because...: Investigating lmms' belief towards misinformation via persuasive conversation. Xr preprint, abs/ abss/231200005, 2023b. URL http:://aaxxw.org/bbbb..//
09085.
Ruoxi Xu, Yingfei Sun, Mengjic Ren, Shiguang Guo, Ruotong Pan, Hongyu Lin, Le Sun, and Xianpei Han. Ai for social science and social science of ai: A survey. Information Processing & Management, 61()): 103665, 2024a.
29

---

Published in Transactions on Machine Learning Research (05/2025)
Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jicchuan Jiang, Zhiming Ding, and Börje F Karlsson. A survey on game playing agents and large models: Methods, applications, and challenges. ArXio preprint, abs/2403.10229, 2024b. URL httpp://taatatt. httpss//arriv.orr///a///22.. 10249.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Explor- ing large language models for communication games: An empirical study on werewolf. ArXiv preprint,
04008, 2023c. URL https://arxiv. http:///aaxiv..o/abb/////// wwtp:/rgg//gg.ooo.//// 04658.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. ArXiv preprint, abs/2310.189400, 2023d. URL https://arxiv.orr org/ abs/2310. 18940.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. ArXiv preprint, abs/2400.10671 2024a. URL https://arxin org/abst aa///b///44/0 10671.
Ruihan Yang, Jiangjie Chen, Yikai Zhang, Siyu Yuan, Aili Chen, Kyle Richardson, Yanghua Xiao, and Deqing Yang. Selfgoal: Your language agents already know how to achieve high-level goals. ArXiv preprint, abs/2406.04777 044844 2024b. URL https://arxis: worg/iiorr///oo /as////bb///...////// 04784.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yauwai Yim, Chunkit Chan, Tianyu Shi, Zheye Deng, Wei Fan, Tianshi Zheng, and Yangqiu Song. Evaluating and enhancing Iims agent based on theory of mind in guandan: A multi-player cooperative game under im- perfect information. ArXiv preprint, abs/2408.02555 02559, 2024. URL https: /ttp///arvvvvv/////////////aaava oow.ooog///...oee abs////s/////////////////////a/e /205.02559.
Haolan Zhan, Yufei Wang, Tao Feng, Yuncheng Hua, Suraj Sharma, Zhuang Li, Lizhen Qu, Zhaleh Semnani Azad, Ingrid Zukerman, and Gholamreza Haffari. Let's negotiate! survey of negotiation dialogue systems. Ari preprint, abss/2402.0000 2024. URL https: //axiv... org/abs///// 01097.
Wenqi Zhang, Ke Tang, Hai Wi,, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. Agent-pro: Learning to evolve via policy-level reflection and optimization. arXiv preprint ar Xiv:24402 1 17774, 2024a.
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan WI, Ting Song, Man Lan, and Furu Wei. Llm mastermind: A survey of strategic reasoning with large language models. ArXiv preprint, abs/2404.01130, 2024b. URL attps://arxiv.ot argg/bb//2044. 021///2440..120.0..
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm mastermind: A survey of strategic reasoning with large language models. ArXiv preprint, abs 220/2240000 2024c. URL https://arxiv.orgg
01230.
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Yan Xia, Man Lan, and Furu Wei. K-level reasoning with large language models. ArXio preprint, abs/ 2b0220202,1, 2024d. URL https://arxx http//aaxxxv.../abbabaaa/
Yadong Zhang, Shaoguang Mao, Wenshan Wu, Yan Xia, Tao Ge, Man Lan, and Furu Wei. Enhancing language model rationality with bi-directional deliberation reasoning. Ar preprint, abs/2407.06112, 2024e. URL https: //arxiv...iive//a http:///wwwwwww./aa/aaa/se http//227000.111.
Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Competeai: Understanding the competition dynamics in large language model- based agents. In Forty-first Interna- tional Conference on Machine Learning, 2023. URL https://apl. semanticscholar wwwphhhllarrrrroooooooooo CrggCorD 270357283.
Qinglin Zhu, Runcong Zhao, Jinhua Du, Lin Gui, and Yulan He. Player*: Enhancing Ilmebased multi-agent communication and interaction murder mystery games. ArXiv preprint, abs/2404..166 2024a. URL http://axxvv..o//abbbe 17662.
30

---

Published in Transactions on Machine Learning Research (05/2025)
Wentao Zhu, Zhining Zhang, and Yizhou Wang. Language models represent belicfs of self and others. ArXiv preprinl, abs/2440.18896, 2024b. URL
wwtp://////........ 18496.
Richard Zhuang, Akshat Gupta, Richard Yang, Aniket Rahane, Zhengyu Li, and Gopala Anumanchipalli Pokerbench: Training large language models to become professional poker players. arviv preprint arXiu:251100832, 2025.
31


Reviews:
[{"review_id": "SXuEJZXTt4", "reviewer": "Reviewer_xzPJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### What I like about this paper\n\n- The paper is well written and well-structured paper. The authors go beyond simply listing the recent works, each section ends on a Takeaways section which is exactly what I look for when reading survey papers.\n- Figure 2 provide a nice taxonomy of many of the references used for this survey paper.\n- I appreciated the many figures illustrating concrete examples of game scenarios and social agent modules. That helped me understand\n- As someone outside the game-theoretic social scenarios field, I feel I have acquired valuable insights and knowledge on that topic. To me, this survey is a good starting point to start digging deeper.\n\n### Opportunities for improvement\n- I feel the survey could do a better job a unifying the different terms used in different papers and that seem to all point to the same thing. For instance, \"AI personas\" (Section 2.1), \"social behaviours\" (Section 2.2), \"traits\", \"preferences\" (Section 3.1), etc. If they are not the same, I believe a comparison between those would be appreciated by the readers.\n\n- When writing papers, I found there is a delicate balance between over-providing term definitions vs. referring the readers to external sources. I do believe survey papers (especially journal papers) should aim to be self-contained as much as possible by at least providing a one sentence summary definition and provide reference for more in-depth details. For instance, I had to look up the Matthew Effect. Another example, when mentioning \"prisoner's dilemma\" is simply refered as \"the most famous and recognized game\". A brief explanation would go a long way, or in this specific case, the authors could refer the readers to Fig 3 since there's actually an example.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is survey about evaluating the social intelligence of Language Agents by making LLMs play social games. The litterature review is split in three parts: 1) what are the existing game frameworks, 2) what are the common architecture designs for such social agents, and 3) what are the main evaluation criteria used to evaluate social intelligence. The authors provide a taxonomy of the field mapping recent works to different research topics. They also make use of several figures to support and explain the different topics discussed in this paper. Overall, this survey paper aims at depicting the recent research in this field as accurately as possible and provides many takeways one can act upon to extend it.\n\nAs a researcher in an adjacent field, it is my impression that I got a great deal of information from this survey paper. I'm certain other in the TMLR community will get something out of reading it as well. I recommend this paper for acceptance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Typos\n- Figure 3 (left). \"the payoff matric\" -> \"... metric\"\n- Figure 3 (middle). \"your best size\" -> \"... bet size\"", "broader_impact_concerns": "I encourage the authors to include a Broader Impact section. For instance, what are the implications of developing such Social Agents and making them play against/with humans? Are those game-theoretic frameworks a valid approach to measure LLMs ability to deceive people? \n\nAlso, it is my understanding that most the recent research papers covered in this survey focus on evaluation of the GPT-family models. While OpenAI's models were the first to be \"easily\" accessible to the research community, I'm sure recent works must have considered other models as well, e.g. Claude, Gemini, and even the open-weights ones like Llama? If that's not the case, I believe this bias should be pointed out in this survey. In my opinion, this is a valid takeaway and would encourage the community to diversify their model choices.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GqXzvwGNJa", "reviewer": "Reviewer_dmrB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. This paper is well-structured with the three lens mentioned above covering the important aspect of the research scope as well as boxes takeaways with discussion. \n2. The papers covered in the survey are recent and timely.\n\n\n## Weaknesses\n\n1. It is hard to evaluate a survey's contribution to the knowledge of practitioners. After reading it, I don't think this paper provides any novel views after summarizing the papers. The completeness of the survey makes it especially hard for readers to grasp key insights. \n2. What are the fundamental problems of LLMs when they are applied to these game-theoretical scenarios? This paper provides some \"balanced\" views, e.g. LLMs could have belief or not have belief, without strong arguments for either. \n3. I think Section 5, is more important than other sections, since it points out potential directions in this field. It should be expanded.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary \nThis paper is a survey on multi-llm-agent literature with a specific focus on game theory inspired settings [^1].\n\nIt discusses the current work in this area through three lens:\n\n1. Game Framework: choice (non-verbal) focused, or communication (verbal) focused\n2. Agent Design: preference, belief, and reasoning modules\n3. Evaluation: outcome evaluation, and behavior evaluation\n\n## Some key takeaways\n\n### Game Framework\n1. Current research lacks unified evaluation frameworks\n2. Data contamination is a concern as classic games may be in pre-training data\n3. Limited exploration of \"strategic behavior\" through \"action language\" in games with minimal verbal communication\n4. More realistic and diverse games promote greater agent behavior diversity\n5. Need for better process evaluation mechanisms beyond just outcomes\n\n### Social Agent\n1. Preference: LLMs show some ability to follow preferences but struggle with less common preferences\n2. Belief: Ongoing debate about whether LLMs possess beliefs; models show limited and unstable belief revision capabilities\n3. Reasoning: Theory-of-Mind reasoning in LLMs remains contentious; Reinforcement Learning combined with search techniques shows promise\n\n### Evaluation Protocol\n1. Win rate alone is insufficient; researchers propose expanded metrics like efficiency-adjusted win rate and comeback win rate\n2. Need for a comprehensive evaluation metrics system that is hierarchical, abstract, and quantifiable\n\n## Contributions\n\n1. A good survey for people who are getting into this field to understand the latest research. \n2. Pointing out important future directions including pluralistic scenarios with diverse goals, cultures, and values\n\n[^1]: Although the definition of game theoretical scenarios is not clearly defined, e.g. role-playing goal-oriented scenarios like Sotopia is not included in this study, but I guess the authors are focusing on the cases where the rules for win and lose are clearly defined.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please expand Section 5 to provide sketches with references regarding the potential future directions:\n\n1. Standardized Benchmark Generation: What are the desiderata for a standardized benchmark? How should we evaluate agents in this benchmark? How to generate the scenarios? \n2. Reinforcement Learning Agents: What insights in MARL should be integrated into LLM agent training? What are the challenges? Efficiency? Generalization? \n3. Behaviour Pattern Mining: What are the potential methods to mine these behavior patterns? \n4. Pluralistic Game-Theoretic Scenarios: Similar to 1. \n\nThis is still a new field, so I appreciate the authors' effort to put together a survey to try to unify the literature. However, a survey should go beyond summarization, but also provides new insights that one could not easily get by reading the summaries.", "broader_impact_concerns": "Agents that learn to deceive or compete with humans should be studied with caution. So a Broader Impact Statement should be provided to discuss potential harms when the agents become socially intelligent.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Znb89l6uzC", "reviewer": "Reviewer_uaWb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \nThis survey is well-written and clear. It provides a comprehensive review of relevant work in the field, with detailed descriptions of each study. I also appreciate the \"Takeaways\" sections at the end of each part, which help readers reflect on key research problems in a timely manner.\n\n**Weaknesses**:\nI don’t see major issues with this survey. However, one weakness is that the three modules discussed in the \"Social Agent\" section—Preference, Belief, and Reasoning—feel somewhat scattered and disconnected. It would be helpful to include a discussion on how these modules interact during agent activity or their roles in the decision-making process. In my opinion, the lack of connections between them is noticeable, as agents' cognition and actions rely on an interplay of these components. For instance, can the Preference and Belief modules influence Reasoning, and vice versa? Clarifying these relationships would strengthen the coherence of this section.\n\nAnother weakness is the lack of an overview of the current performance of game agents. While I understand that providing quantitative results or direct performance comparisons is challenging, some qualitative insights would be valuable. For example, how does agent performance compare to human players? What game scenarios pose the greatest challenges for agents? Addressing these questions would give readers a better understanding of the progress of existing game agents research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey examines LLM-based social agents in game-theoretic scenarios, covering three key aspects: Game Framework, Social Agents, and Evaluation Protocol. For each aspect, it provides a comprehensive review of related work, organizing it in a problem-oriented structure. Additionally, it presents the authors' perspectives on current challenges and limitations in the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In line 6, paragraph 4 of Section 1, \"*... such as classic game-theoretic and poker ...*\". I think replacing \"classic game-theoretic\" with a more concrete example, such as \"prisoner's delimma\" would be better. \"classic game-theoretic\" is not straighforward to understand or imagine.\n2. In Section 3, it would be helpful to include a description or a pipeline illustrating how the three modules (Preference, Belief, and Reasoning) interact during agent activity. \n3. In Figure 7, the two reasoning methods are presented side by side, which may lead readers to unconsciously compare them. To facilitate a more meaningful comparison, it would be better if both methods were applied to the same game scenario, making their tactical differences more apparent. If they are not directly comparable, the authors could clarify this in the figure caption, noting that \"Theory of Mind reasoning\" is particularly suited for multi-agent scenarios, while \"RL reasoning\" is used for dynamic environments.\n4. The authors might also consider adding a discussion in Section 4 on agent performance across different game scenarios.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have expanded their work to a broader literature, recent publications, and included guidance for future work in the area.
Decision: Accept (poster)