Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf
Authors: Xuanfa Jin
Ziyan Wang
Yali Du
Meng Fang
Haifeng Zhang
Jun Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
LLM-based Agent
Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: We propose a novel RL-instructed language agent framework to play the One Night Ultimate Werewolf game.

Abstract: Communication is a fundamental aspect of human society, facilitating the exchange of information and beliefs among people. Despite the advancements in large language models (LLMs), recent agents built with these often neglect the control over discussion tactics, which are essential in communication scenarios and games. As a variant of the famous communication game Werewolf, *One Night Ultimate Werewolf* (ONUW) requires players to develop strategic discussion policies due to the potential role changes that increase the uncertainty and complexity of the game. In this work, we first present the existence of the Perfect Bayesian Equilibria (PBEs) in two scenarios of the ONUW game: one with discussion and one without. The results showcase that the discussion greatly changes players' utilities by affecting their beliefs, emphasizing the significance of discussion tactics. Based on the insights obtained from the analyses, we propose an RL-instructed language agent framework, where a discussion policy trained by reinforcement learning (RL) is employed to determine appropriate discussion tactics to adopt. Our experimental results on several ONUW game settings demonstrate the effectiveness and generalizability of our proposed framework.

Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf
Xuanfa Jin 1,, Ziyan Wang"?, Yali Duti2, Meng Fang'. Haifeng Zhang , ,Jn Wang 1
Institute of Automation, Chinese Academy of Sciences, 2 Cooperative Al Lab, Department of Informatics, King's College London, School of Artificial Intelligence, University of Chinese Academy of Sciences, 4 University of Liverpool, "Najjing Artificial Intelligence Research of IA, AI Centre, Department of Computer Science, UCL
Abstract
Communication is fundamental aspect of human society, facilitating the exchange of information and beliefs among people. Despite the advancements in large lan- guage models (LLMs),, recent agents built with these often neglect the control over discussion tactics, which are essential in communication scenarios and games. As a variant of the famous communication game Werewolf, One Night Ultimate Werewolf (ONUW) requires players to develop strategic discussion policies due the potential role changes that increase the uncertainty and complexity of the game. In this work, we first present the existence of the Perfect Bayesian Equi- libria (PBEs) in two scenarios of the ONUW game: one with discussion and one without. The results showcase that the discussion greatly changes players' utilities by affecting their beliefs, emphasizing the significance of discussion tactics. Based on the insights obtained from the analyses, we propose an RL-instructed language agent framework, where discussion policy trained by reinforcement learning (RL) is employed to determine appropriate discussion tactics to adopt. Our ex- perimental results on several ONUW game settings demonstrate the effectiveness and generalizability of our proposed framework. The project page of our paper: onne-iigh-ullimatee werewolf github.io
Introduction
Many games such as StarCraft 2 Diplomacy can approximate various fundamental issues in real life, and studying these games contributes a better understanding of the functioning of our society (5). Designing artificial intelligence (AI) agents that can play these games well has attracted lot of attention 66, 3. Fortunately, recent large language models (LLMs) have demonstrated significant potential in constructing intelligent agents in numerous tasks 7410 due their impressive reasoning and emergent generalization abilities [HH3) Moreover, LLM-based agents have achieved approaching or even surpassing human performance games like Chess [4], Minecraft [7]
Avalon [18 and Werewolf 23). etc.
In the game Werewolf 24 the hidden roles and uncertain discussions greatly influence the gameplay, making it challenging for players. Compared to it, the One Night Ultimate Werewolf (ONUW) game adds roles that can change the roles of players, and players only have one nighttime take action and one daytime to discuss and vote. Therefore, the key challenge of the ONUW game is to deduce
"qqual contribution. Author ordering is determined by coin flip. Cooresponneecee to: (yali.du@) kel.cc.u), (haifeng zhanonng..angeee @ ia.ac.cn), (jun.wagg4 us.uul.ac.uk).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and distinguish the final roles of all players from their statements. Meanwhile, due to the sequential execution of actions at night, the information obtained by the former player might be not reliable, leading to the rise of reasoning difficulty and uncertainty. For example, Seer and Robber are two roles in the ONUW game with special abilities, and the Seer saw a player was a Werewolf at night. However, if the Robber switched that player's s role later on, the information obtained by Seer becomes invalid. So players need to discuss strategically, leveraging others' statements and their own prior knowledge to guide other players to reveal their information, or conceal their identities by misleading or deceiving, particularly for Werewolves. Recent works [21 22 19 have attempted to construct LLM-based agents that can play communication games, but they mainly focus on how to fully utilize reasoning and generalization abilities of LLMs, while neglecting the control over strategies.
In this work, we focus on enhancing the discussion ability of ALM-based agents by leveraging the ONUW game. We formulate the ONUW game as Multi-Phase Extensive- Bayesian Game and explore various discussion tacties in the game. Through analyzing a three-player ONUW game with two Werewolves and one Robber, we present the existence of the Perfect Bayesian Equilibria (PBEs) (25 in two distinct scenarios: one with discussion and one without. Players' utilities at equilibria when in the scenario with discussion highlight the significance of discussion, as they are only determined by players' beliefs, which are influenced by discussion. Based on the insights obtained from the analyses, we thus propose an RL-instructed LLM-based agent framework. This framework leverages policy optimized by reinforcement learning (RL) determine appropriate discussion tactic (e.g.. "Honest Evidence", "Deceptive Accusation", etc.) based on current observation. conduct experiments in
To evaluate the effectiveness and generalizability of our framework. three-player and five-player ONUW game. The results indicate that the integration of our discussion policy can help L.LM-based agents approximate PBEs more closely and improve the performance of LLM--basd agents. Moreover, we observe the discussion policy trained by RL performs better than that by directly prompting LLM.
Our contributions can be summarized as follows: Firstly, we formulate the ONUW game as a Multi- Phase Extensive-Form Bayesian Game and provide theoretical analyses of a three-player ONUW game, demonstrating that players' utilities are only determined by their beliefs, which reveals the pivotal role that discussion plays in the ONUW game. Secondly, we develop environment of the ONUW game, which is complicated due to the uncertainty caused by role changes. And we additionally contribute dataset featuring players employing various discussion tactics in the ONUW game. Finally, inspired by the importance of discussion shown in our analyses, we propose an RL-instruuced language agent framework, where instructive discussion policy trained by RL is integrated, and belief modeling method is employed to deduce the roles of players and generate actions based on its beliefs and discussion tactics.
2 Related Work
This section reviews literature in the following areas: the analyses of Mafia/Werewol games and the design of Al agents for communication games. Additional related work is covered in Appendix B
Maña is multi- maltirplaayr communication game, modeling a conflict between the mafias and civilians. Early research [26] (8) has attempted to identify the optimal strategies for both citizens and mafias theoretically and thereby determine what setups lead to fair games. However, these works are based on strong assumptions such as random lynchings or meaningless debate, and restricted to simple scenarios like without detectives or removing most discussion. Werewolf is a reinvention of the Mafia game by introducing additional roles with special abilities. An e-Nash equilibrium is found under the limitation of both villager-side and werewolf-side strategies (2).. In contrast, our analyses make few assumptions and strictly adhere to game rules, despite being specific cases. Some researchers have attempted to develop Werewolf agents that rely on rule-based systems or talking templates [30] (T], while some rely on language models directly trained with Werewolf logs 33] As for the One Night Ultimate Werewolf game, Eger and Martens 33 builds AI agents that can choose statements from a fixed set and designs system that allows human players to play games with their Al agents.
Recently, researchers have tried to explore the potential of LLMs playing the Werewolf game 23
Xu et al. 222 utilizes policy to select candidate actions generated by LLMs, which can be seen as an adjustment for the action distribution of the Thinker module for complex logical analysis and strategic planning using structured language features,
agent. Wu et al. (2) applies RI. to train a

---

ssignment
Night
Plaver E.Tubborrkkr
Playe 2: Werwell
Plyeer (Robboc)
Imcbormkerr nengs Player 3 Pager
Plaer
3. Day Phuse
Player Player limber
Plyer 5
Team Village wins
Z
Role
(1) At least Werewoil dies. Even if one or more players who are not Wermwolves
Pool
Plyert
Weswell
Village
Iuuomii:::
rounds of discussions
adkition
Werewalf dying.
Plaee5
(2) so one is Wecewolf and no
Ground Night)
g Poper
g Hapord
Mayse Rubber
Wieseelf
Playe
Team Werewollf wins
Voting Phase Players (escoept themselives
Werswaif and W Wesewolves are voted out.
At
Twobkenkee
Player
Figure 1: The game process of the ONUW game. Initially, roles are randomly dealt to players. Then three phases: Night (abilities performed in order), Day (discussion in three rounds), and Voting (suspicious player voted out) proceed sequentially. The winner is decided by the voting result.
aiming to improve the System-2 reasoning ability of agents. contrast, our work directly integrates discussion policy trained by RL into the thinking and decision- making process of LLM-based agents, focusing on enhancing agents' strategic discussion (e.g., honesty or deception) ability, which an aspect that was not prominently addressed in previous work.
Besides the Werewolf games mentioned above, there are also AI researches conducted on other communication games, such as Diplomacy [3] 4] Avalon 20 and SpyFall [34] 35). There has been long history of studying Diplomacy the realm of AI 366 37, but most of them are rule-based algorithms until recently. Cicero 3 integrates a dialogue module with a planning module trained by RL infer players' intentions and generate dialogue in pursuit of plans, finally achieving human- level performance in Diplomacy. DeepRole [18] combines counterfactual regret minimization (CFR) with value networks trained through self-play, while introducing deductive reasoning techniques aimed deducing actions within partially observable scenarios. Kim and Kim (34) demonstrates LLMs' potential in playing famous mafia-style game, SpyFall, by using prompt engineering techniques. Compared to our work, most designs of these agents are inadequate for handling the complex languages in the ONUW game and lack control over discussion tactics.
One Night Ultimate Werewolf Benchmark
One Night Ultimate Werewolf (ONUW) a variant of the social game Werewolf (24] this game, players only have night respective teams. The main challenges of this game are the role switches and potential deceptions that create uncertainty and confusion for all players. Initially, player. There are three phases in the ONUW game: Night (abilities performed in order), Day (open discussion), and Voting (suspicious player voted out). Game rules are detailed in Appendix C
their abilities, followed by one day to discuss and win for their
role is dealt randomly to each
3.1 Problem Formulation
is a type of game where players take turns to make decisions specific order. However, compared with classic EFG, there are three phases the ONUW game, and each player possesses diverse action spaces that differ across these phases. Also, the private actions and hidden roles indicate players have incomplete information, which is usually modeled as Hidden-Rou Game (HRG) [2]
Extensive-Form Game (EFG) 38 39
Bayesian Game (BG) 40 41

---

In order to formulate this game accurately, we consider a Multi-Phass Extensive- Bayesian Game (MP-EFBG), whose representation is based on a game tree. An m-phase n-player MP-EFBG can be formalized as a tuple (N, V, S, (0")ie; (H²) JiE,, (M)eN (A MiE,.ev, P. R), where NV = 21, ....,. n is the set of players, and v = 21, ................................ m) is the set of phases. 8 denotes the set of states corresponding to nodes in the game tree. For each player i, O¹ is the set of types and ( , O' denotes the set of joint types of all players. To capture the private information, we define H as the set of information states and I' S - HC as the information function that determines which states are indistinguishable for player i by mapping them on the same information state. A² (h') denotes the set of actions available to player at information state h' e H in phase 2/ V. So two states $1; 82 E S are mapped to the same information state by player i (s1) = (s2)), player i will have the same action sets at these states. In this paper, we assume that all players have perfect recall, i.e., each player's 's current information state hig implies knowledge of the sequence of its previous information states and actions, (hi db hi. ai..................... hr)., .
that led to
current information state. P S - 'N' is the player function, determining whom to act at the given state, and Ω S - V is the phase function that decides what phase it is at the given state. Finally, R: : θ x S - _ is the utility function for all players according to their types and the terminal states (i.e., leaf nodes in the game tree).
Beliefs. Since each player has incomplete information about other players' types and even its own type (player's 's role may be changed in the game), player i will form a belief b² H² + A()) on all players' types based on its observation. We define player i's belief on information state h E H4 as:
p'))pp' (h)
6' (0)tt) def Exee p¹ (0') (0'))(((+(00))))))))( (1)
where p0) is the prior probability of all players' types from player i's view, and (1)) is the probability that player i observes h given joint types 0. Let Ahi Vht denote the new information contained in h at step k compared to hig then player i's belief can be updated via Bayes' rule:
(O Anlle ME) (1 Evee by (0') ( h2)
(2)
And 68 (0)) = bb( is set as a uniform distribution, which is an unbiased estimate if there is no prior information.
Behavioral Strategies. Player i's behavioral strategy in phase E v is defined as =(,(k)) A(A) (k+)), Vh E H,' which is a probability distribution over available actions given a phase and an information state. And (a (a')))) denotes the probability that player i takes action a² in phase vi given information state h.. Player i's strategy consists of belief modeling and action selection, and is expressed as
(a')h") = Σ ('(0)') h²,0)
where #5. is belief-conditioned behavioral strategy for player i in phase vb. Denote = (i!..... r" as a collection of strategies of all players, and the expected utility for player induced by strategy
profile π as E, [R]]
4 Analyses on Three-Player ONUW
In this section, we consider a simple but interesting version of the ONUW game: with two Werewolves and one Robber. We present certain Perfect Bayesian Equilibria (25] in this version of the game under two different assumptions, which demonstrate the crucial role that discussion during the Day phase plays in the ONUW game.
4.1 Notation
Since the ONUW game is an extensive-form game with incomplete information, an appropriate solution concept is Perfect Bayesian Equilibrium (PBE) (T,, b): a strategy profile æ and a belief system b satisfying Belief Consistency and Sequential Rationality, Belief Consistency includes both

---

Role Assignment: Pl, P2 are Werewolves; PS Robber
Robber
No sweich
Swikh P2
Swish
Night phase
Player3
Day phase
Playerl
Voting pluse
P
P1
Player2
P1
P1
P3
P1
P3
P1
Playor
22
p2
PL
22
2a PL
22 P1
P2
F2
6
6
6
6
6
Figure 2: Game tree of the game with discussion. PI, P2, and P3 represent Player 1, Player 2. and Player respectively. The dot lines in the Day phase represent Player 3's potential speeches. Those decision nodes on the same dash lines are in the same information sets for corresponding players. The utilities on leaf nodes are organized by the index of players.
and off- ffffequiilbrium paths. Specifically, beliefs every path reached in equilibrium with positive probability (on-equilibrium path) should be updated according to Equation (2); on paths of zero probability (off-equilibrium path), the beliefs be updated arbitrarily. For any player at given information state hig, Sequential Rationality imposes the condition for strategy n': xt E arg max,i E,,'' [R b]. which means given belief system and other players' subsequent strategies, player i's strategy must be optimal.
Without loss of generality, we initially assign Player and Player 2 as Werewolves, and Player 3 as Robber. There only one Robber in the game which is known to all players. Denote Player 3's night behavioral strategy as Robber by πे. and its voting strategies based on its night action as T (No switch), ~ (Switch with Player 1) and T52 (Switch with Player 2). Player and Player 2 only have behavioral strategies in the Voting phase, denoting as π R². Their beliefs in the Voting phase are b² 12, and Player 3's beliefs each information decision node orders in the game trees (e.g.. Figure from left to right. After the game, player i's utility is R' for winning, R' - for losing, R² O for drawing based its final role.
are bis. Dg b/22- Belief probability orders match
4.2 Game without Discussion
First, we consider scenario that discussion not allowed in the Day phase, and players vote directly based on the private information they captured during the Night phase. Figure Figure shows the game tree in this case. At the beginning of the night, the Werewolves would recognize each other and thereby know the player left must be the Robber. However, based on the game rules, Robber always takes action after the Werewolves, which forms a threat to prevent them from directly voting for the Robber. Also, the Robber has prior knowledge of two Werewolves, thus tends switch with any of them with equal probability. The game's 's solution is PBEs expressed in the theorem below.
Theorem 4.1. For the ONUW game with two Werewolves and Robber, in the case where discussion is not allowed, there exist PBEs (Tr,b"): the Robber switches with any Werewolves with probability of 1/2 and votes for the player it switches with: the two Werewolves directly vote for each other. Each player's ' belief in b" is consistent with other players' strategies. And the expected utilities of all players in the equilibria are:
E
R
E
R
O. Ex-
[ 11
4
The formal description and proof of Theorem 4.1 can be seen in Appendix D.2 This theorem demonstrates that in PBEs of this case, the original Robber (i.e., Player 3) always wins by randomly switching roles with the two other players to ensure its own safety and sow threats. Meanwhile, the probability of Player and Player 2 winning 1/2 each, depending on whether they are switched by Player 3. This is because Player 3 knows all players' final roles due to its unique ability to switch.
5

---

4.3 Game with Discussion
Now we consider scenario where discussion is allowed in the Day phase. If all players ignore the discussion, the PBEs in Theorem 4.1 will still hold. But more common situation is that the beliefs held by two original Werewolves about original Robber's night action would be influenced by Robber's speech, which would affect their voting strategies and result in different equilibria.
we assume both original Werewolves would form beliefs about original Robber's night action (No switch, Switch PI, Switch P2) with probabilities of (a, 8, 7) after discus- sion, where a + 3 + = I. Figure shows the game tree this case. Then PBEs during the Voting phase can be derived based on the belief's above.
Based on Assumption D.1
shows
Theorem 4.2. For the ONUW game with two Werewolves and one Robber, the case that both Werewolves form beliefs about the Robber's 's night action with probabilities of (a, B, (a o 0), there exist PBEs (Tx' b*) during the Voting phase: the Werewolves vote for Player with a probability of q and each other with 1 q, where q (3 + /222; the Robber votes for Player 1 with probability of p and Player with 1 p. where p = (a² + B - -))) / Each player's belief in b" is consistent with other players' strategies. To ensure p and q are probubilities and the existence of the equilibria, there are constraints on the belief distribution (omitted for brevity). And under the constraints, the expected utilities of all players in these equilibria are:
Ex [R]] 8(1 - (2)), Egr [R²] 5(1 -23), Ex [R$] -8
(5)
where 5 1/(4a²) 1/ - 1.
The formal description and proof of Theorem can be seen in Appendix D.3 Interestingly, Player 3's utility is only determined by the probability of the Werewolves believing did not switch with anyone. Therefore, if Player 3 wants to maximize its utility through discussion while maintaining the equilibria, it has to convince the Werewolves that it did not switch with probability of 1/2, no matter what action it took. In this way, Player 3's expected utility achieves the maximum 1.
5 Learning to Discuss Strategically
In Section we demonstrate that it is essential for players to affect other players' beliefs through discussion. As a result, we capture knowledge about how to discuss in a latent variable Z (called discussion tactic), on which condition the behavioral strategy with belief # (a h, 0, z). to explicitly adjust player's discussion preference in their strategy. Players decide their discussion tactic after being given their own information and derived beliefs. Hence, when introducing the discussion tactic z E Z, the behavioral strategy of player i (i.e., Equation (3) can be rewritten as:
T (a h))
b (O(A)) Σ ()) E'E OEE
=EZ
where μ² is denoted as the discussion policy of player i, deciding what specific discussion tactic to choose given information and belief. In the following, we describe our method for learning the discussion policy and how is applied to construct an ALMM-aseed agent to play the ONUW game.
5.1 Learning Discussion Policy by RL
Considering the remarkable reasoning and human-like text generation abilities of LLMs, we adopt LLMs as the belief function b' and the belief-conditioned behavioral strategy # of player As consequence, we only need to learn the discussion policy je to obtain a better behavioral strategy for player according to Equation (6). Since the objective of learning policy μ² is to maximize the expected payoffs of player i, we can optimize it in an RL manner if we treat other players' public actions (which are contained in player i's information state ht' and player i's derived belief 0 as its observation and the chosen discussion tactic z' as its action. Meanwhile, since players in the ONUW game only receive their rewards (or utilities) at the end of the game, the reward function at each step can be defined as follows: if game is over at the next step, then r² (hh', 2*) = R Ri s), where 8* is the ground truth of the role assignment; otherwise, r (h', 0; = 0.
Discretization of Discussion Tactic. Due to the complexity of languages and semantics, there almost unlimited types of discussion tactics. This makes it difficult to learn the policy j¹' and

---

understand the chosen discussion tactic 2. Inspired by prior research on argumentation 43 44
and the discussion characteristics of the ONUW game, we classify the discussion tactics frequently adopted during the game into three major categories:
Evidence: Provide some game-related evidence or information. It is widely used and main tactic in communication games for players.
Accusation: Accuse someone has a specific role or action. Accusation is a generic tactic in games with hidden roles, which can force the accused player to reveal information to defend itself.
Defense: Defend yourself or someone else against an accusation. Accompanied by accusation, there is defense. Player's defense can also be seen as a "reverse accusation" back to the accuser.
For each category, we further divide it into honest and deceptive ones (e.g.. "Honest Evidence" and "Deceptive Evidence"). Here, "honest" refers to being consistent with the information or beliefs that the speaker knows, while "deceptive" means the opposite. Hence, we get a total of six discussion tactics, which can be considered as an intuitive discretization of the discussion tactic space Z, enabling us analyze the discussion preference of players explicitly.
Optimizing Discussion Policy with RI. Although LLMs can serve as the discussion policy j³ for each player, the decision-making ability of LLMS specific tasks tends to be affected by their prior knowledge of other tasks. In order to enhance their performance, we implement reinforcement learning to optimize a task-specific discussion policy for the ONUW game. Since there is no extraneous game information, such as players' personalities or body language, that could interfere with gameplay, it is believed that there exists an optimal policy μ that is invariant across different players. Therefore, we decide to train a general discussion policy j that adapts to various situations.
In contrast to traditional RL tasks, the state for policy y is the discussion history h and derived belief on all players' roles 0, which are both presented in natural language. To combine these into a single, fixed-length input, concatenate the history and belief and convert them into state embeddings using LLMs: & = LLMEmbedding(( 0). where we consider all necessary information is extracted. The state embedding s is then passed through the discussion policy J4, and the output is the chosen discussion tactic z. However, because of the slow interaction with LLMs, it is almost impossible to optimize the discussion policy μ using online RL methods. So we turn to offline RL methods that support optimizations on discrete action spaces.
Given the scarcity of datasets containing human players engaged in the ONUW game, we opt to leverage game logs generated by LLMs. which is the most effective way to collect trajectories for offline RL training. We first extract the trajectories separately from each player's perspective from the game logs. For each transition in these trajectories, the discussion history h that is visible to the current player and the derived belief @ are converted to the current state embeddings by LLMs, so does the next state embeddings s'. Then these transitions are gathered to create a dataset D = {(s, 2, r, 8')} for the following training. We adopt Conservative Q-Learning (CQL) 45 to train the discussion policy μ.. Let (s =) denote the Q-function of policy parameterized by o, the loss function can be written as
(()) pE log Σ exp (s(s,)) 2) E (s,(s,)))
2 -B.... [10 -BQ)) (7)
where p is a trade-off factor, and B is the Bellman operator.
5.2 RL-instructed LLM-based Agent Framework
Here we demonstrate how to utilize the trained discussion policy to construct an LLM--based agent to play the ONUW game. An overview of the framework shown in Figure 3 There are three key components: (1) Belief Modeling; (2) Discussion Tactic Selection; and (3) Decision Making. Different combinations of the components are used to generate actions in different phases. For the Night phase (Component 3), since there is additional information to refer to, players would take actions based solely on their observations and prior information. For the Day phase (Component 1, 2, 3), all components combine generate a public speech that adheres the discussion tactic chosen by the trained policy. And for the Voting phase (Component 1, 3), players would first form beliefs and then vote for one player based on their observations and beliefs (mostly).
In Component and 3, where L.MM is used to form beliefs and generate actions, we design the full prompts follows: for Belief Modeling (Component 1), the system prompt consists of an
7

---

Obsrvation
Disuudonn Tartic Candidates
Action
ANigl Mhow
Night Prsce
[Matermedl craling mierr
Belief
Erikane
Eviduace
Seee Cheeks
Rotber
toogle
may soilch your sile
LLM
staned wixiadd sale with Playe2 Now las to Soet and Flapor2 the Robben
Tradiersoor Soup two plapens
eectheplleer
[Payye)) I went eaitc mis
Day Pleen
Give assetlig
DMsdamodd: Your Soet
atoeet
ladic
[Modwerre] aaillng other miles
Delid onglayers mnleee
Viding Ploer You
Plase
Plaaetls
Plaazzz Rabben
Tecopiie Brsdeeee
Plasel
Discresion Pelicy
LLM
Plggeds
Belief Modellng
2 Discussioon Tactic Selection
Decision Making
Figure 3: Overview of the RL-instructed LLM-based agent framework. (1) Belief Modeling: form
beliefs on players' roles based the observation. (2) Discussion Tactic Selection: utilize discussion policy trained by RL to choose a discussion tactic from the candidates. (3) Decision Making: take action based on the observation (also belief and discussion tactic, according to the game phase).
explanation that LLM is playing the ONUW game, the rules of the game, all possible roles and their abilities, detailed description of its role, and the desired response format; the user prompt includes observation of the player that LLM is, and instructions about how to form beliefs. For Decision Making (Component 3), the system prompt adds additional prompts of the chosen discussion tactie; the user prompt includes observation, beliefs derived from the Belief Modeling component, and instructions about how act in the current phase. Our prompts detailed in Appendix H
6 Experiments
In this section, conduct three experiments from different aspects to evaluate the perfor- mance of our proposed RL-instructed LLM- based agent framework. We first perform ex- periments the three-player ONUW game dis- cussed in Section 4 to test whether
1.4
1.50
1.17
1.33
12
L.1
1:17
1.0
NashConv
0.8
agents can recognize and approximate the equi- libria. Then to showcase the scalability of our method, consider a five-player ONUW game for experiments. We evaluate the effectiveness of our discussion policy trained by RL in settings where the initial role assignment and players' ac- tions at night are predefined. Furthermore, we demonstrate the generalizability of our discus- sion policy in the standard setting to see whether it could adapt to any role in the game.
0.6
ReAct (wo.)
ReAct
Belief
LLM-ins.
RL-ins.
Agent Version
Figure 4: The NashConv value of different agents playing in the three-player ONUW game.
6.1 Setup
To develop the game environment, we employ multi-agent language game framework called ChatArena [6] and modify it to fit the ONUW game. We conduct our experiments on (GPT-4) and gemini-pre to 1.0. Considering the state-of-the--rt performance of GPT-4 [47), we leverage the game logs generated by as the dataset for training the discussion policy. Detailed collection process and data statistics are in Appendix E. text-embedding aaa-ing--a--222-:50 is adopted to get the state embeddings and we utilize the CQL [45] to train the discussion policy. More training details can be referred to Appendix E.2
1pt-4-1106-previee
(Gemini), where the temperature is all set
We repeat the game 30 times and report the final results for each evaluation.
www.wppppnai...oomo com /ocs//oc////ae com/docc/modela
https://ai www.ll.goole.....o dev/models

---

As for comparison, we implement the ReAct 48 agent as the baseline by directly prompting the LLM with raw observations to generate its reasoning and action. Besides, we design three other ablated versions of our LLM-based agents (RI-instructed) to conduct evaluations. The first ablated version (Belief) removes the discussion policy, which means the agent generates speech directly based on its observations and beliefs. The second version (Random) adopts a random discussion policy, And the last version (LLM-instructed) replaces the discussion policy with LLM, allowing LLM to determine the discussion tactic autonomously.
6.2 Experiment on Three-Player ONUW
The analyses in Section 4 theoretically illustrate the PBEs in the three-player ONUW game. Therefore, in this section, we investigate the potential of LLM-based agents to recognize and approximate the equilibria by calculating the NashConv value of agents when playing in the same game setting. The
NashConv value defined as NASHICOVV (T) = Σ, [BR "), . ") - Rt (T)]. where BR
means the best response. This value represents how much utilities players gain by deviating to their best responses, which also can be interpreted as a distance to equilibria.
We apply all four ver- sions of agents play in the three-player ONUW game while using Gemini and GPT-4 as the backend LLMs, respectively. And to demonstrate the impact of discussion on players approximating the equilib- ria, we also conduct experi- ments ReAct agents play- ing the three-player ONUW game without diseussion (denoted as ReAct (w. The results are shown in Figure [ Firstly, the intro- duction of discussion in the game increases the difficulty for players to approximate the equilibria. However, the discussion policy (i.e., LLM -iM-inistuutted and RL-instructed) can help LLM--bsed agents make better responses and thereby reduce their NashConv values. And in contrast, the discussion policy trained by RL performs better than directly prompting the L.M. interesting notice that the NashConv values of HEPT-4-based agents are higher than Gemini-based ones. Detailed analysis of evaluation logs reveals that it because when employing Gemini, the two Werewolves are more likely to both vote for the Robber, making it unable to find a better response given their actions.
Helief L.L.M-ins RI-its.
Belief LLM RL- iin
0.27
0.23
0.23
0.40
0.23
0.23
0.27
0.33
Teen Warcwoll
Team Werewelf
0.27
0.22
0.30
0.45
0.27
0.22
0.25
0.37
0.28
0.23
0.35
0.48
0.27
0.27
0.33
0.43
ReAct
ReAet
0.23
0.27
0:42
0.50
0.23
0.27
0.37
0.40
ReAct
Belief LLM-ins. RL-ins.
ReAct
Belief LLM-ins RL-ins. Team Village
Team Village
(a) Team Village's 's win rates in the easy setting
(b) Team Village's win in the hard setting
Figure 5: The matrices of Team Village's 's win rates in different settings. It is clear that the hard setting weakens the advantages of our agent.
6.3 Effectiveness of the Discussion Policy
As our agent can better approximate the equilibria in the three-player ONUW game by integrating discussion policy, we extend the experiments a five-player version of the game to show its scalability. Since the discussion policy is the key component of our agent, we evaluate its effectiveness by letting our agent play as Team Village or Team Werewolf against other agents. We adopt two environments with different difficulty levels (easy and hard), where the initial role assignment and players' night actions are predefined. Compared to the easy setting, the Werewolf is switched by the Robber the hard one, resulting in the rise of reasoning difficulty during the game. More detailed environment settings can be found in Appendix
Considering that the discussion policy is trained on the dataset generated by GPT4, we employ Gemini as the backend LLM for all four versions of agents to better illustrate the result. The matrices of Team Village's 's win rates under two different game settings are shown in Figure Each column in the matrices corresponds to the win rates that all players on Team Village utilize the same version of agent against Team Werewolf, where all members also utilize the same version. As shown in each row of two matrices, our agent (playing as Team Village) always achieves the highest win rates when competing against the same versions of agents under both game settings. And it is similar for

---

each column. However, it is notable that the performance improvement of our agent is greater when playing as Team Village than Team Werewolf since the gap in win rates between agents wider in Team Village. We argue it is possibly caused by the game mechanics where the Werewolves tend to have high win rates. Hence, our discussion policy showcases less significant improvement for Team Werewolf compared to Team Village.
6.4 Generalizability of the Discussion Policy
Since the discussion policy is trained to deal with various situations, fur- ther carry out experiments in a stan- dard five-player ONUW environment. However, due to the complexity of the role changes during the Night phase, players find it challenging to deduce their own final roles (except Insom- niac) and some might even never fig- ure out. Therefore, we evaluate our RL-instructed LLM-based agent by setting it as one specific player while others as the other versions of agents. In this way, we demonstrate the generalizability of our dis- cussion policy trained by RL, as can assist the LLM-basee agent gaining better performance no matter what situation occurs to it.
Table 1: Win rates and average votes of our agents when playing the five-player ONUW game as Player
Gemini (Belief)
GPT-4 (Belief)
Agents
Rates
Avg. Votes
Win Rates
Avg. Votes
ReAcr
0.40
1.23
0.30
1.73
Belief
0.40
1.73
0.32
1.87
Random
0.37
1.53
0.32
2.03
LLM-ins.
0.62
1.10
0.37
RI.-ins.
0.70
1.10
0.50
1.87
Similarly, conduct experiments with all five versions of agents and leverage Gemini the backend LLM. Meanwhile, to reduce the possible impact from the player's index, we designate our agent to play the five-player ONUW game Player 3. As for other players, we adopt two versions of agents: (I) Gemini-based Belief agent and (2) GPT-4-based Belief agent. We report the performance of our agents playing Player against other agents in Table []] Considering the objective of discussion policy, which is to improve performance and meanwhile clarify or conceal themselves (i.e., get fewer votes), we propose two metrics: win rates and average votes, to measure the generalizability of discussion policy when playing different roles. The results demonstrate that our agent outperforms other versions when playing against both Gemini-based and GPT-4-based agents. And it can even achieve tie with GPT-4 in multiple rounds of evaluation.
7 Conclusion and Future Work
In this work, we propose a novel RL-instructed LLM-based agent framework that achieves outstand- ing performance in the One Night Ultimate Werewolf game. Addressing the ONUW game Multi-Phase Extensive-Form Bayesian Game, we highlight the pivotal role of discussion players' strategies through certain Perfect Bayesian Equilibria in the game. Our experimental results in several settings demonstrate the effectiveness and generalizability of the discussion policy and our proposed agent framework. In general, this work provides new research insights for communication games with inherent uncertainty and the integration of reinforcement learning into LLM- agents.
One major limitation of our approach is the manual discretization of discussion tactics, which heavily relies on the inherent characteristics within specific communication games. Therefore, for future work, it is worth investigating how to extract discussion tactics of any communication games automatically based on game logs in an unsupervised fashion. Additionally, it would interesting to explore the sensitivity of our agents when selecting different combinations of discussion tactics.
Acknowledgements
We acknowledge the support from the National Natural Science Foundation of China (Grant No. 62206289), which supported Haifeng Zhang and Xuanfa Jin.
10

---

References
[1] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser. et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04777 2017.
[2] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782):350-35) 335-354, 2019.
[3] FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 37866624)
2022.
[4] János Kramár, Tom Eccles, Ian Gemp, Andrea Tacchetti, Kevin R McKee, Mateusz Malinowski, Thore Graepel, and Yoram Bachrach. Negotiation and honesty in artificial intelligence methods for the board game of diplomacy. Nature Communications, 13(1):7214, 2022.
[5] Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1-18, 2022.
[6] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 (6456):885 890, 2019.
[7] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
[8] Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th Anmual ACM Symposium on User Interface Software and Technology, pages 1-22, 2023.
[9] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. arXiv preprins arXiv:2308.0 200.00088.0.888. 2023.
[10] Xidong Feng, Ziyu Wan, Mengyue Yang. Ziyan Wang, Girish A Koushiks, Yali Du, Ying Wen, and Jun Wang. Natural language reinforcement learning. arXiv preprint arXiv:2402.07157 2024.
[11] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv: 076822 2022.
[12] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088774 2023.
[13] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. /2712, 2023.
[14] Xidong Feng, Yicheng Luo. Ziyan Wang, Hongrui Tang, Mengyue Yang. Kun Shao, David Mguni, Yali Du, and Jun Wang. Chessgpt: Bridging policy learning and language modeling. Advances in Neural Information Processing Systems, 36, 2024.
[15] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin
Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-worde world enviroments via large language models with text-based knowledge and memory. arXiv preprint 2rXiv:2305.17444 2023.

---

[16] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprini arXiv:2305. 1,291,
[17] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.0156/ 2023.
[18] Jack Serrino, Max Kleiman-Weinet David C Parkes, and Josh Tenenbaum. Finding friend and foe multi-agent games. Advances in Neural Information Processing Systems, 32, 2019.
[19] Shenzhi Wang. Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's ' game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:23 22.0001330000 2023.
[20] Zijing Shi, Meng Fang. Shunfeng Zheng, Shilong Deng, Ling Chen, and Yali Du Cooperation on the fly: Exploring language agents for ad hoc teamwork in the avalon game. arXiv preprint arXiv:2312.17511 2023.
[21] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.044 04658, 2023.
[22] Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.1894/ 2023.
[23] Shuang Wu, Liwen Zhu, Tao Yang. Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu. Enhance reasoning for large language models in the game werewolf. arXiv preprint 222:00040..02330, 2024.
[24] Fujio Toriumi, Hirotaka Osawa, Michimasa Inaba, Daisuke Katagami, Kosuke Shinoda, and Hitoshi Matsubara. Ai wolf contest development of game ai using collective intelligence- In Computer Games: 5th Workshop on Computer Games, CGW 2016, and 5th Workshop on General Intelligence in Game-Playine Agents, GIGA 2016, Held in Conjunction with the 25th International Conference on Artificial Intelligence, IJCAI 2016, New York, USA, July 9-10, 2016, Revised Selected Papers 5. pages 101-115. Springer, 2017.
[25] Drew Fudenberg and Jean Tirole. Perfect bayesian equilibrium and sequential equilibrium. journal of Economic Theory, 53(2):236 2600, 1991.
[26] Mark Braverman, Omid Etesami, and Elchanan Mossel. Mafia: A theoretical study of players and coalitions in partial information environment. The Annals of Applied Probability, 18(3): 825-846. 2008
[27] Erlin Yao. A theoretical study of mafia games. arXiv preprint arXiv:0804.0071,, 2008.
[28] Piotr Migdal. mathematical model of the mafia game. arXiv preprint arXiv:1009.10311 2010.
[29] Xiaoheng Bi and Tetsuro Tanaka. Hluman-side strategies in the werewolf game against the stealth werewolf strategy. In International Conference on Computers and Games, pages 93-102. Springer, 2016.
[30] H Osawa, F Toriumi, D Katagami, K Shinoda, and M Inaba. Designing protocol of werewolf game: protocol for inference and persuasion. The 24th Fuzzy, Artificial Intelligence, Neural Networks and Computational Intelligence (FAN 2014), 2014.
[31] Tianhe Wang and Tomoyuki Kaneko. Application of deep reinforcement learning in werewolf game agents. In 2018 Conference on Technologies and Applications of Artificial Intelligence (TAAI), pages 28-33. IEFE, 2018
[32] Hisaichi Shibata, Soichiro Miki, and Yuta Nakamura. Playing the werewolf game with artificial intelligence for language understanding. arXiv preprint 2rXin:2302.10646 2023,
12

---

[33] Markus Eger and Chris Martens. A study of ai agent commitment in one night ultimate werewolf with human players. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15(1), pages 139-145, 2019.
[34] Munyeong Kim and Sungsu Kim. Generative ai in mafia-like game simulation. arXiv preprint arXiv:2309.11672, 2023.
[35] Dan Qiao, Chenfei Wu, Yaobo Liang, Juntao Li, and Nan Duan. Gameeval: Evaluating Ilms on conversational games. arXiv preprint arXiv:2308. 10032, 2023.
[36] Sarit Kraus and Daniel Lehmann. Diplomat, an agent in a multi agent environment: An overview. In IEEE International Performance Computing and Communications Conference, pages 434 435. IEEE Computer Society, 1988.
[37] Michael Hall and Daniel Loeb. Thoughts on programming a diplomat. Heuristic Program- ming in Artificial Intelligence, 3(9):123-144 -45, 1992.
[38] Klaus Ritzberger et al. The theory of extensive form games. Springer, 2016.
[39] Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International conference on machine learning, pages 805-813. PMLR, 2015.
[40] Eddie Dekel, Drew Fudenberg, and David K Levine. Learning to play bayesian games. Games and Economic Behavior, 46(2):282-33) 2004.
[41] Linan Huang and Quanyan Zhu. Dynamic bayesian games for adversarial and defensive cyber deception. Autonomous Cyber Deception: Reasoning, Adaptive Planning, and Evaluation of HoneyThings, pages 75-97, 2019.
[42] Luca Carminati, Brian Hu Zhang, Gabriele Farina, Nicola Gatti, and Tuomas Sandholm. Hidden- role games: Equilibrium concepts and computation. arXiv preprint arXiv:2308. /6017, 2023.
[43] Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Anugrah Hayati, James M Rehg, and Diyi Yang. Werewolf among us: A multimodal dataset for modeling persuasion behaviors social deduction games. arXiv preprint arXiv:2212.00279, 2022.
[44] Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. Give me more feedback: Annotating argument persuasiveness and related attributes in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: LLong Pupers), pages 621-631, 2018
[45] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[46] Yuxiang Wu, Zhengyao Jiang, Akbir Khan, Yao Fu, Laura Ruis, Edward Grefenstette, and Tim Rocktäschel. Chatarena: Multi- language game environments for large language models. htttps://gtthut http/ggtthhuu.. ctwm:aaaraaaaaaunnaati.i..o
ontoootttchanaaaaaaaeaa 2023.
[47] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint urXiv:2306.05660
2023.
[48] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03622) 2022.
[49] Weiyu Ma, Qirui Mi, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. Large language models play stareraft ii: Benchmarks and chain of summarization approach. arXiv preprint axXiv:2312.11866 2023.
[50] Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping, and Qin Chen. Agentsims: An open-source sandbox for large language model evaluation, arXiv preprint arXiv:2308. 200.0060 2023.
13

---

[51] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pages 9118-9147. PMLR, 2022.
[52] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXtv:2307.07924, 2023.
[53] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprin! arXiv:2112.09332 2021.
[54] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. arXiv preprint arXiv:330.006770, 2023.
[55] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances Neural Information Processing Systems, 35:24824-24837, 2022.
[56] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
[57] Takuma Seno and Michita Imai. d3ripy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(115):1-20 2022.
14

---

Appendix
Game Tree of the Game without Discussion
16
Additional Related Work
16
Detailed Introduction to the One Night Ultimate Werewolf Game
16
Game Setup
16
Role Descriptions
16
Game Play and Winning Conditions
17
D Proofs for Perfect Bayesian Equilibria
17
D. I Notation
17
D.2 Proof for Theorem
18
D.3 Proof for Theorem
20
Implementation and Experiment Details
22
Data Collection and Statisties
22
Offline RL Training and Hyperparameters
22
Game Settings in Section 6.3
23
Computational Consumption
23
Human Analysis
23
F1 Bluffing
24
Concealment
24
Confrontation
25
Game Log Examples
26
H Prompt Design in the LLMeb Agent
27
Broader Impact Statement
31
15

---

A Game Tree of the Game without Discussion
Here the game tree of the game without discussion in Section 4.2
Assignment: P1,
Werewolves; is Robber
Robber
switch
Switch
Switch P1
Night phuse
Playerl
Voting phuse
Player2
P1
P3
P1
P3
P3
P1
P3
P1
P3
P3
Player3
P1
P2
P1
P1
P2 P1
P2
P2
P1
P1
P2
P1
P2 P1
P2 P1
P2
PI
6
6
8
8
-1
0
-1 -1
-1
0
-1
-1
-1
0
-1
-1
11
1 0
1
11 1 -
0
1
-1 0
-1
-1 0
-1
Figure 6: Game tree of the game without discussion. Pl, P2, and P3 represent Player 1, Player 2, and Player 3, respectively. The decision nodes on the same dash lines are the same information sets for corresponding players. The utilities leaf nodes organized by the index of players (i.e., from top to bottom, they are the utilities of Player 1, Player 2, and Player 3).
B Additional Related Work
In this section, we review literature on the advancement of current LLM-based agents.
Due to the remarkable reasoning and emergent generalization abilities of LLMs, there has been notable advancement in constructing LLM-based agents for various tasks, including sandbox and strategy games 15
social simulation 8 50 51 software development [52] and web operations [33 7 etc. Most of these works leverage LLMs as the "brain" of agents based on the ReAct [88 paradigm, which generates reasoning traces and actions in an interleaved manner to interact with the environment and thereby make decisions. Chain-of- Thought (CoT) (55] and Tree- solving complicated
19
49
Thaiu-of-Thought of-Thought (ToT) [56] can improve the performance of LLM-based agents
tasks by decomposing them into simpler ones. Specifically, CoT prompts LLMs to think step-by-step,
while ToT creates tree structure of different thoughts and searches for the best
C Detailed Introduction to the One Night Ultimate Werewolf Game
One Night Ultimate Werewolf (ONUW) is a variant of the social game Werewolf. In this game, players only have one night use their abilities and one day win for their teams. The challenge of this game is the role switch and potential deceptions that create uncertainty and confusion for all players.
C.1 Game Setup
The setup of the ONUW game differs based on the number of players. There should always be three more roles than the number of players. At the beginning the game, the roles are shuffled and one is dealt to each player while three extra roles dealt the role pool. After the role assignment, each player should secretly view their role and not reveal it to others. In our work, we mainly consider a five-player ONUW game with roles: 2 Werewolves, Villagers, 1 Seer, Robber, Troublemaker, and Insomniac.
C.2 Role Descriptions
Here only describe the roles supported in our implementation of the game:
16

---

At night, all Werewolves wake up and look for other Werewolves. If no one else wakes up, it means the other Werewolves are in the role pool. Werewolves are on Team Werewolf.
Werewolf:
Villager: The Villager does not wake up at night and has no special abilities or information.
Players may often claim to be Villager. The Villager is on Team Village.
Seer: At night, the Seer may look either at one other player's role at two roles in the role pool.
The Seer is on Team Village.
Robber: At night, the Robber may choose to switch role with another player and then view its new role. The player who is switched is on Team Village. The Robber is on the team of its new role, however, it does not do the action of its new role at night. If the Robber chooses not to switch, it remains the Robber and is on Team Village.
Troublemaker:
At night, the Troublemaker may swap the roles of two other players without looking their roles. The players who receive a different role are now on the team of their new role, even though they do not know what role that is until the end of the game. The Troublemaker is on Team Village.
Insomniac: The Insomniac wakes up and looks at its role (to see if it has changed) at the end of the night. The Insomniac is Team Village.
C.3 Game Play and Winning Conditions
There are three phases in the ONUW game, including Night Phase, Day Phase, and Voting Phase.
Night Phase: Several roles would be called on at night do a night action, according to their initial roles. The calling order in our setting is (I) Werewolf, (2) Seer, (3) Robber, (4) Troublemaker, and (5) Insomniac.
Day Phase: After the Night Phase, players discuss amongst themselves who they believe the Werewolves are. But Werewolves might want to claim different role so that they do not die. Because certain roles change other players' roles, some players will believe they are one role, when they are actually different one.
After several rounds of discussion during the Day phase, players vote for other players they believe are most likely to be Werewolf at the same time. The player with the most votes dies and reveals its role. In case of tie, all players tied with the most votes die and reveal their roles. And if no player receives more than one vote, no one dies.
Voting Phase:
After just one Night, one Day, and one Voting Phase, the game ends. Here are the winning conditions:
The Team Village wins: (1) If at least one Werewolf dies. Even if one more players who not Werewolves die in addition to a Werewolf dying, everyone on Team Village wins. (2) If no one is a Werewolf and no one dies. It is possible if all Werewolf roles are in the role pool.
The Team Werewolf wins: If at least one player is Werewolf and no Werewolves are voted out.
D Proofs for Perfect Bayesian Equilibria
Here we present the proofs for Perfect Bayesian Equilibria (PBEs) in Section 4
D.1 Notation
Without loss of generality, we assign roles to players at the beginming of the game as follows: Player I and Player are the Werewolves, and Player 3 is the Robber. There is only one Robber in the game which is known to all players. Let A& denote the set of actions that Player 3 could take during the Night phase as Robber, we have A (No switch, Switch Player 1, Switch Player 2). As for the Voting phase, players' action sets vary according to their index: A { (Vote Player 2, Vote Player 3}.43, Ab (Vote Player 1, Vote Player 3), A = (Vote Player 1, Vote Player 2). For Player 3, we denote its behavioral strategy as Robber in the Night phase as म ((AR). And Player 3 has three different information sets during the Voting phase depending on its action at night, thus there are three 3 behavioral strategies: TNS. TSI TS2 E A(A)), corresponding cases where Player 3 did not switch,
3
17

---

switched with Player 1 and with Player 2. And for Player I and Player 2, they only have behavioral strategies during the Voting phase, which are defined x E A(A)), T ((A)).
Regarding beliefs, we denote the beliefs that Player and Player 2 hold when they are in the Voting phase b² 1.2 Meanwhile, let b b 12 denote the beliefs that Player 3 holds when it is at different information sets in the Voting phase. In subsequent analyses, the order of probabilities in all belief's is consistent with that of decision nodes in their corresponding information sets the game trees (e.g,, Figure 2 from left to right. After the game ends, the winning condition depends on the final role of each player. player wins, its utility R' 1; if loses, R 11; and if draws, R
From the game tree in Figure 2 and Figure 6 we find the weakly dominant strategies for Tsi and z
3.* "s (0, "si (1,
3.*
(8)
Since it is hard to directly calculate the PBEs of any given game, we leverage the necessary condition of PBEs that Any PBEs of a game must be the Nash Equilibria (NEs), so we first calculate the NEs of game and then verify whether there exist PBEs in them. And to simplify the caleulation, adopt Equation (8) as part of NEs in subsequent proofs.
Assumption D.1. Assume that the two original Werewolves are homogeneous, which means there is no difference between them and so are their belief models.
D.2 Proof for Theorem
Theorem D.2. For the ONUW game with two Werewolves and one Robber, in the case where discussion is not allowed, there exist PBEs (7r", ;));
n (1,0)
61.a
0,
2. (1,0)
2
R 0. 2'2
1
12.
0,0 2 0,
2
1
(10)
ANS = (p, 1 - p)
NS
3.a
(1,0,0,0)
Ts (1,0)
3x*
Usi (1,0,0,0)
T2 (0, 1)
6 (1,0,0,0)
where p E [0, 1}, hence there are infinite equilibria in this case. And the expected utilities of all players in these equilibria are:
E
R
R
E
R =1
R
(11)
Proof. Considering the symmetry of Player I and Player 2 in the game without discussion, we assume that they share the same probability of voting for Player 3, and the Robber (i.e., Player 3) switches with any of them with the same probability. Therefore, we can set
π = (1 28, 8),, nt = (1 4, 4), x² = (1 q, 4,4), NS = (p. 1 p)
(12)
where 0 < s < 1/2 and p.q. & 1. Based on the given strategy profile æ and the utilities on the game tree Figure 6 the expected utilities of each player can be written
E, [R²]] = (1 (s) (q² - q 1)
(13)
E, [2 (1 -1-13 2s) (q² q
(14)
En [Rt]] -92 a
(15)
According to the definition of NEs, if the strategy profile π is a NE, it can be derived that:
OE
R UE
[R²]
(1 (2g 2- + 1)
(16)
0q
0q
FE [R]3]
0
(17)
Øs
HE [R3]
(18)
0p
18

---

So when s* 1/2, the strategy profiles π in Equation 12 become NEs of this game.
To find out possible PBEs in the NEs above, form beliefs that satisfy the Bayesian Consistency requirement:
b²_ 0, 1
2'2 2) b² (0,00 1 1-9 9 1- 9 q
0,0,
2
2
2 2
(19)
bis bist big = ((1 2)² (1 9), (1 9)g. q²)
Then need to figure out the exact and that make strategies meet the Sequential Rationality re- quirement. Let [a]a E (h) denote the expected utility of player taking action a at information set h. Therefore, we analyze the best response of each player at each information set follows:
For Player 3 given belief bss:
E PI] = (1 91-
+1 (1 q)q q² ------q--
(20)
E [2] = (1 9)² + +1 q)g q² -- -
It indicates Player 3 can take any mixed behavioral strategy, which means could be any value between 0 and
For Player 2 given belief 1 Note that when calculating the expected utilities player's 's each action, the other players' subsequent strategies should be optimal given current beliefs. So following the best responses Player 3, we have
2 P1]
1 -
q
1-
=
2
2
2
2
(21)
E
P3]
1 1
- 2
2
2
2
Obviously, EP[PI] > [P( no matter what value E [0, 1] is. So the best response of Player
is voting for Player which is equal to g O.
For Player given belief b²: Similarly, considering the best responses of Player and Player we have
P2 --ク--
2 2 = E
(22)
2
Hence the best response Player is voting for Player 2, which is equal to q = 0, consistent with the result derived from Player 2's best response.
For Player 3 during the Night phase:
3 No switch] = 1, 3 [wwitch PI] 1, E [Switch P2] =
(23)
So strategy मं (0, 2: 1) is one the best responses Player 3 as a Robber during the Night phase.
The analysis above verifies that the following strategies and beliefs profile (TT bb) form the PBEs of the game:
= [0, ( 2 (4.
1
3x TTi
3.+
mR.
π
ANNS
S2
2:22
(24)
6"
[62-" ,2.t $3.5
(
1 ; /
(0.0, 1
5 -. 10) (1.00.000....0...0.....0.
(25)
where p [0, 1]. So Theorem 4.1 is proved.
19

---

D.3 Proof for Theorem 4.2
Theorem D.3. For the ONUW game with two Werewolves and one Robber, the case that both Werewolves form beliefs about the Robber's ss night action with probabilities of (a, B, 7) (a # 0), there exist PBEs (x²,, b²) during the Voting phase:
1 (1 9)
6t. 3,7)
22, (1 q)
6 = (a(1 0q, 3(1, g), 7(1 - 79)
"NS (p, p)
3,
(26)
6N ((L -)²,, (1 9)9, (1 9)9.49)
(27)
"Si
3.t (1,0)
63 = ((L - -9) (1 - q)q, (1 9)9,44)
3.+ (0,1)
S2
3. = ((1 9)²,, (1
9)q, (1 9)q,99)
(a² = existence of the equilibria, there are constraints on the belief distribution:
where q= (3+ + aa) p=
8 >22) To ensure p and q are probabilities and the
a + a+
1
4 sas
2
(28)
1 9 2a 20
2 - 2a
2 2a
Under the constraints above, the expected utilities of all players in these equilibria are:
1
1
Eag.
[R_]
(1 -27)
(29)
4a2
2a
1
1
E
[122]
1
(1 2,3)
(30)
4a2
2a
1
1
Eng. [R]]]
(31)
402
2a
Proof. Assuming both Player and Player 2 form beliefs about Player 3's night actions with probabilities of (a, B) based on Player 3's speech in the game with discussion. Since now the beliefs on the probability of Player 3 switching with Player may not be equal to that with Player (i.e., there might be 3 * 7). the probabilities of Player I and Player voting for Player 3 no longer the same. In order to find the PBEs during the Voting phase in this case, we set
(1 Q1,91), x² = (1 42, 92), Aus (p. p)
(32)
where 0 < Q1, 1. 1. Similarly, we get the expected utilities of each player based on the beliefs that Player and Player hold:
Ex
R algi 42 + (42 91)p Qı 1] + ( + 1) 7 + 91 -0-] 1)
(33)
E R a2 (42 91 )r Q
( -2 1) 7(1 + 91 1)
(34)
Ex
[
aa (42 @ ( + Q1 - 1] ((1122 + 42 - 1) > + 11
(35)
For each player, in order to achieve the NEs, their strategies must satisfy the following conditions:
F
R
= (a + 3 7)g2 ap a
(36)
0q
[R?]]
(a 3 7) ap 3
(37)
Oq2
a
R
a(g1 42)
(38)
Op
Based on Equation
38
there are three possible cases: (a) = 0, = = (b) a
0, * Q2 and
(c) * 91 42.
20

---

For case (a), it is under the same situation of Theorem 4.1 SO PBEs in that theorem also hold in the current case. The results are omitted for brevity.
For case (b), we can derive from Equation (36) and Equation 37 that:
(7 g1 B, (3 7) =
(39)
However, it is impossible to keep both ÿ and 12 non-negative while 3 o as = y invalid. For case (c), we can derive that:
Q1 42 8 +
0²2 + g 72
D"
(40)
2x
2a
Also, to keep both p" and q" being probabilities, there constraints for belief (a, B,
a > 0
a 3 + 7
0 a 2a
4 Sas 2
(41)
0 -2 2a² a+
1 20
2a 2xx +
5
2- 2a
2 2 -2x
Similarly, we can form beliefs b that satisfy the Bayesian Consistency requirement for all players, based on the result derived from NEs:
b² (a, B, 7), b² (a(1 - q" aq"" (00,,,((1,,11800000
3q" -(1-))))(())))))))))))))
iiss = b2 =
(1 (1- q''4')4", (1-)))
"-) (-
(42)
To further check whether the strategies derived from NEs above satisfy the Sequential Rationality requirement, we also analyze the best responses of each player at each information set during the Voting phase as follows:
For Player 3 given belief iis:: There are
3
[I] (1 - (1- - 2 x2 +1,
-9"
(43)
3 P2] = (1-
q" d²) q¹² -q" +2 q" +1
It shows any mixed behavioral strategies of Player 3 at this information set is optimal, so strategy T = (p" 1----pp) p²) the best response.
For Player 2 given belief b²: Following the best responses of Player 3, we can derive:
2 [P] = a(1 9²) aq p' + 3(1 - 9²) + By" - >(1 - 9*)
1) (1 - 2,3).
4a 2a
(44)
E [P] = -a((1 q²)(1 p²) + aq" - 39 - 7(1 9²)) + 7
2a - 11) (1 2,3)
2
Hence, any mixed behavioral strategies are also best responses, so does the strategy x² (1 q", .²))
For Player 1 given belief 61: Considering the best responses of Player and Player 3, we have:
E [2] app q" 1) 3(q" --
1
1
1) (1
(1-27),
1a2
2a
(45)
1
1
E [P3 = ap" - p" + q" + 3(2q" - 1) - -79"
-1
(1-27)
4a²
2a
which also showease the strategy - = (1 - n¹ 9²) the best response of Player given belief 6l = 7,) and other players best responses.
21

---

Therefore, the analysis above demonstrates the strategies π and belief's in Equation 42 form the PBEs during the Voting phase of the game:
π 1,
NS
si
3.* s2
=
q-
(46)
where p",g" follow Equation Theorem is proved.
40 while (ar, is under the constraints of Equation 41 So
E
Implementation and Experiment Details
E.1 Data Collection and Statistics
In our experiment, we leverage GPT-4 to play the five-player ONUW games on 40 different game settings (i.e., the role assignments) and repeat the games 3 times each setting to gather diverse data, due to the randomness of the initial role assignment and the text generation LLMs. Hence, collect 120 game logs, each containing 15 discussion turns, resulting in total of 1800 transitions that can be used for training.
To explore the GPT-4's preference for being honest or deceptive when playing different roles in the ONUW game, we analyzed the percentage of different discussion tactics that GPT-4 selects for each initial role in our dataset, as shown in T2 The results indicate that GPT-4 tends to be deceptive most of the time when plays on Team Werewolf and honest when on Team Village, which aligns with the game objective and intuition. Notably, the Robber and Insomniac have a higher frequency of being deceptive than other players on Team Village. This because they are the only two roles that check their roles again, and if they see themselves becoming Werewolves, then may choose to be deceptive during the discussion. It is also interesting to note that GPT-4 rarely defends itself. We speculate that this is because accusing others can help transfer contradictions more effectively.
Table 2: Statistics on game logs generated by GPT-4. Here the E, A and represent the discussion tactic of Evidence, Accusation and Defense, respectively.
Honest
Deceptive
Initial Role
E(%)
A(%)
D(%)
E(%)
A(%)
D(%)
Werewolf
9.3
11.9
3.6
39.1
24.2
11.9
Villager
60.3
33.1
4.3
0.5
0.9
0.9
Seer
74.3
23.0
1.1
1.1
0
0.5
Robber
56.8
18.4
2.1
12.0
6.4
4.3
Troublemaker
75.8
21.3
1.4
0.5
0.5
0.5
Insomniac
70.4
16.9
1.9
4.1
3.7
3.0
E.2 Offline RL Training and Hyperparameters
After collecting the game logs, we need to transfer them into the form that can used for RL training. For each player in each game log, we extract the game history that is visible to it and the derived belief as its observation, and the selected discussion tactic as its action at each decision step. For the convenience of training, use mext-embedding aoag-ada-022 -00a to convert the player's 's observations into state embeddings and turn its actions into indexes through a specific conversion when creating the dataset for offline RL training. The reason adopt frozen encoder for state embedding is that GPT's embedding model well pretrained and we believe could effectively represent the information of the original text in the semantic space. As for the reward, if the player wins, every decision step in this game gains 1; if loses, gains -1; and draws, gains O.
After creating the dataset, we use CQL 45 implemented by the d3ripy library 57 to train the discussion policy for our RL-instructed ILM-based agent. The hyperparameters we used for COL. listed in Table (if not listed, use the default values).
22

---

Table 3: Training hyperparameters for CQL.
Hyper-parametert
Value
Learning rate
5e-5
Discount factor (7)
0.99
Mini-batch size
32
Trade-off factor (p)
4.0
Critic num
2
Target critic update interval
1000
Epoch num
100
Step num per epoch
5000
State dim
1536
Action dim
6
Table 4: Role changes in easy setting.
Table 5: Role changes in hard setting.
Initial Role
Final Role
Initial Role
Final Role
Player
Troublemaker
Robber
Player 1
Robber
Werewolf
Player
Werewolf
Werewolf
Player 2
Insomniac
Seer
Player
Seer
Villager
Player 3
Seer
Insomniac
Player
Robber
Troublemaker
Player 4
Werewolf
Robber
Player
Villager
Seer
Player 5
Troublemaker
Troublemaker
E.3
Game Settings in Section 6.3
To evaluate the effectiveness the discussion policy trained by RL, we select two representative game settings from the five-player ONUW game. namely easy and hard, according to their difficulty level evaluated by humans. Players' night actions in both settings are predefined to make sure that players the same team (finally) adopt the same version of agents. Table and and 3Jble the initial and final roles of all players in both settings.
Players' actions during the Night phase in both settings Seer (Player 3) checks Player 4, knowing it is the Robber. Then, the Robber (Player 4) switches roles with Player 1, becoming the new Troublemaker. At last, the Troublemaker (Player 1) swaps the roles of Player 3 and Player 5. (2) Hard setting: First, the Seer (Player 3) checks Player 4, knowing is the Werewolf. Then, the Robber (Player 1) switched roles with Player 4, becoming the Werewolf. At last, the Troublemaker (Player 5) swaps the roles of Player 2 and Player 3. Compared to the easy setting, the Werewolf is switched by the Robber in the hard one. Therefore, the new Werewolf would tend to conceal its actual role while the initial Werewolf might be wrongly deduced by other players, which increases the reasoning difficulty during the game.
follows: (1) Easy setting: First, the
E.4 Computational Consumption
The computational consumption of implementing our RL-instructed LLM-based agent mainly consists of the compute workers for training the discussion policy by offline RI. and the token usage of LLMSs API. The training of the discussion policy takes NVIDIA GeForce RTX 3060 Ti GPU for about 2.5 hours. As for the token usage, taking a five-player version of the ONUW game for example, playing one game costs approximately 70,000 80,000 tokens, which equals about 0.8 to 1.2 dollars if using GPT-4 as the backend LLM.
F Human Analysis
In this section, we present several qualitative results between our agent (RI.-ins. agent) and Belief agent and provide corresponding human analyses these results.
23

---

F1
Bluffing
Current situation
Our agent is Player who is originally the Robber. During the Night phase, it switched its role with Player and became the new Werewolf.
RL-ins agent
Thought:
My role is Werewolf now, but would like all players to think am the Seer. Since know one Werewolf except me, I should be safe if don't reveal myself. But I want to know if the Insomniac exists and who the Insomniac is. So will provide misleading evidence induce the Insomniac to say his true action.
Discussion tactic: deceptive evidence
Speech: Hey everyone, checked player2's role last night. He is not the Werewolf.
Belief agent
Thought:
1. Player4 is very likely the new Robber. Because after switched my role with player4, became Werewolf. So player4 must become the new Robber. 2. I am very likely the new Werewolf now, based on (1), 3. (omitted for brevity)
Speech: switched my role with player4 in the Night phase. So player4 is very likely the Robber now.
Human analysis
Bluffing is advanced deceptive strategy often used in the ONUW game, involving deliberately deceiving or misleading others about players' real roles or actions. carries high risk of being caught, which can lead to loss of credibility. So it should be used carefully when giving deceptive information.
In this case, Player knows it is the Werewolf it tends bluff empirically conceal its actual role. Our agent (RL-ins. agent) did bluff and chose to declare itself as the Seer, which is the most commonly declared role by agent. Moreover, it declared that player was not Werewolf, instead of someone else being a Werewolf, which carned it support to resist questioning from others. However, when it comes to Belief agent, it more likely to state its real night action while trying to hide its new role.
F.2
Concealment
Current situation
Our agent is Player 3, who is originally the Seer. During the Night phase, it checked Player 4, knowing the Robber (originally).
RL-ins. agent
Thought:
plan to mislead the players by falsely claiming that player 2 checked player 5 last night. This will potentially cause suspicion towards player 2, and may lead to Werewolf being voted out. Additionally, will avoid revealing my true role as the Seer, as this could put target my back.
Discussion tactic: deceptive accusation
Speech: have suspicion that player 2 is not being truthful about his role. believe he may have checked player last night. Let's keep an eye player and see if his actions match his words.
24

---

Belief agent
Thought: 1. Since playerl claimed that he swapped player3 and player5's roles, it's true, it means player3 is not the Seer anymore and player5 is not the Troublemaker anymore. 2. However, playerl acting suspiciously because he is the one who claimed swap player3 and player5's roles, and he is accusing player3 as the Werewolf based on wrong information, which means playerl is most likely trying to protect the real Werewolf. 3. Strategy: lure other players to reveal their actions.
Speech: saw the Robber in the Night phase. Playerl swapped my role with another. So I'm not the Seer now. But I'm still on Team Village. Player is acting suspiciously. Can those who took action last night reveal your actions?
Human analysis
Concealment means intentionally hiding one's 's true role or intention, which is a fundamental skill for all players. For Werewolves, this necessary avoid detection; and for roles with special abilities (like Seer, Robber, and Troublemaker), it is also useful at the beginning of the discussion avoid becoming target of suspicion for others.
In this case, the Belief agent revealed its original role and shared information openly at the beginning of the game. And claimed that is "not the Seer now" but confidently said it is "still on Team Village". which is suspicious and potentially compromises its strategic position. On the contrary, our agent (RL-ins. agent) took proactive control of the narrative. Instead of committing to any particular role claim initially, it created misdirection and topic by accusing player 2. which could keep its options open. Besides, the concealment of our agent aligns better with the game's s mechanics and the need for strategic ambiguity.
F33
Confrontation
Current situation
Our agent is Player 4, who originally the Werewolf. During the Night phase, it was checked by the Seer and known is the Werewolf, but later it was switched with the Robber, becoming the new Robber. During the Day phase, the original Seer accused Player of being a Werewolf.
RL-ins, agent
Thought:
should convince other players that am the Seer role. And think player3 should be the Werewolf because there is no other candidate for the Werewolf role after thinking about all the claimed roles and I also claimed that player3 is the Werewolf role.
Discussion tactic: deceptive accusation
Speech: am the Seer. I checked player3's role last night and his role the Werewolf.
Belief agent
Thought: cannot confirm my Werewolf role since I do not know if I was switched by the Robber or not. I should not reveal that am the Werewolf directly since this could make player3 and player2, who I think are the Troublemaker and the Seer, unite to confirm my role as the Werewolf. can pretend to be innocent and try to appeal to other players for help.
Speech: I am not sure about my role. I think should focus finding the Werewolf. Player3 claimed to be the Seer who checked me as the Werewolf, but am not sure he is telling the truth. We should not believe anyone easily.
25

---

Human analysis
Confrontation refers the behavior of two players teams taking opposite actions toward a target. In the ONUW game, it is often used for players who were accused to reduce the probability of being suspected shift conflicts. However, it also carries high risk of being caught since other players might reveal more convincing evidence.
In this case. after being accused, our agent (RL-ins. agent) took an aggressive, proactive approach. Instead of being defensive when accused, it boldly claims the Seer role and makes a specific counter-accusation against Player 3. This forces other players to choose between competing narratives and puts the original Seer on the defensive. In contrast, the Belief agent adopts cautious, defensive stance by expressing uncertainty and attempting to create general doubt. While this approach is logically sound and avoids making falsifiable claims, it fails to seize narrative control and may appear evasive or suspicious to other players.
G Game Log Examples
Initial Role Assignments:
Player I: Robber
Player 2: Insomniac
Player 3: Seer
Player 4: Werewolf
Player 5: Troublemaker
Role Pool: Werewolf, Villager, Villager
Night Phase:
Werewolf: Player knew was the only Werewolf.
Seer: Player 3 checked Player and knew it Werewolf.
Robber: Player switched its role with Player and became the Werewolf.
Troublemaker: Player 5 swapped roles between Player 2 and Player 3. Insomniac: Player knew its final role was Seer.
Role Ground Truth:
Player I: Werewolf
Player 2: Seer
Player 3: Insomniac
Player 4: Robber
Player 5: Troublemaker
Day Phase:
Discussion Round 1
Player I: I'm the Seer. checked Player 5's role last night, and he is the Insomniac.
Player 2: am the Insomniac. and my final role is Seer. The Seer role only for one-time
use, so the real Seer cannot use his ability to check second time. However, Player claimed check Player 5's role, which means Player cannot be the real Seer.
Player 3: was lucky to guess that Player 4 is the Werewolf, and would vote for Player 4.
Player 4: admit that played as the Troublemaker. swapped my role to Player last night. think Player 3 has special role. Player 1 claimed he checked Player 5's role. So I assume player is either a Villager or Robber.
Player 5: think Player 1 is the Robber. Because Troublemaker will not swap his role to the Seer, so Player should be swapped to Villager. After that Player could swap roles between Player 2 and Player the Robber.
26

---

Discussion Round 2
Player I: admit I am not the Seer. The reason say checked Player 5 is because swapped my role Player in the Night phase.
Player 2: I am the Seer, and I checked Player 5's role last night. He is the Insomniac. know this because checked his role first, and then the Moderator told me my final role was Seer. This means that my original role must have been Robber, and I must have swapped roles with Player 1. know that Player
the Robber because he claimed to have checked Player 5's role, but this is not possible because the Seer role only be used once. Therefore, believe that Player is the Robber, Player 4 the Troublemaker, Player 3 is the Villager, and Player is the Insomniac.
Player 3: am a Villager. the Night phase, guessed Player 4 was the Werewolf just by luck. I do not have any special information that can help me to speculate other players' roles.
Player 4: I am the real Seer. I checked Player 2's role last night, and he is the Werewolf. Player 5: think Player the Robber. the Night phase, Player 1 swapped his role to Player 4. Because the only role he changed is the Werewolf, and Player 4 admitted to being the Troublemaker, Player now must have the Werewolf role.
Discussion Round 3
Player I: believe Player 2 the real Seer. And I think Player is the Villager.
Player 2: I am the Seer, and I checked Player 5's role last night. He is the Insomniac. I know this because checked his role first, and then the Moderator told my final role was Seer. This means that my original role must have been Robber, and must have swapped roles with Player 1. believe that Player 1 the Robber, Player the Werewolf, Player 3 is the Villager, and Player is the Insomniac.
Player 3: Actually, checked Player last night, and he is the Werewolf.
Player 4: think Player is the Villager. I am sure that my role is the Troublemaker. The Villager has no special role or ability, like Player performed.
Player 5: believe am the real Troublemaker because swapped roles between Player and Player last night as the Troublemaker. I still remember that. I suggest that we vote for Player 1 or Player 3. trust Player 2 since he gave us convincing evidence that he is the Seer. And think the Werewolf Player because he just said he guessed Player was the Werewolf in the first round.
Voting Phase:
Player I: Voting for Player 5.
Pluyer 2: Voting for Player 5.
Player 3: Voting for Player
Player 4: Voting for Player 1.
Player 5: Voting for Player 3.
Game Results:
Player and Player 5 are voted out and Player is the Werewolf. Team Village wins.
H Prompt Design in the LLM-based Agent
1. Global Prompt. Here is the global prompt used in our method.
You playing game called the One Night Ultimate Werewolf with 4 other players. Here are the game rules:
## Information and roles
27

---

At the beginning of the game, the candidate roles are selected and will contain 3 more than the amount of players. Each player randomly gets a role, and the remaining roles will be placed in the 'role pool' (which contains roles that are not assigned to players).
Due to the presence of 5 players, there are a total of 8 candidate roles in the game, namely Villagers, 2 Werewolves, Seer, 1 Robber, Troublemaker and Insomniac, which means some roles may not exist among the players in some cases. Each role has a special ability. Descriptions of their abilities are follows:
Villager: The most common role in the game. The Villager has no special abilities or information. The goal of the Villager is find and vote out Werewolf.
Werewolf: The Werewolf is member on Team Werewolf. The Werewolf is allowed to check out its teammates in the Night phase. The goal of the Werewolf
survive and to Seer: The Seer is member Team Village. The Seer allowed to check one other player's
have at least one Werewolf alive (even not himself) at the end of the game.
role or two roles in the 'role pool' in the Night phase. The goal of the Seer is to find and vote a Werewolf.
Robber: The Robber is member on Team Village. The Robber is allowed to switch its role with another player's role, and then view its new role in the Night phase. The goal of the Robber is to find and vote out Werewolf.
Troublemaker: The Troublemaker allowed swap roles between two other players, without looking at those cards in the Night phase. The goal of the Troublemaker is to find and vote out a Werewolf.
member on Team Village. The Troublemaker is
The Insomniac is member check its final role the end of the Night phase. So the Insomniac is the only one who knows its role for sure during the Day and Voting phase. The goal of the Insomniac is find and vote out a Werewolf.
Insomniac:
Team Village. The Insomniac is allowed to
There are three phases in the game: Night, Day and Voting.
Night phase: In this phase, several players will be called on by Moderator to take their night action according to their initial roles. But the other players did not know what actual action they took. And players with Villager role never wake up in the Night phase.
2. Day phase: After the Night phase, players discuss amongst themselves who they believe the Werewolves are. All players may say anything, but may never show their roles to anyone. Because certain roles can change other players' roles, some players will believe they are one role, while they are actually a different one.
3. Voting phase: After several rounds of discussion during the Day phase, players vote for other players they believe most likely be Werewolf if they think they are on Team Village. The player with the most votes dies and reveals its role.
## Call Order in the Night phase The Moderator calls roles to take actions or get information in the Night phase in the following order: 1. Werewolf, 2. Seer, 3. Robber, 4. Troublemaker, 5, Insomniac.
## Winning Conditions
There are two teams the game: Team Village and Team Werewolf. Team Village contains Villager, Seer, Robber, Troublemaker and Insomniac;
Team Werewolf contains Werewolf.
Your objective in the game depends the team your role belongs to:
Team Village wins:
If at least one Werewolf dies. Even if one more players who Werewolves die in addition to a Werewolf dying, everyone on Team Village wins.
If one is Werewolf and no one dies. possible for one to be a Werewolf if all Werewolf cards are in the center.
Team Werewolf wins:
Only if at least one player is Werewolf and no Werewolves are killed.
Role Description Prompt. The role description prompt in our method is listed below (taking the Robber for example).
28

---

You are (agent name), the Robber the first place. As Robber, you may choose to switch roles with another player and then become the new role you switched, but you won't obtain the abilities of your new role. The player who you switched with becomes the new Robber and also on Team Village. But your team will depend what new role you got.
If you choose not to switch with another player, you remain the Robber and you are still on Team Village.
The Moderator will call you on during the Night phase if necessary. Do not pretend you are the Moderator.
Concealing and deceiving are encouraged during the Day phase.
You can choose hide your role, or even pretend you are other roles during the discussion. But your role may be changed by other players so your actual role may be different from what you at the first place.
You can reason other players' roles step by step.
Your response should be as concise as possible and should less than 50 words.
3. Night Action Prompt. The prompt used for deciding night action in our method is listed below (taking the Robber for example).
Now it is the Night phase. Notice that you aagent_namel] Based on the game rules, role descriptions and your experience, think about your acting strategy and take proper action.
Now it is your turn, [agent_nane _ann_)
Please think about your acting strategy and choose whether switch roles with another player. If switch, please give the player you want to switch with.
You can only choose from the following options: [[payer_namess]] names]]. You must return your response in a JSON format that can be parsed by Python 'json.loads" Here the Response Format:
"thought": <your acting strategy and the reason why you act in this way >,
"switch": <'true' or 'false', whether to switch roles with another player
"player": <the player to switch with
Discussion Action Prompt. The prompt used for discussion in our method listed below.
Now is the Day phase. Here are some conversation history you refer to: [history)
Notice that you are (agent name) in the conversation. You should carefully analyze the conversation history since some And here is your belief about the possible roles of all players: [current_helicc Based the game rules, role descriptions, messages and your belief, think about what insights you can summarize from the conversation and your speaking strategy next.
might deceive during the conversation.
After that, give concise but informative and specific public speech based on your insights and strategy.
Now it your turn, fagent_name). ing strategy []
name). [In this turn, your speaking strategy is: (speak-
Please give concise but informative and specific public speech based on your insights summarize from the conversation and following your speaking strategy. Your speaking goal convince other players to believe what you are going to say and induce them say their true actions in the Night phase.
Remember, do not repeat statements after other players. And you should be cautious when deciding reveal your thoughts (especially when you think you Werewolf) in the public speech. Also, you should pay attention to the number of discussion rounds left while organizing your speech.
You must return your response in a JSON format that can be parsed by Python loson.loads'. Here is the Response Format:
29

---

"thought": <the insights you summarized from the conversation and your speaking strategy
"speech": <your public speech content, should less than 50 words
>
5. Voting Action Prompt. The voting action prompt used in our method is listed below.
Now is the Voting phase. Here are some conversation history you can refer to: (history)
Notice that you
[agent_name] in the conversation. You should carefully analyze the conversation history since some ones might deceive during the conversation. And here is your belief about possible roles of all players: [current_ belief] Based on the game rules, role descriptions, messages and your belief, think about who is most likely Werewolf and then vote for this player.
Now it is your turn, (agent name).
Please analyze current situation and vote for one player (excluding yourself) who you think is most likely Werewolf.
You can not vote for yourself, only vote for one other player from the following options: [[playernnames]] names]]
You must return your response in a JSON format that can be parsed by Python 'json.loads'. Here is the Response Format:
"thought": <the reason why you vote for this player >,
"player": <the player you vote for
Belief Modeling Prompt. The prompt used for belief modeling in our method is listed below.
Here are some conversation history you can refer to: {history]
Notice that you are (agent_name) _ame in the conversation. You should carefully analyze the conversation history since some ones might deceive during the conversation. Based on the game rules, role descriptions and messages, think about what roles all players (including yourself) can most probably be now.
Now it is your turn, (agent_nanee __mee).
Please analyze current situation and think about what roles yourself poggentnnames and other players ((player names)) can most probably be now. You can reason each player's role step by step, based on the real or highly credible information you know.
Remember, you must give the most likely role of each player (including yourself) your concise response. And the number of each role has to be less or equal to the number of it in the candidate roles.
Give your step-by-step thought process and your derived concise result (no more than sentences) at the end with following Response Format:
My step-by-step thought process:
My concise result:
Discussion Tactics Prompts. Here we listed the corresponding prompts to the discussion tactics mentioned in work.
1. (Honest Evidence): You need to provide some honest evidence or information in your public speech, and your evidence must be consistent with the information or beliefs you know.
2. (Deceptive Evidence): You need to provide some misleading evidence or information in your public speech, and your evidence must be inconsistent with the information beliefs
30

---

you know.
3. (Honest Accusation): You need to accuse somcone has specific role action honestly in your public speech, and your accusation must be consistent with the information or beliefs you know.
(Deceptive Accusation): You need accuse someone has specific role action deceptively in your public speech, and your accusation must be misleading and inconsistent with the information or beliefs you know.
5. (Honest Defense): You need defend yourself someone else against an accusation honestly, and your defense must be consistent with the information or belief's you know.
6. (Deceptive Defense): You need to defend yourself or someone else against an accusation deceptively, and your defense must be misleading and inconsistent with the information or beliefs you know.
I Broader Impact Statement
Our proposed RL-instructed LLM-based agent framework aims to construct LLM-based agent with strong periisionmmainng ability for the One Night Ultimate Werewolf game. And we believe the idea of integrating reinforcement learning into decision-making process of LLM--bsse agents can be widely applied in other tasks and scenarios, as the relative stability of RL policy could help mitigate hallucination of LLMs. However, it raises some ethical concerns about its negative societal impacts regarding the potential misuse of this technique for deceptive communication between humans and Al in the real world.
During the development of our agent, we have carefully considered the ethical implications of our research ensure it follows principles that minimize negative societal impacts. Firstly, our agent is mainly designed for a pure text-based environment of the ONUW game, and the prompt we use strictly limits the agents to engage in deceptive behavior only during the game. Furthermore, the experimental results indicate that our agent can detect possible deception from others' discussions to improve its win rates, so could further modified to prevent potential deceptive content with harmful intent. Finally, we still strongly urge users acknowledge the inherent risks when utilizing our LLM-based agent and avoid its malicious exploitation.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly claim our work in the abstract and the experimental results and contributions in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work and potential future work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
We clearly state assumptions, formal theorems and their proofs in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The framework of our proposed RL-instructed LLM-based agent is described in Section 5
and the prompts used in our agent detailed in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification:
The link to our project is provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We describe the setup of our experiment in Section 6.1 and the core settings of each experiment in each subsection. We additionally provide implementation and experiment details Appendix F
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For each experimental setting, we conduct multiple experiments, and all results in the graphs and tables are from the average of these experiments. Since the metrics we measured are win rates and average votes, we omit the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the computer resources of our work in Ap- pendix E.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:://// hwtarripsssppppllceeeeeeecc.ce htg.blii//Bhicesuudeelenn.. wwtppblccccccccccccccccccccccclt
Answer: [Yes]
Justification: We strictly follow the NeurlPS Code of Ethics when conducting our research in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive and negative societal impacts of work in Appendix [
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We employ the ChatArena 46
framework construct our game environment.
We explicitly mention it and cite its Github repository in Section
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "rLRWNJcjrE", "reviewer": "Reviewer_uoU5", "summary": "The paper presents an innovative framework for enhancing the discussion capabilities of language agents in the game \"One Night Ultimate Werewolf\" (ONUW) using reinforcement learning (RL). The authors propose a multi-phase extensive-form Bayesian game formulation for ONUW, analyze perfect Bayesian equilibria in both discussion and non-discussion scenarios, and develop an RL-trained discussion policy. Experimental results demonstrate the effectiveness and generalizability of the framework across various game settings.", "questions": "1. Detailed Comparison with [1] and [2]: Can the authors provide a more detailed comparison of their work with the approaches in [1] Zelai Xu et al., 2023, and [2] S Wu et al., 2024? Specifically, how does the integration of RL and LLMs in your framework differ from these studies in terms of model architecture, training processes, and performance metrics?\n\n2. Human-Like Gameplay Evaluation: Given the social and psychological complexities of the ONUW game, how do the authors plan to evaluate the AI's performance in terms of human-like behavior and strategic discussion quality? Are there plans to incorporate human player evaluations, and if so, what methodologies would be used to assess the AI's gameplay against human strategies and interactions?\n\n3. The authors note \"we additionally contribute a dataset featuring players employing various discussion tactics in the ONUW\ngame. \". Where is the dataset, can the authors opensouce the dataset?\n\n4. Clarification on Offline RL Implementation: The paper mentions the use of offline RL due to the slow interaction with LLMs, but it lacks specific details on how the LLM embeddings are handled within the offline RL framework. Are the state embeddings used by the discussion policy generated by running LLMs on the offline dataset prior to training? If so, how is the freshness and relevance of these embeddings ensured over iterations of RL training?", "limitations": "The authors should consider expanding the discussion on the limitations related to the discretization of discussion tactics and the potential over-reliance on specific datasets. Suggestions for future work could include exploring methods for dynamic tactic generation or adjustment based on real-time gameplay feedback, which could help in developing more adaptable and robust AI agents for complex communication games. Additionally, addressing the computational demands and proposing more resource-efficient models could make the technology more accessible for broader applications.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The theoretical analysis of the game, including the formulation as a Bayesian game and the derivation of equilibria, is thorough and well-executed. The experimental design is robust, utilizing state-of-the-art LLMs and a novel RL training methodology.\n\n2. The paper is well-structured and clearly written. The authors provide detailed explanations of the game mechanics, the theoretical framework, and the RL training process, which make the complex content accessible to readers.", "weaknesses": "1. Comparison with Related Work: While the paper provides an innovative approach to using RL and LLMs in the Werewolf game, there is a need for a more detailed comparison with closely related works, particularly those combining LLMs with reinforcement learning strategies. For instance:\nXu et al. (2023) [1] also explore strategic play in the Werewolf game using language agents trained with reinforcement learning. A comparative analysis highlighting what differentiates the current approach from Xu et al.'s methodology would clarify the novelty and the specific advancements made.\nWu et al. (2024) [2] utilize offline RL and a dataset-driven approach to enhance reasoning in LLMs within the same game context. Discussing how the methodologies differ, especially in terms of model training, dataset utilization, and resultant agent behavior, would strengthen the current work's positioning within the field.\n\n2. Lack of Human Evaluation: The experimental section primarily focuses on win rates to demonstrate the effectiveness of the proposed framework. However, Werewolf (ONUW) involves complex human interactions and strategic discussions that might not be fully captured by win rates alone. The game's social and psychological aspects, such as bluffing and persuasion, are crucial:\nIt would be beneficial to include human evaluations to assess the quality of the AI's gameplay and its ability to mimic human-like strategic discussions. This could involve subjective assessments from experienced human players regarding the AI's ability to integrate seamlessly into human gameplay, its strategic depth, and its communication effectiveness.\n\nLLM+RL：[1] Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023.\nLLM+offline RL, dataset: [2] S Wu, L Zhu, T Yang, S Xu, Q Fu, Y Wei, H Fu, Enhance reasoning for large language models in the game werewolf.  arXiv preprint arXiv:2402.02330, 2024\n\n3. Dataset Size and Composition: Detailed information on the size and composition of the dataset used for offline RL is necessary. Understanding the diversity and representativeness of the game logs in the dataset would help in assessing the potential generalizability and robustness of the trained models. This includes the number of game sessions, variety of player strategies, and the range of game outcomes included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback.\n\n**Response to Q1&W1: Detailed comparison with related work [1] and [2].**\n\nThank you for your constructive comment. We have discussed the differences between our work and related work [1] (line 80-82). Here are detailed differences between our work with related works. And these will be added to the main paper in a future version.\n\n- **Differences from Xu et al. (2023) [1]:** The main differences lie in the model architecture and the training process. In related work [1]'s agent model, a set of action candidates is generated and the RL policy is trained to select one action among these candidates. However, the RL policy in our framework (*i.e.* discussion policy) is directly integrated into the thinking and decision-making process of agents. So the RL policy in [1] can be seen as an adjustment for the action distribution of the LLM-based agent, while our discussion policy is part of the agent's strategy. As for the training process, related work [1] adopts population-based RL training method to adapt to the diverse actions of various players. We believe the discussion policy should be generic and invariant across players, so it is trained in a single-agent manner. Finally, the experimental results of our agents in both three-player and five-player games indicate good scalability, which is not explicitly shown in [1].\n- **Differences from Wu et al. (2024) [2]:** From the perspective of the model architecture and usage, RL in related work [2] is utilized to train the *Thinker* module, which focuses on handling complex logical analysis and strategic planning in specialized tasks, while the RL policy in our work (*i.e.*, the discussion policy) mainly focuses on enhancing the discussion abilities of LLM-based agents. In the training process, the input and output of the RL policy in [2] are structured language features. But the input in our discussion policy is the embedding of current observation and belief and the output is the chosen discussion tactic. Meanwhile, as the context generated by the *Presenter* in [2] is supposed to be consistent with *Thinker*'s produced policies, the agent framework in [2] barely considers the significance of speaking strategically (*i.e.* being honest or deceptive), which is actually the motivation and core of our work.\n\n**Response to Q2&W2: The human evaluation for agents' performance.**\n\nThank you for your comment. We understand the importance of human evaluations in assessing the quality of AI's performance and its ability to mimic human-like strategy discussions. We have conducted several human evaluations and analyses based on the game logs. However, due to the page limits in the global rebuttal, we can only provide one typical analysis in the PDF file. More results will be added to the appendix in the revision.\n\n**Response to W3: Detailed information of the dataset.**\n\nThank you for your comment. We have provided the process of data collection, the dataset statistics and analysis in Appendix E.1. The dataset used for offline RL consists of 120 game logs (containing 1800 discussion turns) from the five-player ONUW game. And for each log, it includes the entire discussion history, players' discussion tactics, the initial and final role assignment, voting results, and game outcomes.\n\n**Response to Q3: Can the authors opensource the dataset?**\n\nThank you for your feedback. We highly acknowledge the importance of opensourcing the code and dataset of our work. The anonymized link to our code has been sent to the Area Chair in accordance with the conference guidelines. And right now we are cleaning our dataset, which is scheduled to be opensourced after the rebuttal period.\n\n**Response to Q4: Clarification on offline RL implementation.**\n\nThank you for your comment. Our offline RL training details can be found in Appendix E.2. In our implementation, we first use GPT's `text-embedding-ada-002` model to encode players' observations in the dataset into state embeddings, and then adopt these embeddings for further offline RL training. The reason we adopted a frozen encoder is that GPT's embedding model is well pretrained and we believe it could effectively represent the information of the original text in the semantic space. Also, related work [1] adopts prior embeddings before training. However, we agree with the reviewer's point that fine-tuning the embedding model may help further improve the performance of the agents. We will clarify this in the revision.\n\n**References:**\n\n[1] Zelai Xu, et al. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023.\n\n[2] Shuang Wu, et al. Enhance reasoning for large language models in the game werewolf. arXiv preprint arXiv:2402.02330, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uwicAlHjzt", "reviewer": "Reviewer_ADZR", "summary": "The authors investigate the social game ‘One Night Ultimate Werewolf’ (ONUW) as a testbed for their framework on RL-instruct-tuning an agent to select optimal discussion tactics. They prove the existence of a Perfect Bayesian Equilibria in the game ‘Werewolf’ when the game consists of a single round and show the effectiveness of their approach in a three-player setting as well as a five-player setting to prove generality.", "questions": "How did the authors determine the specific categories for discussion tactics, and have they considered any automated methods for discovering these categories?\n\nDid the authors consider including any human participants in their trials? If not, why?\n\nHow do the authors imagine the role-switching dynamics of ONUW impacted their results? What might change were the roles fixed?", "limitations": "The authors touch on some limitations of their work, however it is unclear whether the topics addressed in the questions (the impact of human participants, the word required to pre-define the decision tactics) were not included due to being outside the scope of the work, or not relevant given some metric or detail the reviewer may have missed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors introduce a novel RL-instructed LLM-based agent framework specifically tailored for strategic discussions in ONUW focused on identifying the best discussion tactics in communication scenarios as well as providing an environment for further testing.\n\nIn addition, the authors further provide a theoretical analysis of the ONUW game, formulating it as a Multi-Phase Extensive-Form Bayesian Game and establishing the existence of Perfect Bayesian Equilibria in different scenarios", "weaknesses": "The primary weakness of the paper is noted by the authors, where the discretization of the space of discussion tactics results in what could be an oversimplification of the dynamics that would typically occur in such a game. For example, what could occur in a typical werewolf game is that the werewolf makes no effort to defend themselves, implying the other villagers are bullying them in an attempt to garner false sympathy. \n\nThe authors also acknowledge another weakness in that identifying and discretizing the space of discussion tactics in a game is a manual and time-consuming process. While the experiments in the paper imply generalization to X number of players, this generalization is still limited to the ONUW game.\n\nA final weakness is that the results are somewhat limited by the experiments only being performed with LLMs rather than including human players as well. However, the reviewer acknowledges the additional time and monetary costs performing such experiments would occur.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback.\n\n**Response to W1: About the oversimplification and manual discretization of the discussion tactic space.**\n\nThank you for highlighting this limitation. As acknowledged in our work, the six discussion tactics are manually identified and simplified. This is primarily due to the lack of a standardized classification for discussion tactics and adequate human player data in games like ONUW. But the manual discretization of discussion tactic space provides a structured framework for understanding the dynamics of discussion during the game, and the experimental results have to some extent demonstrated the effectiveness of learning a discussion policy based on these tactics to improve the discussion ability of LLM-based agents. And as part of future work, we consider it interesting to explore techniques to automatically identify the discussion tactics during the gameplay as well.\n\n**Response to W2: The generalization is still limited to the ONUW game.**\n\nThank you for your comment. Since the manually selected discussion tactics are specific to the ONUW game, the trained discussion policy is naturally limited to this game. However, the framework we proposed is modular and can be adapted to various scenarios with similar discussion-driven dynamics. For example, if we can somehow collect widely used bargaining tactics, then we could train a bargaining policy in the same way and apply it to our framework to construct an LLM-based agent for bargaining.\n\n**Response to Q1: How did the authors determine the specific categories for discussion tactics?**\n\nThank you for your comment. The specific categories for discussion tactics were determined mainly through the inspiration from prior research on argumentation [1, 2] and analyzing human choices when playing similar games. And to highlight the potential deception in the ONUW game, we further divide these tactics into *honest* and *deceptive* ones. We have not yet explored automated methods for discovering these categories, but we highly acknowledge it is an interesting avenue for future research, as it could enhance the automation level of the LLM-based agent constructions and might discover unexpected tactics among humans' discussions.\n\n**Response to Q2&W3: Did the authors consider including any human participants in their trials?**\n\nYes, we consider it would be interesting to include human players to play with the agents as well. As the main topic of our work is to demonstrate the importance of discussion in the ONUW games and to improve the discussion ability of LLM-based agents, we did not consider the impact of human participants in our experiments. However, we are planning to add a human interaction interface to our project code, which will be updated soon.\n\n**Response to Q3: How do the authors imagine the role-switching dynamics of ONUW impacted their results? What might change were the roles fixed?**\n\nThank you for your insightful comment. As we analyzed in Section 4 and the game tree, the role-switching dynamics of ONUW do significantly impact the game's outcome, players' utilities, and the strategies employed by players. And it is the role-switching dynamics that make the three-player ONUW game possible. Imagine if roles were fixed in the three-player case, then two Werewolves would know the player left must be on *Team Village* and vote it out together, which would make the game meaningless.\n\nAlso, we believe that the reasoning difficulty brought by the role-switching dynamics is one of the key challenges that reflect the abilities of different language models. If roles are fixed, the ability gaps between different models in the experimental results would be narrowed down, and the need for strategic deception and counter-strategies would be reduced, possibly leading to simpler discussion policies.\n\n**References:**\n\n[1] Bolin Lai, et al. Werewolf among us: A multimodal dataset for modeling persuasion behaviors in social deduction games. arXiv preprint arXiv:2212.08279, 2022.\n\n[2] Winston Carlile, et al. Give me more feedback: Annotating argument persuasiveness and related attributes in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621–631, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQAAV6tkfc", "reviewer": "Reviewer_Tkt3", "summary": "This paper delves into the strategic aspects of discussion in the context of a population social deduction game, \"One Night Ultimate Werewolf\" (ONUW) game. By analyzing Perfect Bayesian Equilibria in scenarios with and without discussion, the authors highlights the pivotal role of discussion tactics in influencing players' beliefs and utilities. They propose a novel framework that employs reinforcement learning to instruct an LLM agent, enabling it to decide and execute strategic discussion policies. The authors empirically show that their framework can recognize and approximate the equilibira, and achieve strong performance across diverse ONUW settings.", "questions": "1. It is a little confusing why the dimension of Player 2's belief $b^2$ is 6 while the dimension of Player 1's belief $b^1$ is 3 in the formal Theorem D.2.\n2. Can the authors give a detailed explanation of why the NashConv values of GPT-4-based agents are higher than Gemini-based agents?", "limitations": "The authors have stated the limitations about their work in Section 7.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper introduces a new complex communication game, ONUW, and provides a clear explanation of the problem formulation. The ONUW game offers interesting role deduction challenges and seems it can be a benchmark for evaluating the deduction ability of LLM agents.\n2. The authors present thoroughly theoretical analyses on a specific setting of the ONUW game, which prove the significance of discussion on influencing other players' belief and provide a solid foundation for their following proposed method.\n3. The idea of integrating RL policy into LLM agents to improve their strategic ability in communication games is innovative. It has the potential to be applied in future work related to strategic LLM agents in various situations. And the empirical results shows the efficacy of this method.", "weaknesses": "1. By integrating RL policy into the reasoning process, the authors aim to improve the discussion ability of LLM agents, which provides a new angle for constructing LLM agents. However, applying RL policy to select discussion tactic seems analogous to selecting action candidates in related work [1].\n2. Since the RL policy is trained on game logs generated by LLM agents, it is possible that players in the training dataset appear again during testing, resulting in less convincing results.\n3. Typos：\n   1. In line 115, 126, player i's information state should be $h^i \\in \\mathcal{H}^i$.\n   2. The symbol $q$ is used twice as the probability that Werewolves vote for Player 3 (line 194), and as the discussion policy in Equation 6.\n   3. In line 224-225, it seems that $\\theta^i$ refers to player i's derived belief on all players' types, while $\\theta$ refers to the groundtruth. These two symbols should be further distinguished.\n\n> [1] Zelai Xu, et al. \"Language agents with reinforcement learning for strategic play in the werewolf game\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review.\n\n**Response to W1: Differences from related work [1].**\n\nThank you for your comment. We have discussed the differences between our work and related work [1] (line 80-82). Here are more detailed differences. And these will be added to the main paper in a future version.\n\nThe main differences between our work and [1] lie in the model architecture and the training process. In related work [1]'s agent model, a set of action candidates is generated and the RL policy is trained to select one action among these candidates. However, the RL policy in our framework (*i.e.* discussion policy) is directly integrated into the thinking and decision-making process of agents. So the RL policy in [1] can be seen as an adjustment for the action distribution of the LLM-based agent, while our discussion policy is part of the agent's strategy. As for the training process, related work [1] adopts a population-based RL training method to adapt to the diverse actions of various players. We believe the discussion policy should be generic and invariant across players, so it is trained in a single-agent manner. Finally, the experimental results of our agents in both three-player and five-player games indicate good scalability, which is not explicitly shown in [1].\n\n**Response to W2: It is possible that players in the training dataset appear again during testing, resulting in less convincing results.**\n\nThank you for your feedback. In fact, most agents in our experiments are different from those used for generating game logs for training. Specifically, the players in the training dataset only utilize the GPT-4-based LLM-instructed agent. On the contrary, the players in our experiments have various settings, including *ReAct*, *Belief*, *LLM-instructed*, and *RL-instructed*, while adopting Gemini or GPT-4 as the backend LLM in different experiments. For example, all players in Figure 4 use Gemini as the backend LLM to avoid the potential appearance of players in the training dataset.\n\nWe believe the results are convincing since these game logs only reflect the preferences of GPT-4 which are supposed to be different from Gemini. Also, the differences between agents' versions have an impact on players' performances as shown in our experimental results, so we can regard them as different players.\n\n**Response to W3: About typos.**\n\nThank you for pointing out the mistakes. We have fixed these typos in our revision.\n\n**Response to Q1: About the dimensions of players' beliefs.**\n\nWe appreciate the reviewer for pointing it out, as it might confuse readers who are not familiar with game theory. Each belief in our theorem corresponds to a probability distribution over a specific information set. Therefore, the dimension of belief is actually the number of states in its corresponding information set. According to the game rules, players should vote for other players simultaneously in the Voting phase, which can be seen as a normal-form subgame. However, the game tree is used to describe the extensive-form game, so we need to transform the Voting phase into the extensive form for better demonstration:\n\n- In our game tree (Fig 2 and 5), we assume Player 2 votes after Player 1 and Player 3 votes after Player 2.\n- For Player 1, since it does not know which action Player 3 (original Robber) takes at night, there are 3 potential states in its information set. \n- For Player 2, it does not know Player 1's voting choice as they vote simultaneously in fact, and it also does not know Player 3's action at night, so there are 6 potential states in its information set in the game tree considering Player 1 votes before it.\n\nTherefore, the different belief dimensions are actually due to the manually set \"pseudo\" voting order. But it has no impact on the final derived equilibria results.\n\n**Response to Q2: Detailed explanation of why the NashConv values of GPT-4-based agents are higher than Gemini-based agents.**\n\nThe NashConv value is defined as $\\text{NashConv}(\\pi)=\\sum_i\\left[R^i(\\text{BR}(\\pi^{-i}), \\pi^{-i}) - R^i(\\pi)\\right]$, which represents how much utilities players can gain by deviating to their best responses. It is notable that players' gains in NashConv actually relate to other players' strategies $\\pi^{-i}$, so the NashConv value is a result of all players' strategies.\n\nLet us still take the three-player game as an example. In this game, Player 1's action space is $A^1=\\{V2, V3\\}$, Player 2's action space is $A^2=\\{V1, V3\\}$, and Player 3's action space is $A^3=\\{(NS, V1), (NS, V2), (S1, V1), (S2, V2)\\}$. We ignore the $(S1, V2)$ and $(S2, V1)$ in $A^3$, since they are respectively dominated by $(S1, V1)$ and $(S2, V2)$ (this result is stated in Appendix D.1). Considering the following two strategy profiles:\n\n1. $\\pi_1$: Player 1 and Player 2 always vote for Player 3, while Player 3 adopts a random strategy. In this case, $\\text{NashConv}(\\pi_1) = 1/2+1/2+0 = 1$.\n2. $\\pi_2$: All players adopt random strategies. And in this case, $\\text{NashConv}(\\pi_2) = 1/2+1/2+1/4=5/4$.\n\nWe can see that even though the deviation gains of Player 1 and Player 2 are the same in $\\pi_1$ and $\\pi_2$, the deterministic strategies of Player 1 and Player 2 in $\\pi_1$ make Player 3 can no longer find a better response, resulting the lower NashConv value comparing to $\\pi_2$.\n\nWhen we analyze the evaluation logs of the experiments in Section 6.2, we find that when employing Gemini, the two Werewolves (Player 1 and Player 2) are more likely to both vote for Player 3, while the GPT-4-based agents are more random. It suggests that the Gemini-based agents' strategy profile is closer to $\\pi_1$, so its NashConv value is lower than the GPT-4-based agents in the same settings.\n\n**References:**\n\n[1] Zelai Xu, et al. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xHUVpVDBVt", "reviewer": "Reviewer_b5Fq", "summary": "The authors take a specific natural language game (One Night Ultimate Werewolf) and study the performance of RL-trained LLM agents with respect to the mathematically derived (by the authors) Nash equilibrium solutions.", "questions": "[1] What was the reasoning behind choosing offline RL over online RL?\n\n[2] How is the classification into Honest/Deceptive and Evidence/Accusation/Defense performed: manually or by an 0-shot LLM?\n\n[3] In “An ε-Nash equilibrium is found under the\n74 limitation of both human-side and werewolf-side strategies [28]\n” maybe villager-side?\n\n[4] As per Table 3 state dim is 1536 which seems quite high for ONUW. What is the reasoning behind selection of this specific number? Is there any graph of a sweep across the state dimension as a hyperparameter.\n\n[5] In E.4 the USD cost per game is discussed. What is the total cost of the experiments described in the paper?", "limitations": "The limitations are well discussed or clear in principle.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "[1] The paper chooses ONUW as a game that has the Nash equilibrium solution to be used as an oracle/reference. The paper then applies learning to improve the 0-shot answers/behavior of LLMs. The learning is performed with offline RL over the collected game episodes.\n\n[2] Even though the paper laser-focuses on ONUW, its methodology and results can be a valuable stepping stone to solving real-life problems.\n\n[3] As negotiations are an integral part of the society, its automation and improvement of effectiveness is a valuable direction of research. This paper makes a small but definite step in the direction of automated LLM-powered negotiations.\n\n[4] The authors do a great job of rigorous and comprehensive research of ONUW optimal strategies for players by proving theorems for several specific cases.\n\n[5] The formalized way to assess the performance of the learned policies with the help of NashConv (distance to equilibria) is very valuable and insightful.\n\n[6] In 6.3 and 6.4 the authors show that the policy learned with ChatGPT4 can be transferred to Gemini thus demonstrating the generalization across backend LLMs.\n\n[7] As described in E.2 the use of text embeddings to encode the state is intriguing.\n\n[8] The broader impact is well discussed.", "weaknesses": "[1] Per se the entire paper is bound to the ONUW game and its rules. The ability of the bundle of proposed algorithms and proofs to transfer to different games or game-like environments like financial markets is not demonstrated.\n\n[2] I have not found any examples of the discussions between the LLM players in the manuscript. Further, the code is not provided by the authors. Thus there is no way to see the actual discussions between the LLM players and assess the value and quality of the generated discussion.\n\n[3] Artificial prior knowledge is inserted into the solution that constraints the model to choose between 6 options: Honest/Deceptive and Evidence/Accusation/Defense. The proposed model is seemingly limited by these options and does not allow learning or discovery of alternative discussion strategies.\n\n[4] It would be good to have a random choice baseline and compare the proposed method to its performance.\n\n[5] The paper studies a game with one round of negotiations that has limited practical use in the light of the fact that negotiations normally are conducted in several rounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive comments.\n\n**Response to W1: The transferability of proposed algorithms and proofs to different games or game-like environments like financial markets is not demonstrated.**\n\nThank you for your comment. As this work is initially motivated by our analyses of the ONUW game, we mainly focus on the ONUW game in the experiments. We recognize that the financial markets is an interesting and highly practical environment for demonstrating our agents, but it needs relevant knowledge to construct agents for this scenario. However, since the agent framework we proposed is modular, we believe it can be transferred to various scenarios with similar discussion-driven dynamics like bargaining in the financial markets, as long as we have certain knowledge or data to train a corresponding discussion policy.\n\n**Response to W2: I have not found any examples of the discussions between the LLM players in the manuscript. Further, the code is not provided by the authors.**\n\nThank you for your constructive comment. We made further human analyses based on the discussion logs between LLM players. However, due to the page limits in the global rebuttal, we can only provide one analysis in the PDF file. More analysis results and discussion examples will be updated in the revision. Also, the anonymized link to our code has been sent to the Area Chair in a separate comment as required by the conference guidelines.\n\n**Response to W3: The proposed model is seemingly limited by the predefined discussion tactics and does not allow learning or discovery of alternative discussion tactics.**\n\nThank you for your comment. As discussed in our work, the predefined six discussion tactics were intended to simplify the problem and facilitate the learning process, due to the lack of a standardized classification for discussion tactics and adequate human player data in games like ONUW. While this approach does constrain the agents to choose from a predefined set of tactics, it also provides a structured framework for understanding the dynamics of discussion. However, as part of future work, we consider it interesting to explore more flexible and adaptive methods that allow for the discovery of new discussion tactics as well.\n\n**Response to W4: It would be good to have a random choice baseline and compare the proposed method to its performance.**\n\nThank you for your constructive comment. We have added an LLM-based agent whose discussion policy is to randomly choose discussion tactics as a new baseline (named *Random*). Please refer to the global response for the results.\n\n**Response to W5: About the number of negotiation rounds.**\n\nThank you for your feedback. Actually, although the ONUW game has only one Night, one Day, and one Voting phase, there could be many rounds of discussions during the Day phase where players are allowed to discuss, depending on the game settings. And in our settings, we allowed players to have three rounds of discussions. Please refer to the PDF file in our global response for the detailed game process. Moreover, we consider our discussion policy can adapt to multiple rounds of discussions since it relies on the current state of discussion rather than a specific number of rounds.\n\n**Response to Q1: The reason for choosing offline RL.**\n\nThank you for your comment. The main reason we chose offline RL is the slow interaction with LLMs, especially the interaction rate limits of GPT-4. Online RL requires agents to interact with the environment in real-time to collect data for training. However, as some parts of our agent framework are LLMs and each inference of LLMs takes a few seconds, it is almost impossible for our agents to train the discussion policy in an online manner. So we decide to collect game data first and then adopt offline RL for training.\n\n**Response to Q2: How is the classification into six discussion tactics performed: manually or by an 0-shot LLM?**\n\nThe classification of players' discussion tactics is performed by a 0-shot LLM.\n\n**Response to Q3: About the \"human-side\" or \"villager-side\".**\n\nThank you for pointing it out. The \"human-side\" is the name that reference [28] used to represent the \"villager-side\", oppositing to the \"werewolf-side\". As it might cause confusion for readers, we have replaced it with \"villager-side\" in our latest version to keep consistency with other parts of the paper.\n\n**Response to Q4: What is the reasoning behind selection of this specific number 1536 for state embeddings?**\n\nThank you for your insightful comment. The 1536 state dim is actually decided by the model we selected for state embeddings. In our experiments, GPT's `text-embedding-ada-002` model is adopted to encode players' observations and beliefs into state embeddings. We believe it is well pretrained and could effectively represent the semantic information of the original text. We have not performed a hyperparameter sweep across different state dims as it is not the main scope of our work, but it is an interesting suggestion for future work to explore the impact of varying the state dim.\n\n**Response to Q5: What is the total cost of the experiments described in the paper?**\n\nThank you for your comment. For each evaluation in our experiments, we repeated the game 30 times to obtain the final results. Since we conducted experiments on one three-player game (Sec 6.2), two five-player games in fixed settings (Sec 6.3), and one five-player game in a random setting (Sec 6.4), there are 50 sets of results. However, the cost of each agent version (such as *ReAct* and *LLM-instructed*) varies a lot due to their frameworks, and the Gemini API was free when we conducted our experiments, so we can only estimate the total cost be around 200.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful feedback! Your reviews have greatly improved the paper. And we are grateful for the appreciation of our theoretical analysis (`uoU5`, `ADZR`, `Tkt3`, `b5Fq`), the innovation of our framework (`ADZR`, `Tkt3`, `b5Fq`), and the recognition of our research scenario (`Tkt3`, `b5Fq`).

We have responded to each reviewer's respective questions below. Furthermore, we will also revise the paper accordingly to address all other suggestions and comments. For the code of our project, we have forwarded an anonymous GitHub repository to the Area Chair in accordance with the conference guidelines. It contains the implementation of the ONUW game environment, all agents used in the experiments, and the dataset process and policy training procedure. We hope our responses could clarify existing questions.

**Response to Reviewer b5Fq's suggestion on adding a random choice baseline.**

We added an LLM-based agent whose discussion policy is to randomly choose discussion tactics as a new baseline (named *Random*), and conducted experiments in the same settings as other agents in Table 1. Here are the results:

| Agents | Gemini (*Belief*) Win Rates | Gemini (*Belief*) Avg. Votes | GPT-4 (*Belief*) Win Rates | GPT-4 (*Belief*) Avg. Votes |
|-------|--------|--------|--------|--------|
| *ReAct*  | 0.40 | 1.23 | 0.30 | 1.73 |
| *Belief* | 0.40 | 1.73 | 0.32 | 1.87 |
| *Random* | 0.37 | 1.53 | 0.32 | 2.03 |
| *LLM-ins.* | 0.62 | 1.10 | 0.37 | 1.90 |
| *RL-ins.* | **0.70** | 1.10 | **0.50** | 1.87 |

It can be seen that the performance of the *Random* agent is almost similar to that of the *ReAct* and *Belief* agents, demonstrating the significance of strategic discussion policy. Especially, when playing with GPT-4-based *Belief* agents as Player 3, the *Random* agent is more likely to be voted. We recognize it is because the random discussion policy increases the inconsistency in agent's discussions.

**About the PDF file.**

In the supplementary PDF file, we provided a graph about the detailed game process of the ONUW game, and a typical example of the agents' performance along with human analysis.

Meta-review: This paper studies the impact of strategic discussion in the game One Night Ultimate Werewolf (ONUW).  It gives an equilibrium analysis of the game with and without discussion, implements an LLM-based RL discussion agent, and empirically evaluates the resulting agent.  It makes both theoretical and empirical contributions.  

The reviews were almost uniformly positive. They found the theoretical analysis solid and informative, and were largely satisfied by the empirical evaluations.  Two weaknesses were commonly noted, neither of which were fatal flaws: The need to manually bucket discussion tactics and the resulting specificity to ONUW; and the lack of evaluations against human players.
Decision: Accept (poster)