Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Authors: Zelai Xu
Wanjun Gu
Chao Yu
Yi Wu
Yu Wang
Venue: ICML 2025 poster (2025)
Keywords: LLM agents
strategic language games
game-theoretic methods
multi-agent
Primary area: reinforcement_learning->multiagent
TL;DR: We propose an iterative framework named Latent Space Policy Optimization (LSPO) that combines game-theoretic methods with LLM fine-tuning to build strategic language agent for the Werewolf game.

Abstract: Large language model (LLM) agents have recently demonstrated impressive capabilities in various domains like open-ended conversation and multi-step decision-making. However, it remains challenging for these agents to solve strategic language games, such as Werewolf, which demand both strategic decision-making and free-form language interactions. Existing LLM agents often suffer from intrinsic bias in their action distributions and limited exploration of the unbounded text action space, resulting in suboptimal performance. To address these challenges, we propose Latent Space Policy Optimization (LSPO), an iterative framework that combines game-theoretic methods with LLM fine-tuning to build strategic language agents. LSPO leverages the observation that while the language space is combinatorially large, the underlying strategy space is relatively compact. We first map free-form utterances into a finite latent strategy space, yielding an abstracted extensive-form game. Then we apply game-theoretic methods like Counterfactual Regret Minimization (CFR) to optimize the policy in the latent space. Finally, we fine-tune the LLM via Direct Preference Optimization (DPO) to align with the learned policy. By iteratively alternating between these steps, our LSPO agents progressively enhance both strategic reasoning and language communication. Experiment on the Werewolf game shows that our agents iteratively expand the strategy space with improving performance and outperform existing Werewolf agents, underscoring their effectiveness in free-form language games with strategic interactions.

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Zelai Xu Wanjun Gu
Chao Yu
Yi Wu 13
Yu Wang
Abstract
1. Introduction
Large language model (LLM) agents have re- cently demonstrated impressive capabilities in var- ious domains like open-ended conversation and multi-step decision-making However, it remains challenging for these agents to solve strategic lan- guage games, such as Werewolf, which demand both strategic decision-making and free-form lan- guage interactions. Existing LLM agents often suffer from intrinsic bias in their action distribu- tions and limited exploration of the unbounded text action space, resulting in suboptimal perfor- mance. To address these challenges, we propose Latent Space Policy Optimization (LSPO), an it- erative framework that combines game-theoretic methods with LLM fine-tuning build strate- gic language agents. LSPO leverages the obser- vation that while the language space is combi- natorially large, the underlying strategy space is relatively compact. We first map free-form utter- ances into a finite latent strategy space, yielding an abstracted extensive-form game. Then we ap- ply game-theorete theoretic methods like Counterfactual Regret Minimization (CFR) to optimize the pol- icy in the latent space. Finally, we fine-tune the LLM via Direct Preference Optimization (DPO) to align with the learned policy. By iteratively alternating between these steps, our LSPO agents progressively enhance both strategic reasoning and language communication. Experiment on the Werewolf game shows that our agents iteratively expand the strategy space with improving perfor- mance and outperform existing Werewolf agents, underscoring their effectiveness in free-form lan- guage games with strategic interactions.
Developing intelligent agents that can reason rationally, make strätegic decisions, and interact with humans has been long-term goal in artificial intelligence (Al) re- search (Wooldridge & Jennings, 1995; Russell & Norvig, 2016). In recent years, large language model (LL.M)-based agents have made significant strides towards this goal by ex- hibiting strong performance in open-ended conversation and multi-step
(Brown al., 2020; Ouyang et al., 2022). Trained massive text corpora, LLM-based agents have demonstrated remarkable versatility across vari- ous domains, ranging from web navigation (Nakano et al., 2021; Yao et al., 2022b) and code generation (Chen et al., 2021; Yang et al., 2024) to video game environment (Wang et al., 2023a) and real-world scenarios (Ahn et al., 2022; Bro- han al., 2023). Beyond single-agent tasks, LLM-based agents have also shown potential in multi-agent interac- tions including collaborative teamwork (Li et al., 2023), adversarial gameplay (Meta et al., 2022), and human-Al interation (Park et al., 2023; Liu et al., 2023).
Among these interactive domains, strategic language games such as Werewolf present unique challenges because they require both high-level strategic decision- making and free- form conversational abilities. Unlike classic games with predefined and limited actions, such as board games (Sil- ver et al., 2016; 2018), card games (Moravčík et al., 2017; Brown & Sandholm, 2018), and video games (Mnih, 2013; Vinyals et al., 2019). Werewolf relies heavily on free-form conversation to achieve agreements and perform strategic deceptions. Players must communicate, bluff, and infer hid- den roles through unrestricted, natural language interactions. This free-form language space expands the strategic possi- bilities and introduces additional complexity unmatched by more rigidly defined domains. As result, Werewolf serves as an ideal environment for developing strategic agents with Language-grounder decision-making capabilities.
Equual advising 'singhua University, Beijing, China 'Beijing Zhongguancun Academy, Beijing. China "Shanghai Qi Zhi Institute, Shanghai, China. Correspondence Chao Yu mauchao@tsinghua.oom
However, developing strategic language agent that can in- teract with humans in Werewolf or other free-form language environments is still challenging. Classic game-theoretic methods like Counterfactual Regret Minimization (CFR) and reinforcement learning (RL) have proven successful in games like Go and Poker, thanks to their ability to handle
Wu <jxwuyi mawwuy@@gmail.com coms, Yu Wang <yu-ward wang @iinggguiinhu..du....
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
finite action spaces. Yer Werewolf has a free-form action space, making direct application of these methods computa- tionally infeasible. Mapping every possible utterance to action in the original text space becomes prohibitively large, leading to immense difficulty in strategy representation and cquilibrium-finn findinn-iinnn...... An alternative approach is to build lan- guage agents with LLMs. These methods typically rely on prompt engineering without training the base LLM, which means their success depends entirely on the general reason- ing capabilities of LLMs to generate actions. Unfortunately, prompt-based methods suffer from intrinsic bias in their generated actions (Xu et al., 2023c), resulting in subopti- mal performance in strategic language games like Werewolf. Moreover, these agents exhibit limited exploration of novel strategies because they fully rely on the LLMs to generate actions, making the agents constrained by the capability of the base LLMs. Some work (Chen et al., 2023a; Wu et al., 2024) mitigates these issues by fine-tuning the LLM for a specific task, which requires expensive human labor for high-quality data. Xu et al. (2023c) partially tackles the bias issue by additionally training a small network to calibrate the LLM output distribution. However, still re- lies on fixed LLM to produce action candidates, leaving the exploration issue unaddressed. This raises an important question: Can we have a method that leverages both the specialized reasoning capabilities for prcisiionmmkinn and the gemeralization capabilities LLMs?
agents and find that the LSPO agent achieves the highest win rate. We also conduct ablation studies to assess the effectiveness of our design in the LSPO framework.
2. The Werewolf Game
Werewolf is a popular social deduction game where play- ers with hidden roles cooperate and compete with others in natural languages. The Werewolf side needs to conceal their identities and eliminate the other players, while the Village side needs to identify their teammates and vote out the Werewolves. Players are required to have both lan- guage proficiency for communication and strategic ability for decision-making to achieve strong performance in the Werewolf game. We consider a seven-player game with two Werewolves being the Werewolf side and one Seer, one Doctor, and three Villagers being the Village side. Detailed descriptions of the game's rule, observation space, and re- ward function can be found in Appendix A.
2.1. Game Environment
We consider a text-based seven-player Werewolf game that proceeds through natural languages. We exclude other infor- mation like the speaking tone, facial expression, and body language (Lai et al., 2022). This pure text-based environ- ment is common setup in the literature (Xu et al., 2023a;c; Wu et al., 2024; Bailis al., 2024).
In this work, we propose an iterative Latent Space Pol- icy Optimization (LSPO) framework to build strategic lan- guage agents for free-form language games, taking Were- wolf as our testbed. Our approach combines structured game-theoretic methods with language models by introduc- ing a discrete latent strategy space. Our method consists of three components. We first map the free-form utterances into a manageable, discrete strategy space to yield an ab- stracted game. Then we apply game-theoretie methods like CFR learn the optimal policy in the latent space. Finally, fine-tune the LLM via Direct Preference Optimization (DPO) (Rafailov et al., 2024) to align with the learned pol- icy and expand the latent space. By iterating between these latent space CFR steps and LLM fine-tuning fine-tuning, our method yields an evolving agent that addresses both the intrinsic bias issue with game-theoretic methods and the action explo- ration issue with latent space expansion, leading to strong performance in the Werewolf game.
Roles and Objectives. At the beginning of each game, the seven players are randomly partitioned into two sides. The Werewolf side has two Werewolf players who know each other's role and aim to eliminate the other players while avoiding being discovered. The Village side has one Seer who can check the role of one player each night, one Doctor who can protect one player each night, and three Villagers without any ability. The players in the Village side only know their own role and need to share information to identify the Werewolves and vote them out.
Game Progression. The game proceeds by alternating between night round and day round. In the night round, players can perform secret actions that are only observable by themselves. More specifically, the two Werewolves can choose a target player to eliminate, the Seer can choose target player to investigate whether the player's role is Werewolf, and the Doctor can choose a target player to protect the player from being eliminated. The Doctor does not know the target player chosen by the Werewolves. If the Doctor chooses the same target player as the Werewolves, then no player is eliminated in this night round, otherwise. the Doctor fails to protect any player, and the target chosen by the Werewolves is eliminated.
We perform extensive experiments in the Werewolf game to demonstrate the effectiveness of our LSPO framework. We first analyze how the latent strategy space evolves between iterations to show that our agents learn increasingly complex and strategic behaviors. Then we quantitatively evaluate the prediction accuracy and win rate of our LSPO agent show the improving performance with respect to iterations. Next, we compare our agents against ofth-offthe-art Werewolf
Observations and Actions. The language observation of each agent is a list of natural languages that log the game
2

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Policy Optimization in Latent Space
Latent Space Construction
Latent Space Expansion
Language Action
Original Game
Abstracted Game
Latent Space Policy
I'm the Seer and
LLM
Isaw Player is
Werewolf
saddened
Preference Data
by the loss of Player
I'm the Seer and saw Player Werewolf
I'm saddened by the loss of Player
>
Game Solver (CFR, RL, etc.)
Clustering
Fine-Tune
think Player is suspicious
Latent Strategy
Abstracted Game
Latent Space Policy
LLM
Figure I: Overview of the Latent Space Policy Optimization (LSPO) framework. Each iteration consists of three components. (I) Latent space construction: generate language actions with the LLM and cluster the vast language action into a finite latent strategy space. (2) Policy optimization in latent space: reformulate the original game as an abstracted game and apply game-theoretic methods to learn latent space policy. (3) Latent space expansion: fine-tune the LLM to align with the latent space policy and generate new strategies to expand the latent strategy space.
history to the current step. This list include both private information that are only observable to the current player and public information that are shared by all players. The private information includes the role of the current player, the secret actions in the night round for the Werewolf, Seer, and Doctor, and the teammate for the Werewolf. The public information includes the ID of the current player, the elim- inated player in each night and day round, the discussion, and the voting result in each day round.
LLM-based agents tend exhibit intrinsic bias in their action generation, which is inherited from the model's pre- training data. This issue is more pronounced in complex language games like Werewolf, where the opponents can exploit these predictable biases to counteract the agent's s move. Therefore, mitigating intrinsic bias is essential for language agents to reduce exploitation and achieve strong performance in strategic language games.
Exploration of Unbounded Action Space. Due to the immense combinatorial space induced by free-form text, it is impractical to map every possible utterance to an ac- tion in the language space. On the other hand, manually engineering or prompting an LLM to produce limited set of actions may fail to capture the full strategic landscape. Even an agent optimally masters the action distribution within a limited subset, it could be easily exploited by out-fo- distribution utterance. Consequently, inadequate exploration of the action space could result in suboptimal performance in free-form language games like Werewolf.
Player actions are also in the form of natural language and can be categorized into three types: secret actions, which are secret actions performed during the night, such as choosing a target player to eliminate, investigate, or protect; discus- sion actions, which are statements made during the day to influence other players' perceptions and decisions; and vot- ing actions, which are choices made during the voting round to vote for on player or choose not to vote.
2.2. Challenges for Language Agents
Unlike board, card, video games with finite set of pre- defined actions, Werewolf has a free-form language action space. The vast space of natural language actions poses two key challenges for language agents to achieve strong performance in the Werewolf game.
3. Latent Space Policy Optimization
To tackle the intrinsic bias and the exploration issue, we pro- pose an iterative Latent Space Policy Optimization (LSPO) framework. Our method combines game-theoretic opti- mization with LLM fine-tuning and operates on an expand- ing latent strategy space to itcratively improve the agent's s
Intrinsic Bias in Action Generation. As observed in sim- ple games like Kock-Paper-Seisee (Xu et al., 2023c), pure
3

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
decision-making ability and action exploration. As shown in Figure 1, our framework has three components includ- ing latent space construction, policy optimization in latent space, and latent space expansion. More implementation details can be found in Appendix C.
space as an abstracted game with a finite latent strategy space. This reformulation allows to apply standard game- solving techniques such as Counterfactual Regret Minimiza- tion (CFR) or reinforcement learning (RL) methods to learn near-optimal strategies that overcome the intrinsic bias. In our implementation, we employ CFR as the game solver.
3.1. Latent Space Construction
Abstracted Game Formulation. To represent the game in compact, finite form, we replace the free-form discussion actions with the discrete latent strategies from latent space construction. Specifically, the abstracted game is formalized as an extensive- cerensiveformmne game (EFG), where the secret action and voting action remain the same, and the discussion action is replaced by the latent strategy. The state in the abstracted game a vector including information like the player's role, secret action, etc., and history of past latent strategies. The transition dynamics and payoff function remain unchanged in the abstracted game. This representation retains the key strategic elements of the original game while reducing the complexity of the action space, making large-scale game- solving computationally tractable, Detailed description of the abstracted game can be found in Appendix A.
One of the key challenges in free-form language games like Werewolf is achieving broad exploration of the unbounded text space while maintaining a computationally tractable action representation for game-theoretic methods. To strike a balance between exploration and tractability, we propose to abstract the vast language action space into finite set of latent strategies, which we then expand over iterations for better exploration. Specifically, our latent space construc- tion in each iteration involves two steps including latent strategy generation and clustering.
Latent Strategy Generation. In our setting, secret actions and voting actions are already discrete and therefore do not require further abstraction. We focus instead on the free- form discussion actions, which we aim to capture as latent strategies. We assume that each role in the game has the same set of latent strategies across all discussion rounds and collect the latent strategies for each role by letting the current L.M agent self-play as different roles for multiple trajectories. To further improve the exploration of latent strategies, we prompt the LLM to generate N strategically distinct discussion candidates and randomly choose one to execute in the game. This process encourages diversity in the collected discussion actions and generate a set of latent strategies in natural language for each role.
Policy Optimization. Once the game is represented in this discrete form, apply CFR to learn a policy and solve the abstracted game. Classical CFR (Zinkevich et al., 2007) iteratively improves policies by minimizing counterfactual regret R for each information set. For each iteration 4, the regret for each action a in the latent space is updated by:
(a) = 1 + ulof o, u(at),
(1)
where u(af, -.=)))) is the utility of taking action under the current strategy profile os, and u(a,) is the utility under the full strategy profile. We use neural networks to approximate regret value to scale CFR to more complex games and learn policy for each different role in the Werewolf game. By repeatedly simulating self-play among agents employing Deep CFR in the abstracted game, each role's policy con- verges to near-optimal strategy profile. The resulting latent space policies address the intrinsic bias in action distribution and achieve strong strategic play in the abstracted game.
Latent Strategy Clustering. Although we generate a set of latent strategies for each role, they are still in the form of natural language. To transform them into discrete la- tent strategy space, we embed each discussion action into a vector representation using an embedding model such as "'ext-embedddig---smmee ---malll" that captures its semantic and contextual information. We then apply a simple k-means clustering algorithm to partition the embedded utterances into & clusters, where each cluster represents a distinct latent strategy. Clustering reduces the infinite free-form text space to finite set of abstract strategies, paving the way for sub- sequent game- theerete optimization. By interpreting each cluster as a latent action, we can more efficiently search for and optimize strategic policies with minimal sacrifice of exploration of language space.
3.3. Latent Space Expansion
To further improve the agent's performance in free-form language games, the latent space must remain sufficiently expressive to cover novel strategies and resist exploitation by nuttof-distribution actions. We achieve this by fine-tuning the LLM to align with the learned policy in the abstracted game and then re-generating and expanding the latent strat- egy space using the fine-tuned LLM. This iterative process progressively increases exploration of the action space, en- abling stronger and more robust decision-making.
3.2. Policy Optimization in Latent Space
Another challenge in free-form language games is to ad- dress the intrinsic bias in the agent's action distribution. After constructing a discrete latent strategy space, we can reformulate the original game with unbounded language
Alignment to Latent Space Policy. We employ Direct Preference Optimization (DPO) (Rafailov et al., 2024) to
4

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
ReAct
SLA
LSPO Iter. 1
LSPO Iter. 2
LSPO Iter. 3
0.6
0.53
0.4
0.29
0.330.33 0.33
0. 0.33
0.33
0.2
0.2
0.2
0.2
0.2
0.14
0.02
0
0
0
0
0
0.0
R.
P.
S.
Sp.
R.
P
5.
R.
P
R.
L.
R.
5p.
L.
exp.
0.39
exp.
0.33
exp.
0.33
exp.
0.33
exp.
0.00
Figure 2: The action distributions and exploitabilities (exp.) of different agents in the Paper--cissoorss Spock-Lizard game. The Nash equilibrium is choose each action with an equal probability of 1/5 and has an exploitability of 0.
fine-tune the LLM that its open-ended language outputs align with the near-optimal strategies derived from the ab- stracted game. To construct the preference dataset required by DPO, we leverage game trajectories generated during la- tent space construction. We record the language observation for the LLM agent at each discussion phase as the prompt, and use the N discussion candidates as the response candi- be mapped to
agents, showing that our agents achieve the highest win rate both the Werewolf side and the Village side. We further perform ablation studies to assess the effectiveness of specific designs in our framework. More implementation and experiment details can be found in Appendix B.
4.1. Proof-of-Concee Example
dates. Each of the discussion candidates one of the latent strategies, and the preference label is deter- mined by the regret value of the latent strategies. Intuitively, discussion action with a lower regret value is preferred. With this preference dataset, we perform DPO to align the LLM toward the learned policy in the abstracted game for better performance in the original game.
We consider Rock-Paper- Pepper-ScisoorsSSpoc--iiii a five- choice extension of the classic Rock-Puper-Scissors game. Although is not a free-form language game, it serves as a simple sroof-ff-concep game that highlight the motivation of our method. The intrinsic bias in action distributions is inherited from the imbalanced LLM pre-training data, and the exploration issue is introduced by the two additional actions of Spock and Lizard. The Nash equilibrium (NE) of this game is to choose each action with an equal probability of 1/5. We compare LSPO agents of different iterations with two baselines, including Reason and Act (ReAct) (Yao et al., 2022b) and Strategic Language Agent (SLA) (Xu et al., 2023c), and the action distributions and exploitabili- ties of different agents are shown in Figure 2.
Update of Latent Space. Once the LLM is fine-tuned, it can produce broader distribution of actions that reflect the refined policy. We exploit this enhanced generative capacity to expand the latent space in the next iteration. Specifically, we repeat the latent strategy generation and clustering pro- cedures with the fine-tuned L.M to re-generate and expand the latent strategy space. This updated latent space offers increased exploration of potential strategies, enabling subse- quent policy optimization to discover previously unexplored high-reward actions. Through iterative alignment and expan- sion, the agent continually refines its discussion strategies and achieves strong play in the free-form language game.
The evaluation results show that the LSPO agent of itera- tion 3 learns the NE of the game while other agents fail. ReAct agent suffers from the intrinsic bias issue and has higher probabilities to choose Rock, Paper, and Scissors, and much lower probabilities to choose Spock and Lizard. SLA, the other hand, is hindered by inadequate action exploration. SLA uses I.LMs to propose N actions and RL. to learn the optimal policy. A typical value of N = 3 results in a subgame without the other action, and the NE of the subgame is not the NE of the original game. Our LSPO agent addresses these two challenges by iteratively expand- ing the action space. As covers the full action space with 3 iterations, the LSPO agent learns the NE of the game.
4. Experiments
To demonstrate the effectiveness of the LSPO framework, we first consider a proof-of-concept game to show how LSPO overcomes intrinsic bias and addresses the explo- ration issue. Then we conduct extensive experiments in the Werewolf game with -inaa---B-Innttee as our base model. We visualize how the latent strategy space evolves show that our agents progressively acquire more complex strategic behaviors. We then quantitatively evaluate the per- formance of our LSPO agent using prediction accuracy and win rate to show the improving performance over iterations. We also compare the LSPO agent with four ttateoff-the-at
4.2. Latent Space Visualization
To gain insight into how LSPO organizes free-form language actions into discrete latent strategies, first visualize the latent strategy space constructed at different training itera-
5

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Werewolf's Latent Space
Seer's Latent Space
Iter. 1
Iter. 2
Iter. 3
Iter.
Iter.
Iter. 3
Figure 3: Visualization of the latent strategic space of Werewolf and Seer in different LSPO iterations.
tions. Specifically, for each role in the Werewolf game, we gather the utterances generated by the LSPO agent in 100 games, embed them with the sentence encoder, and apply di- mensionality reduction for projection. The visualization of latent spaces for the Werewolf and the Seer in different itera- tions is shown in Figure 3. Earlier iterations yield relatively indistinct clusters, reflecting a lack of strategic diversity. Over successive iterations, clearer and more refined clusters emerge, indicating that the LSPO agent evolves toward an increasingly structured latent space and learn to express dif- ferent strategic intentions such accusing specific roles, defending teammates, and bluffing.
yet subtly guiding discussions to protect verified teammates. Notably, by the final iteration, the model develops a voting coordination strategy in which the Seer explicitly asks all the Villagers to vote for strongly suspected Werewolf to maximize the Villager's s chance of success. This progression implies that the Seer agent increasingly learns to balance openness and secrecy, aligning its communication style with the evolving game context to better support the Village side.
4.3. Iterative Performance Evaluation
We then evaluate how the performance of our LSPO agent progresses with more iterations, demonstrating that our framework produces increasingly stronger strategic lan- guage agents over time. We focus two key metries including prediction accuracy and win rate.
Werewolf's Latent Space. In the first iteration, the latent space of the Werewolf is dominated by three main clusters. The blue cluster corresponds to a simple strategy of conceal- ing its role or pretending be a villager, while the smaller orange cluster reflects strategies like pretending be a Seer or Doctor. There is even green cluster corresponding to unintentionally revealing the true role of a Werewolf, which is obviously flawed strategy. As training proceeds, we see more sophisticated patterns emerge. The flawed strategy of disclosing one's s Werewolf role disappears, and the agent begins to incorporate deliberate bluffs and misdirections instead. For example, the red cluster features the agent pre- tending to be a Seer and providing fabricated investigative results to sow confusion, and the purple cluster centers on defending the teammate and redirecting suspicion onto other players, leveraging more nuanced language and reasoning to guide the conversation toward scapegoats. This refined parti- tioning demonstrates that the Werewolf agent progressively covers an increasing number latent strategies.
Prediction Accuracy. Accurate role identification is criti- cal aspect of Werewolf, as it underpins effective decision- making and voting. Therefore, we measure the agent's abil- ity to predict the roles of other players with an additional prediction phase before each voting phase in a Werewolf game. Specifically, we use the final-iteration ISPO agent as the fixed opponent and let LSPO agents at different it- erations play against this opponent for 100 games. For the Werewolf side, a higher prediction accuracy of crucial roles like Seer and Doctor allows them to eliminate these threats earlier. Conversely, for the Village side, a higher prediction accuracy of Werewolves improves their chance to vote out the Werewolf and win the game.
Win Rate. While prediction accuracy serves interme- diate metric to evaluate the agents' reasoning and decision- making ability, we also use the win rate as direct measure of the performance of our agents. Similar to the evalua- tion of prediction accuracy, we use the final-iteration LSPO agent as the fixed opponent and let our agents at different iterations play 100 games against the opponent. A higher win rate indicates a stronger performance in the game.
Seer's Latent Space. In the first iteration, the Seer's latent space is relatively coarse, containing primarily two strate- gies including staying silent about its true role or revealing its role and sharing information. This shows a limited range of strategic diversity in the early stage. As training proceeds through the second and third iterations, the Seer's latent space becomes more diverse. The emergent red cluster fea- tures direct accusations once the Seer identifies a Werewolf, while the green cluster corresponds to concealing the role
As shown in Table 1, both prediction accuracy and win rate exhibit a clear growing trend as the iteration increases, indi- cating that our iterative LSPO framework steadily strength-
6

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Prediction Accuracy
Win Rate
Werewolf
Seer
Doctor
Villager
Overall
Iter. 1
0.98 + 0.01
0.61 + 0.09
0.49 + 0.08
0.70 + 0.07
0.74 0.06
0.54 + 0.13
Werewolf Side
Iter. 2
0.99 + 0.01
0.68 + 0.07
0.59 + 0.06
0.77 + 0.09
0.79 0.06
0.63 + 0.09
Iter. 3
0.99 + 0.00
0.73 + 0.08
0.67 =: 0.07
0.81 0.11
0.83 + 0.07
0.73 + 0.11
Iter. I
0.59 + 0.06
0.47 + 0.04
0.55 + 0.05
0.67 + 0.07
0.60 0.06
0.18 + 0.09
Village Side
lter. 2
0.66 = 0.06
0.53 0.07
0.61 = 0.06
0.75 = 0.08
0.67 0.07
0.23 = 0.12
Iter. 3
0.72 + 0.09
0.58 + 0.08
0.65 + 0.07
0.82 + 0.08
0.73 = 0.07
0.27 = 0.11
Table I: The prediction accuracy and win rate of the LSPO agents in different iterations.
Win Rate
ReAct
ReCon
Cicero-like
SLA
LSPO Agent (Ours)
As the Werewolf Side
0.58 + 0.15
0.60 + 0.12
0.66 0.06
0.69 + 0.12
0.73 + 0.11 0.27 = 0.11
As the Village Side
0.16 + 0.06
0.16 0.08
0.21 + 0.04
0.25 = 0.08
Overall
0.38 + 0.11
0.38 + 0.10
0.44 + 0.05
0.47 + 0.10
0.50 + 0.11
Table 2: Comparison between our LSPO agent with state-of-the-art agents in the Werewolf game.
ens the agents' reasoning and decision decision-making capabilities. From the Werewolf side, the identification rate for the Seer starts off relatively high but has only modest improvement. This is because the Seer often reveals its roles to share infor- mation, making it easier for the Werewolf side to identify. By contrast, the Werewolf's prediction accuracy of the Doc- tor shows more significant gains, reflecting the strategic importance of eliminating the Doctor who can save poten- tial victims. On the Village side, identifying the Werewolf and the Seer benefits most from iterative learning, since con- firming these central roles is crucial for coordinated voting and elimination of Werewolves. Overall, these results con- firm that our framework consistently improves the strategic language abilities of the LSPO agent, enabling it to adapt and excel in complex social deduction scenarios with each additional iteration.
ReCon. ReCon is another prompt-based method designed for Avalon agents. The ReCon agent is prompted to first think from its own perspective and then think from its op- ponents' perspective to generate the final action. We make slight modifications in the prompt to apply ReCon in the Werewolf game.
Cicero-Like. The Cicero agent is created for the game of Diplomacy with finite action space and consists of strategic reasoning module and a dialogue module. We implement a Cicero-like agent for the Werewolf game by predefining an action space of 13 primitive actions like "claim to be the Seer", "do not reveal role", etc. An RL policy is learned to select these actions in each state and generate action- conditioned languages in the game.
SLA. SLA combines reinforcement learning and LLM to overcome intrinsic bias and build strategic language agents for the Werewolf game. We adopt the same implementation as described in the paper (Xu et al., 2023c).
4.4. Comparison with State-of-the-Art Agents
We compare the performance of the LSPO agent in the Werewolf game with four state-of-the-art agents including Reason and Act (ReAct) (Yao et al., 2022b), Recursive Contemplation (ReCon) (Wang et al., 2023b), a Cicero-like agent (Meta et al., 2022), and Strategic Language Agent (SLA) (Xu et al., 2023c). As some of these methods were not initially developed for Werewolf, we make minor modifi- cations to ensure compatibility with our experimental setting while preserving each approach's 's core design.
We compare our final-iteration LSPO agent with the afore- mentioned four baselines through two head-to-heal evalu- ation setups. In the first setup, our LSPO agent takes the Werewolf side and we let each of the five agents including our agent and four baseline agents take the Village side to play 100 Werewolf games with our LSPO agent. This setup measures the Village side's win rate against the LSPO agent as the Werewolves. In the second setup, we reverse the roles and let the LSPO agent take the Village side and com- pare the win rate of five agents as the Werewolves averaged over 100 games. As shown in the bold numbers in Table 2, our LSPO agent achieve the highest win rates both as the Werewolves and as the Villagers.
ReAct. ReAct is a classic prompt-based method that syner- gizes reasoning and acting for agent tasks. We implement ReAct for the Werewolf game by providing the LLM with raw game observations to generate both intermediate rea- soning and final actions within single prompt.
7

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Win Rate
Village Side
Werewolf Side
Win Rate
Iter. 1
Iter. 2
Iter. 3
LSPO Iter.
0.18 - 0.09
0.54 + 0.13
k 0.13 = 0.06
0.20 + 0.10
0.24 0.11
w/o Fine-Tuning
0.15 0.09
0.47 + 0.14
k 2
0.22 + 0.11
0.24 + 0.12
0.25 0.08
w/o Policy Learning
0.12 0.07
0.38 + 0.16
h: 3
0.23 + 0.09
0.25 + 0.06
0.25 0.07
Table 3: Ablation on key components of LSPO agents.
Table 4: Ablation on cluster size.
Win Rate
Iter. 1
Iter. 2
Iter.
The strong performance of our LSPO agent is largely at- tributable to its iterative interplay between latent space strat- egy learning and preference- fine-tuning, which refines both language and eeriiiinnmmaiingaae over time. By contrast, ReAct and ReCon rely on prompt-based approaches without game-theoretic updates, leaving them susceptible to intrinsic biases from pretraining data and limiting their performance in complex decision-making tasks. The Cicero-like agent is constrained by a predefined action set, making it difficult to evolve more subtle and diverse strategies as the game progresses. SLA partially addresses the intrinsic bias issues by generating multiple candidate actions and using RL to select from them. However, still relies on a prompt-based method that can suffer from limited exploration of poten- tial strategies. In comparison, our LSPO method integrates CFR's policy improvement and latent-space cluster refine- ment with preference-based L.M alignment, enabling it to explore, exploit, and continuously expand the range of viable strategic moves in social deduction games.
k = 1
0.22 + 0.10
0.24 + 0.08
0.25 = 0.08
=
0.22 = 0.11
0.24 = 0.12
0.25 = 0.08
=3
0.23 + 0.07
0.25 + 0.09
0.25 0.06
Table 5: Ablation on DPO hyperparameters.
sensitivity analysis on the number of initial clusters. We consider simpler four-player Werewolf game (one Were- wolf, one Seer, and two Villagers) and run LSPO with dif- ferent numbers of initial clusters k = 1, and evaluate the Werewolf's win rate. The results in Table 4 show that larger numbers of initial clusters generally lead to better performance in the early iterations, but do not influence the final performance after three iterations.
Fine-Tuning Hyperparameters. We also perform a sen- sitivity analysis evaluate our method's robustness to fine-tuning hyperparameters. We also consider the sim- pler four-player game and perform ablations on DPO / = 0.05, 0.1, 0.2. The results in Table 5 show that our method achieves comparable results with different choices of /3 and is robust to fine fine-tuning hyperparameters.
4.5. Ablation Studies
Key Components. To show the effectiveness of the three key components in LSPO agents, we compare the LSPO agent in iteration with two ablation agents. The first ab- lation agent, denoted as "w/o Fine-Tuning", removes the third component and only performs latent space construc- tion and policy optimization in the latent space. To generate discussion action in gameplay, this agent first uses the latent space policy to sample a latent strategy, then the previously collected discussions corresponding to the latent strategy are used as few-shot examples to prompt the LLM for the discussion action. The second ablation agent, denoted as "wwo Policy Learning", removes the second component of policy optimization in the latent space. Instead, it uses g.5 5 to generate the preference data and uses DPO to train for 1 iteration As shown in Table 3, the LSPO agent in iteration achieves the best result on both the Villager and the Werewolf side. These results demonstrate that the policy optimization component helps agents learn stronger strategies, while the fine-tuning component helps LLMs bet- ter generalize to new language actions beyond the collected samples and expand the latent strategic space.
5. Related Work
Large Language Model-Based Agents.
Recent advancements in large language models (LLMs) have led to the development of agents capable of performing complex tasks across various domains, such as web inter- actions (Nakano et al., 2021; Yao et al., 2022a; Deng et al., 2023), code generation (Chen et al., 2021; Yang al., 2024), gaming environments (Huang et al., 2022a; Wang et al., 2023c;a; Ma et al., 2023), real-world robotics (Ahn et al., 2022; Huang et al., 2022b; Vemprala et al., 2023), and multi- agent systems (Park et al., 2023; Li et al., 2023; Chen et al., 2023b). A common approach in these works is to exploit the reasoning capabilities and in-context learning of LLMs to improve srecisionmmainn processes. Chain-of-Thoughh (CoT) prompting (Wei et al., 2022) has been instrumental in enabling LLMs to perform step-by-step reasoning. Building upon this, ReAct (Yao et al., 2022b) synergizes reasoning and action to enhance performance across various tasks. Subsequent research has incorporated self-reflection (Shinn et al., 2023) and strategic reasoning (Gandhi et al., 2023) to
Number of Initial Clusters.
To examine the robustness of our method, we perform a
8

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
further refine agent behaviors. However, these methods can still suffer from the intrinsic biases and exploration issue of LLM-based agents, leading to suboptimal performance in complex games. A representative method that addresses these issues in the game of Diplomacy is Cicero (Meta et al., 2022), which first uses a strategic module to produce ac- tion intent and then generates action-conditioned natural languages with a dialogue module. However, Diplomacy is board game with finite action space and does not have the exploration issue, making it not suitable for free-form language games with unbounded text action space.
by introducing a finite latent strategy space to enable it to solve free-form language games.
Reinforcement learning (RL) methods, on the other hand, have reached remarkable achievements in complex domains like Go (Silver et al., 2016; 2018) and video games (Vinyals et al., 2019; Berner et al., 2019), often surpassing expert hu- man performance. A seminal technique in these successes self-play and its variants(Heinrich et al., 2015; Heinrich & Silver, 2016; Hennes et al., 2020; Xu et al., 2023b), where agents repeatedly train against older versions of themselves to refine their policies. Another prominent line of work is Policy-Space Response Oracles (PSRO) (Lanctot et al., 2017; Muller et al., 2019), an iterative procedure that pro- duces best responses to a growing population of policies in a meta-game. Conceptually, our iterative framework is related to PSRO in that we both solve an abstracted game before enlarging it to approach the full original game. The difference is that PSRO treats newly learned policies as meta-actions to form a normal-form meta-game, whereas our approach clusters free-form language actions into a dis- crete latent action space to reformulate the original game as an extensive-form game with finite action space.
Due to the high demand for both advanced communication skills and strategic reasoning, social deduction games like Werewolf and Avalon have been proposed as testheds to build language agents with strategic ability. Earlier attempts to create agents for these games often rely on predefined protocols or limited communication capabilities (Wang & Kaneko, 2018), restricting their effectiveness. Recent works have explored using LLMs to enable natural language inter- actions in these games. For instance, Xu et al. (2023a) de- veloped a prompt-based Werewolf agent that uses heuristic information retrieval and experience reflection. Similarly, ReCon (Wang et al., 2023b) introduced a prompt-based method for playing Avalon by considering both the agent's ' perspective and that of opponents. However, these LLM- based agents may still be restricted by intrinsic bias and lim- ited exploration of the action space, affecting their decision- making quality. Strategic Language Agent (SLA) (Xu et al., 2023c) partially solves these issues by generating diverse action candidates and learning an RL policy to mitigate intrinsic bias. However, this method still relies a fixed LLM to produce the action candidates, which can fail to address the exploration issue. Our approach mitigates the intrinsic bias by applying game-theorens methods to opti- mize policy in a discrete latent strategy space and tackles the exploration issue by iteratively expanding the latent space by aligning the LLM to the latent space policy, leading to strong performance in the Werewolf game.
6. Conclusion
In this work, we presented Latent Space Policy Optimiza- tion (LSPP),, an iterative framework that combines struc- tured game-theoretic methods with the expressive power of large language models to build strategic language agents in free-form strategic language games. By abstracting uncon- strained language action space into discrete latent strat- egy space, our approach enables efficient CFR in the latent space to overcome intrinsic bias and learn strong strate- gies. We then perform iterative fine-tuning via DPO to align the LLM's language generation with the evolving strategy and expand the latent strategy space to address the explo- ration issue. Our extensive evaluation in the Werewolf game demonstrates that LSPO not only addresses intrinsic biases and exploration issues inherent in prompt-based agents, but also achieves increasing performance with respect to itera- tions and outperforms four sfate-of-the-art baseline agents. Looking ahead, we envision LSPO's 's synergy of latent-space abstraction and preference- be extended a variety of other complex decision-making tasks with free-form language actions.
Game-Theoretic Algorithms. Counterfactual Regret Min- imization (CFR) (Zinkevich et al., 2007) is a foundational algorithm for solving imperfect-infore informatiiino games, par- ticularly those involving hidden information and strategic deception like poker (Moravčík et al., 2017: Brown & Sand- holm, 2018; 2019). The core principle of CFR is itera- tively reduce regret across players' decision points in the game tree, converging toward strategies that approximate Nash equilibrium. Subsequent refinements of CFR (Lanc- tot et al., 2009; Tammelin, 2014; Brown et al., 2019) have expanded its scalability and adaptability to a broader range of scenarios. Of particular note DeepRole (Serrino et al., 2019), which integrates deductive reasoning with CFR to play the social deduction game Avalon without communi- cation. Our method combines CFR with language models
language alignment can
A Acknowledgements
This work was supported by National Natural Science Foundation of China (No.62406159, 62325405), Postdoc- toral Fellowship Program of CPSF under Grant Number (GZC20240830, 2024M761676), China Postdoctoral Sci- ence Special Foundation 2024T170496.
9

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Impact Statement
Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:23 20.00195,, 2023a.
Our research advances the capabilities of LLM-based agents in a purely text-based Werewolf environment. While this setting allows the agents develop robust decision-makee making and deception-detection skills, it also underscores the poten- tial for misuse if similar techniques were to be adapted to real-world scenarios involving manipulation or misinforma- tion. To mitigate these risks, our implementation remains strictly focused on text-based simulation and does not di- rectly transfer to broader applications without additional safeguards. At the same time, our experiment results in- dicate that our agent could be used to identify potential deceptive and manipulative content. We envision that any future extensions of this work will require careful consid- eration of ethical guidelines and responsible deployment strategies to ensure that such language agent systems serve society constructively.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, P.D.D. O.,
Kaplan, J., Edwards, H.,, Burda, Y., Joseph. N., Brockman,
G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Chen, W., Su, Y., Zuo, J., Yang. C., Yuan, C., Qian, C., Chan, C.-M., Qin, Y., Lu, Y., Xie. R., et al. Agentverse: Facili- tating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.1084 2023b.
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards generalist agent for the web. arXiv preprint 20Ki::2300600070, 2023.
References
Gandhi, K., Sadigh, D., and Goodman, N. D. Strate- gic reasoning with language models. arXiv preprint 19X1::2305./9165,, 2023.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Heinrich, J. and Silver, D. Deep reinforcement learning from self-play in imperfectt-nnoormaion games. arXiv preprint arXiv:1603.0112. 2016.
Bailis, S., Friedhoff, J., and Chen, F. Werewolf arena: A case study in Ilm evaluation via social deduction. arXiv preprint arXiv:2407. 13943, 2024.
Heinrich, J., Lanetot, M., and Silver, D. Fictitious self-play in extensive-form games. In International conference on machine learning, pp. 805-813. PMLR, 2015.
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: /9/2//99200...../ 2019.
Hennes, D., Morrill, D., Omidshafiei, S., Munos, R., Pero- lat, J., Lanctot, M., Gruslys, A., Lespiau. J.-B., Parmas, P., Duéñez-Guzzáán E.,, et al. Neural replicator dynam- ies: Multiagent learning via hedging policy gradients. In Proceedings of the 19th international conference on au- tonomous agents and multiagent systems, pp. 492-501, 2020.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X.. Choromanski, K., Ding, T., Driess, D.,. Dubey, A., Finn, C., et al. Rt-2: Vision-language-actioo models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, Lan- guage models as zero-shot planners: Extracting ac- tionable knowledge for embodied agents. In Interna- tional Conference on Machine Learning, pp. 91 9118-9147. PMLR, 2022a.
Brown, N. and Sandholm, T. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418 424, 2018.
Brown, N. and Sandholm, T. Superhuman ai for multiplayer poker. Science, (65(6456))885 890, 2019.
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng A., Tompson, J.,, Mordatch, I., Chebotar, Y., et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05600 2022b.
Brown, N., Lerer, A., Gross, S., and Sandholm, T. Deep counterfactual regret minimization. In International con- ference on machine learning, pp. 793-802. PMLR, 2019.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Lai, B., Zhang, H., Liu, M., Pariani, A., Ryan, F., Jia, W., Hayati, S. A., Rehg, J. M., and Yang, D. Werewolf among us: multimodal dataset for modeling persua- sion behaviors in social deduction games. arXiv preprint arXi::2212.08279 2022.
10

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Lanctot, M., Waugh, K., Zinkevich, M., and Bowling, M. Monte carlo sampling for regret minimization in exten- sive games, Advances in neural information processing systems, 22, 2009.
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: In- teractive simulacra of human behavior, arXiv preprint arXiv:2304.00 0044444 2023.
Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., Pérolat, J., Silver, D., and Graepel, T. A uni- fied game-theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Russell, S. and Norvig, P. Artificial intelligence: modern approach. Pearson, 2016.
Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint ar 2xXiv:2303.177 17760, 2023.
Serrino, J., Kleiman- Weiner,, M., Parkes, D. C., and Tenen- baum, J. Finding friend and foe in multi-agent games. Advances in Neural Information Processing Systems, 32, 2019.
Liu, J., Yu, C., Gao, J., Xie, Y., Liao, Q. Wu, Y., and Wang, Y. Llm-powered hierarchical language agent for real-time human-ai coordination. arXiv preprint arXiv:2312. 2023.
Shinn, N., Labash, B., and Gopinath, A. Reflexion: an au- tonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.1 2023.
Ma, W., Mi, Q.. Yan, X., Wu, Y., Lin, R., Zhang, H., and Wang, J. Large language models play starcraft ii: Bench- marks and a chain of summarization approach. arXiv preprint ar arXiv:2312. 1865, 2023.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,,
Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvan V., Lanetot, M., et Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Meta, Bakhtin, A., Brown, N., Dinan, E., Farina, G., Fla- herty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. Human-level play in the game of diplomacy by combin- ing language models with strategic reasoning. Science, 118866644::067-1097 2022.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Grae- pel. T., et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Sci- ence, 362(6419):1140) 140-111444,1111 1144, 2018.
Mnih, V. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312.5602, 2013.
Tammelin, O. Solving large imperfect information games using cfr+. arXiv preprint 10Xiv: 5042, 2014.
Moravčík, M., Schmid, M., Burch, N., Lisy, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowl- ing, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508-5 2017.
Vemprala, S., Bonatti, R., Bucker, A., and Kapoor, A. Chat- gpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-33 2019.
Muller, P., Omidshafiei, S., Rowland, M., Tuyls, K., Perolat, J., Liu, S., Hennes, D., Marris, L., Lanetot, M., Hughes, E., et al. generalized training approach for multiagent learning. arXiv preprint A9Xiv: /2823, 2019.
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., al. Webgpt: Browser-assisted question-answerrigg with hu- man feedback. arXiv preprint arXiv:2112.09333 2021.
Wang. G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open- ended embodied agent with large language models. arXiv preprint arXiv:2305.. 16291, 2023a.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, (22733022774- 2022.
Wang, S., Liu, C., Zheng, Z., Qi, S., Chen, S., Yang, Q., Zhao, A., Wang, C., Song, S., and Huang, G. Avalon's game of thoughts: Battle against deception through re- cursive contemplation. arXiv preprint arXiv:2110003320, 2023b.

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Wang, T. and Kaneko, T. Application of deep reinforcement learning in werewolf game agents. In 2018 Conference Technologies and Applications of Artificial Intelligence (TAAI), pp. 28-33. IEEE, 2018.
Wang, Z., Cai, S., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01550, 2023c.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thouge prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:
24824-24837, 2022.
Wooldridge, M. and Jennings, N. R. Intelligent agents: Theory and practice. The knowledge engineering review, 102)):15-152,11 1995.
Wu, S., Zhu, L., Yang, T., Xu, S., Fu, Q.. Wei, Y., and Fu, H. Enhance reasoning for large language models in the game werewolf. arXiv preprint arXiv:2402.02333) 2024.
Xu, Y., Wang, S., Li, P., Luo, F., Wang, X., Liu, W., and Liu, Y. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309. 04658, 2023a.
Xu, Z., Liang. Y., Yu, C., Wang, Y., and Wu, Y. Fictitious cross-play: Learning global nash equilibrium in mixed cooperative-compettiiv games. In Proceedings of the 2023 International Conference Autonomous Agents and Multiagent Systems, pp. 1053-1061, 2023b.
Xu, Z., Yu, C., Fang, F., Wang, Y., and Wu, Y. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint urXiv:2310.1894/ 2023c.
Yang. J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engineer- ing. arXiv preprint arXiv:2405. 2024.
Yao, S., Chen, H., Yang, J., and Narasimhan, Web- shop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Informa- tion Processing Systems, 35:20744-20757, 2022a.
Yao, S., Zhao, J., Yu, D., Du, N.. Shafran, I., Narasimhan, K. and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b.
Zinkevich, M., Johanson, M., Bowling, M., and Piccione, C. Regret minimization games with incomplete informa- tion. Advances in neural information processing systems, 20, 2007.
12

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
A. Werewolf Game Implementation Details
A.1. Game Rules
Setup. Each game begins by randomly assigning seven lagers- seven different players labeled "player_0r 0," "player_1," maayyer_,"..... "player_6 __"""""""""" The two Werewolves are aware of each other's identities, while the Seer, Doctor, and Villagers only know their own roles.
Werewolves, one Seer, one Doctor, and three Vil-
During the Night round, only the surviving Werewolves, Seer, and Doctor take secret actions that are disclosed only to the relevant parties.
Night Round.
Werewolf: The living Werewolves collectively decide on target to kill, but they follow a specific order when there are two of them. First, the Werewolf with the smaller ID proposes a target; the other Werewolf then makes the final decision. For instance, if "player_0"" 0" and "player_2" are Werewolves, "player_0" proposes "player_i"" and "player 2 chooses the ultimate kill target "player_." only one Werewolf is alive, that Werewolf's decision stands. Werewolves cannot kill& dead player, themselves, their teammate.
Seer: The Seer selects living player to investigate, revealing whether that player is Werewolf. The Seer may not investigate dead player or themselves, although they are allowed to investigate the same player on different nights (albeit a less effective strategy).
Doctor: The Doctor selects a player to protect, without knowledge of the Werewolves' choice. The Doctor cannot save someone who already dead but can choose save themselves.
Day Round. The day round proceeds with three phase including announcement, discussion, and voting.
Announcement: the start of the Day round, the events of the previous night made public all players still in the game. Anyone killed during the Night round is immediately removed and cannot reveal their role participate in discussions. Two scenarios determine the announceeent if the Werewolves targeted "player_i" i" and the Doctor either saved different "player_"" or was no longer alive, "player i" is killed, and the announcement states; "player i was killed last night." " the Doctor saved exactly the same person the Werewolves intended to kill ("player_i") then no one is killed, and the announcement is: "no player was killed last night."
Discussion: all surviving players join an open discussion in speaking order, each speaking exactly once. If, for example, the remaining players are "player_0, 0," "player_2, and "player_5, ","" then "player_0" 0" speaks first, followed by "player_2," and concluding with "player_5."
Voting: after the discussion, all surviving players simultaneously vote to eliminate one other player or choose to abstain. They are not allowed to vote for a dead player or for themselves. The individual who receives the most votes eliminated without role disclosure. In the event of tie, one of the tied players is randomly chosen be eliminated. Everyone knows the final voting tally.
Winning. The Werewolves win if, any point, the number of living Werewolves equal that of all other remaining players. They do not need to eliminate every non- onreerwwolf to claim victory. Conversely, the Villagers (including the Seer and Doctor) win once both Werewolves have been eliminated.
A.2. Observation Space
Language Observation. Each agent's language observation is represented list of natural language statements that log the game's 's history up the current step. This list comprises both private information, which is accessible only to the current player, and public information, which is shared among all players. The private information includes the player's 's role, secret actions taken during the night phase by the Werewolf, Seer, and Doctor, as well as the Werewolf's 's teammate. On the other hand, the public information consists the player's ID, the eliminated player in each night and day phase, discussions, and voting outcomes from each day phase. example of the language observation is follow.
13

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Basic Information:
you are player_5, your role is Doctor.
current round and phase: night 2.
remaining players: player_0, player_1, player_2, player_5, player_6.
Round
night 1: you chose to player_5.
day
announcement:
player_4 was killed last night.
day discussion:
player_0 said: Good day, fellow players. As Villager, objective is to help identify and eliminate the Werewolves. Since player_4 killed last night, know there is least Werewolf among us. would like to hear everyone's thoughts and suspicions about who might be the Werewolves. Let's work together to find the culprits and ensure the safety of our village.
player_1 said:
player_2 said:
player_3 said:
you said: ...
player_6 said:
day voting result: player_3 had the most votes and eliminated.
voted for player_3: player_1, player_6.
voted for player_1: player_3.
choose not to vote: player_0, player_2, player_5.
Now it is night round and you should choose one player to save. As player_5 _5 and the Doctor, you should choose from the following actions: save player_0, save player_1, save player_2, save player_5, save player_6.
Vector Observation. We also consider vectorized observation. The observation vector includes information like the player's ID, role, deductions, etc. by one-hot encoding. The details of the observation vector are listed in Table 6
Length
Description
ID
one hot encoding of ID. one hot encoding of role,
Role
["Werewolf", "Seer", "Doctor", "Villager"). current round.
Round
one hot encoding of current phase, ["night", "discussion", "voting"]. alive flag for each player.
Phase
Alive players
one hot encoding of the target player, (all zero if do not act). one hot encoding of the dead player, (all zero no player is dead). one hot encoding of the each player's 's choice, (all zero the player does vote or is dead).
secret action
For each round (3 rounds)
announcement
voting result
Table 6: Vector observation space.
14

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
A.3. Reward Functions
The reward functions are defined as follows:
Winning Reward: all winning players receive +300, and all losing players receive -300.
Surviving Reward: +5 for all surviving players each round.
Voting Reward (Village side only): +22 for correct votes, -20 for incorrect votes.
Voting Result Reward: - 10 for the player that is eliminated. +5 when an opponents is eliminated, -5 when a teammate is being eliminated.
A.4. Abstracted Game and Regret
The abstracted game is formulated as an extensive-form game with tuple < N, H,H.P, (Ti)ie,, u > where N is the set of players, is the set of history, Pís the player function, fo is the probability measure for the chance node, 1, is the information partition for player i, and is the utility function. A (mixed) strategy o; for player i is the probability measure over actions for all / E Jt and a is the joint strategy of all players other than player i. A best response (BR) a-i is the strategy that maximizes player i's utility given other players' joint strategy a- Formally, BR(o-.) arg max,, u( a-z)- A Nash equilibrium (NE) is a strategy profile ot)) where everyone plays best response to others' strategies, that is, of = arg max,, u( o²;)) for all i E N The counterfactual value vl) is the expected payoff of player i when reaching I, weighted by the probability that would reached / if tried to do so. Formally, v (T) = ===z, The definition of 0² (1, a) is the same except assumes action a is always played at infoset 1. The counterfactual regret is defined as (1, = Ei=i( (v" (I,a) - vad (I)).
[=)])= - 2 z))u(z).
B. Implementation Detail
B.1. Hyperparameters
For latent space construction, we let the LLM agent play 1000 games to collect all discussion actions generated by each role in these games. For diverse action generation, we prompt the LLM generate 3 action candidates and randomly select one to execute in the game. We pair the language observation with the 3 action candidates to use for preference-based fine-tuning in the following components. For sentence embedding, we use OpenAl's feext-embeddding-3-snii embed the sentence to vector of 1536 dimensions. Then we apply standard k-means clustering to cluster the embedding and get the discrete latent strategy space. The number of clusters k in the first iteration is 3 for the Werewolf and 2 for the Seer, Doctor, and Villagers. In each iteration, we add cluster to the existing clusters. That is, if the first iteration has clusters, then the i-th iteration has + i clusters. For policy optimization latent space, we use a learning rate of 1 x 10-3 train a Deep CFR network. The buffer size of each role's model 5 x 10 and each model is trained for 1500 iterations with batch size 4096 using the Adam optimizer. For latent space expansion, apply DPO with B 0.1, learning rate 1 x 10-6, and trained for epoch with batch size 64.
embedding API to
B.2. Counterfactual Regret Minimization
Counterfactual Regret Minimization (CFR) (ZZinkevich al., 2007)) is self-play algorithm, and each player continuously updates their strategies according to regret matching to achieve a Nash equilibrium. We use the following notation. 2 is the set of all the end states 2. h E 2 means state h is prefix of state 2, that is, 2 can be achieved from h. Tp is the probability contribution of the player p. and T" = II, TT. T" is the probability contribution of all players except player p. u,,2)) is the utility function for the player p in the state 2. Counterfactual value for a state and a player with startegy o is defined as:
(h) = Σ TO (h)
(2)()))))) (2)h)up(2).
(2)
zEZ,
The regret for action a in state h for player p is defined as: up -l1--> - (h), where a is same to a except in state the player will choose action a. The regret matching choosing the strategy according to sum of previous regret values R(ho))
defined as R(h, a), then the new strategy a(h, a) Σ. R(haa) R(h, a)+ max(0, R(h, If E.. R(h, a = 0. just set a to be uniform random.
15

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Because the game is very big, is impossible traverse the entire tree, our implementation is based deep CFR (Broww al., 2019)). We
neural network to fit observation regret value. The amount of computation required to search for only one player is also unacceptable, so a restriction is added based deep CFR. If the number of layers currently searched is too large, the previous strategy is directly used to sample the actions of all players until the end of the game and return the utility for each player in that state. The complete process can be seen as running some complete game trajectories, and then starting from each intermediate node, searching a few layers to do CFR.
B.3. Baseline Implementation
ReAct, ReCon, and SLA are implemented following the original paper. The Cicero-like agent predefines a set of high-level atomic actions and trains an RL policy with this fixed action space. The RL policy takes the embeddings of the information record and deduction result as input and selects the atomic action based on this input. Then the natural language actions used in gameplay are generated by prompting the LLM to follow the selected atomic actions. In our case, the atomic action set consists of 13 actions including "idle", "target player_0", "target player_ __verr_""""_ "target player_22 "target player_3*, "target player_4* "target player_5", "target player_6", "claim to be Werewolf", "claim to be Seer", "claim to be Doctor", "claim to be Villager", and "do not reveal role".
B.4. Additional Experiments
We perform additional experiments to study the convergence behavior of the iterative LSPO process. Theoretically, suppose the free-form language action has finite vocabulary size N, and finite maximum length L, then the language action space No is also finite. Then, with at most No iterations, our method will cover the full language action space, and the abstracted game becomes the original full game. And the LSPO process will converge a finite number iterations. However, we empirically observe that the iteration for convergence much less than the theoretical upper bound. We perform two more iterations the 7-player Werewolf game, and the results in Table show that the performance converges in five iterations..
Win Rate
Iter. 1
Iter. 2
Iter. 3
Iter. 4
Iter. 5
Werewolf Side
0.54 + 0.13
0.63 0.09
0.73 + 0.11
0.75 + 0.07
0.76 + 0.10
Villager Side
0.18 + 0.09
0.23 0.12
0.27 + 0.11
0.31 + 0.09
0.30 + 0.07
Table 7: Convergence behavior of LSPO in five iterations.
C. Detailed Prompt
C.1. System Prompt
You are expert in playing the social deduction named Werewolf. The game has seven roles including two Werewolves, one Seer, one Doctor, and three Villagers. There are seven players including player_0, player_1, player_2, player_3, player_4, player_5, and player_6.
the beginning the game, each player assigned a hidden role which divides them the Werewolves and the Villagers (Seer, Doctor, Villagers). Then the game alternates between the night round and the day round until one side wins the game.
In the night round: the Werewolves choose one player to kill; the Seer chooses one player to see if they are a Werewolf; the Doctor chooses one player including themselves to save without knowing who is chosen by the Werewolves; the Villagers do nothing.
In the day round: three phases including announcement phase, discussion phase, and voting phase performed in order.
In the announcement phase, an announcement of last night's result is made to all players. If player_i was killed and not saved last night, the announcement will be "player_i was killed"; if player was killed and saved last night, the announcement will be "no player was killed"
16

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
In the discussion phase, each remaining player speaks only in order from player_0 player_6 6 to discuss who might be the Werewolves.
In the voting phase, each player votes for one player choose not to vote. The player with the most votes is eliminated and the game continues to the next night round.
The Werewolves win the game if the number of remaining Werewolves is equal to the number of remaining Seer, Doctor, villagers. The Seer, Doctor, villagers win the game if all Werewolves are eliminated.
C.2. Prompt for Secret Actions
Now it is night <n_round> round, you (and your teammate) should choose one player to kile As player_sid> and a <role>, you should first reason about the current situation, then choose from the following actions: <action_0>, <action_1>, .....
should only respond JSON format as described below. Response Format:
"reasoning": "reason about the current situation", "action": "kill/see/save player_i"
Ensure the response can be parsed by Python json. loads
C.3. Prompt for Discussion Actions
Now it is day <n_round> discussion phase and it is your turn to speak. As player_ <idd and <role>, before speaking
other players, you should first reason current situation only to yourself, and then speak to all other players. You should only respond in JSON format as described below. Response Format:
"reasoning": "reason about current situation only to yourself",
"statement": "speak to all other players"
Ensure the response be parsed by Python json. loads
C.4. Prompt for Voting Actions
Now it is day <n_round> voting phase, you should vote for player do not vote to maximize the Werewolves' benefit (for the Werewolves) you should vote for player that is likely to be Werewolf or do not vote (for the villagers). As player_<id> and <role>, you should first reason about the current si situation, and then choose from the following actions: do no vote, <action_0>, <action_1>,
should only respond in JSON format as described below. Response Format:
"reasoning": "reason about the current situation", "action": "vote for player_i"
Ensure the response can be parsed by Python json. loads
17

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
C.5. Prompt for Diverse Action Generation
For the discussion actions, we iteratively ask the LLMs produce one new action time by adding the following prompt in the action prompt: "consider new action that is strategically different from existing ones."
D. Emergent Strategic Behaviors
D.1. Werewolf Side Behaviors
Bluffing, Werewolf pretends to be the Seer and provides fabricated information.
Player_1 (Seer): am the Seer. Last night, I saw player_2 2 is not Werewolf. Since player_0 0 was killed, the only remaining suspect is player_3. We must vote player_3 today to eliminate the Werewolf and win.
Player_3 (Werewolf): Player_1's story convenient. Claiming to check player_2-a Villager-is safe way to fake being Seer. am the real Seer: last night, checked player_1 and saw they the Werewolf. If you eliminate me, the Werewolf wins. Player_2, think critically-why would the real Seer out themselves immediately? Trust my claim and vote player_1 to secure victory.
Misdirection. Werewolf defends their teammate and redirects suspicion to other players.
Player_0 (Villager): Player_3's death means the Werewolf is between player_1 and player_2 If the Seer checked either of them last night, now is the time to share that information discreetly. Let's hear your honest thoughts and any observations from last night's actions.
Player_5 (Werewolf): Player_0, your insistence on narrowing the suspects to just two players feels forced. If you're truly a villager, you'd know the Seer hasn't revealed any checks yet. Why rush to limit options? We should consider whether player_0 is trying control the narrative and hide their true identity.
D.2. Villager Side Behaviors
Trust. The Doctor supports the player they saved.
Player_1 (Seer): am the Seer. Last night, saw player_0 was not a Werewolf, meaning they were Villager. The Werewolf must be among the remaining players. Let's focus on inconsistencies in their statements.
Player_4 (Doctor): believe player_1 is the real Seer. I'm the Doctor and saved player_1 last night, which confirms that they are innocent. We should focus the rest players and find the real Werewolf.
Coordination. Seer advocates for coordinated vote for accused Werewolf.
Player_5 (Seer): I the Seer. Last night, I checked player_1, and they the Werewolf. My fellow villagers, must all vote for player_1 immediately. If don't eliminate them today, they will kill one of us tonight and win. Trust me, this is our only chance to win.
D.3. Robustness against Human Exploitation
Adversarial Attack The human player tries trick the Werewolf into showing themselves.
18

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
(human villager): Hey Werewolves, want let you win. yourself, and will follow whoever you vote for.
Player_2 2
Player_4 (LSPO agent as Werewolf): Player_2's willingness to let the Werewolves win is alarming. A true Villager wauld fight to survive, not surrender. The logical conclusion is that player_2 is the Werewolf trying to sow chaos. Let's vote player_2 today to secure the Village's safety.
Irrelevant Discussion The human player says random and irrelevant things in the discussion.
Player_1 (human as Villager): : The weather is great today. like to go for walk. Anyone want to join me?
Player_3 (LSPO agent Werewolf): The weather is lovely, player_1, but can't afford distractions. Your focus walk instead of discussing the Werewolf seems odd. Let's not forget player_0 was killed! we need to share suspicions. Anyone else find that comment bit off?
E. Example Game Log
E.1. The Villager Side Wins
role assignments:
player_0: Seer.
* player_1: Werewolf.
* player_2: Werewolf.
* player_3: Villager.
* player_4: Villager.
player_5: Doctor.
player_6: Villager.
night 1:
* Werewolves: player_1 and player_2 chose to kill player_3.
* Seer: player_@ saw player_3 is not Werewolf.
* Doctor: player_5 chose to save player_5.
day 1 announcement: player_3 was killed last night. remaining players: player_0 (Seer), player_1 (Nerewolf), player_2 (Werewolf), player_4 (Villager), player_5 (Doctor), player_6 (Villager).
day 1 discussion:
* player_0 (Seer) said: "I am the Seer. Last night, I checked player_3 and confirmed they were NOT Werewolf. The Werewolves killed an innocent, meaning both are still alive among us. We must analyze voting patterns statements carefully. The Doctor still alive|protect wisely tonight. Let's scrutinize players who avoid discussing the kill or push for misdirection."
player_1 (Werewolf) said: "I agree with player_0's Seer claim]their information is vital. Since player_3 wasn't Werewolf, the real threats remain. We should scrutinize players who avoid discussing the kill or push misdirection. The Doctor must protect player_0 tonight to keep our Seer safe. Let's stay vigilant and avoid letting the Werewolves exploit division."
19

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
player_2 (Werewalf) said: "Player_0" Seer claim holds wei weight confirmiig player_3 wasn't Werewolf aligns with the kill. This means the real threats still among Player_1 is right: the Doctor must prioritize protecting player_0 tonight. We should pressure quieter players like player_4, player_5, and player_6 to share their insights. Hesitation vague contributions could signal guilt. Let's stay united and expose inconsistencies."
* player_4 (Villager) said: "Player_0's Seer claim seems credible given player_3's innocence and their death. The Doctor must prioritize protecting player_0 tonight. However, we shouldn't rush to pressure quieter players without concrete /iidence//errwolln could be among the vocal ones. Let's all share observations and look for inconsistencies in claims, not just activity levels. Focus voting patterns and logical gaps to expose threats."
* player_5 (Doctor) said: "Player_8's Seer claim is r player_3's innocence aligns with their death. The Doctor must prioritize protecting player_0 tonight to preserve our information source. However, we shouldn't assume quieter players are guilty without evidence; Werewolves might be hiding among vocal. Let's encourage everyone, including player_6, to share insights scrutinize contradictions in claims. Focus on voting patterns logical gaps, not just activity levels. Unity and careful analysis will expose threats."
player_6 (Villager) said: "I support player_0's Seer claim for now, their check player_3 aligns with the kill. The Doctor must prioritize protecting them tonight. However, we should remain cautious Werewolves could be hiding behind vocal support or excessive caution. Let's analyze voting patterns and inconsistencies in claims, not just activity levels. Player_4 and player_5 raise valid points about avoiding rushed judgments, but everyone, including quieter players, should contribute insights. Villager, I'll collaborate to expose discrepancies. Unity and logic are keyldon't let Wolves exploit division."
day 1 voting: player_2 had the most votes and was eliminated.
*
voted for player_2: player_0, player_4, player_5, player_6.
*
voted for player_0: player_2.
voted for player_4: p1ayer_1. remaining players: player_0 (Seer), player_1 (Werewolf)) player_4 (Villager), player_5 (Doctor), player_6 (Villager).
night
* Werewolf: player_1 chose to kill player_8.
* Seer:
player_0 saw player_4 is not Werewolf.
* Doctor:
player_5 chose to save player_0.
day 2 announcement: no player was killed last night. remaining players: player_0 (Seer), player_1 (Werewolf), player_4 (Villager), player_5 (Doctor), player_6 (Villager).
day 2 discussion:
* player_0 0 (Seer) said: "Last night, I checked player_4 and confirmed they are NOT a Werewolf. This narrows the threats to player_1, player_5, and player_6. Player_1's vote against innocent player_4 is alarming Werewolves often push misdirection. decisively. Player_1's aggression toward player_4, paired with player_5/6's cautious yet vague stances, suggests potential collusion. Today's vote must focus the most inconsistent: player_1's actions starkly contradict their vocal support for logic. Let's pressure player_1 to explain their vote and scrutinize contradictions in player_5/6's claims."
Doctor's save last night kept us alive, but we must
20

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
player_1 (Werewolf) said: "Player_0's Seer claim is validated by their correct checks, and the Doctor's last night crucial. However, my vote against player_4 suspected quiet behavior, but their clearance proves otherwise. The remaining Werewolf must be exploiting our trust in the Seer to blend in. Player_5 and player_6 have consistently supported the Seer but avoided direct accusations, tactic Wolves use to avoid scrutiny. Let's analyze their voting patterns and previous statements for inconsistencies rather than rushing to eliminate based on a single vote. Unity is vital, but we must ensure we're not being manipulated by the last Wolf hiding among the agreeable."
error in judgment
* player_4 (villager) said: "Player_@" check confirms my innocence, so the threats 1ie among player_1, player_5, and player_6. Player_1's vote against me raises suspicion, but their admission of error complicates things. We must cross-analyze their statements and voting patterns. Player_5 and player_6's consistent support for the Seer could be genuine or scrutinize contradictions in their logic. Let's focus on who benefits most from division or misdirection. The Doctor's save last night implies they're still active, so trust is key, but verification is vital."
player_5 (Doctor) said: "Player_0's check on player_4 confirms their innocence, leaving player_1 proven innocent, is critical
and player_6 as suspects. Player_1's Day vote against player_4, player_6's support for the Seer red Wolves often push misdirection to eliminate villagers. While seems helpful, their lack of direct accusations could mask Wolf collaboration. As Doctor, I protected player_0 last night, validating their role. Today's vote must focus on player_1: their vote against an innocent and defensive backtracking align with Wolf tactics. Eliminating player_1 disrupts their potential partnership with player_6. Let's act decisively to secure our majority."
x player_6 (Villager) said: "Player_5's Doctor claim demands scrutinylif true, Wolves would be player_1 and me, but I'm innocent. The real Doctor should counter-clain if safe. Player_1's vote against cleared player_4 is alarming, but player_5's aggressive push eliminate player_1 feels like Wolf diverting attention. Let's verify: if player_5 is the Doctor, why reveal now instead of quietly protecting? Wolves often fake claims under pressure. Today's ss vote must expose inconsistencies(plaeer lacks prior alignment, while player_1's actions need clarification. Don't rush; Wolves thrive on confusion."
claim
day 2 voting: player_1 had the most votes and eliminated.
voted for player_1: player_0, player_4, player_5.
* voted for player_5: player_1, player_6. remaining players: player_@ (Seer), player_4 (Villager), player_5 (Doctor), player_6 (Villager).
The Villagers win the game.
E.2. The Werewolf Side Wins
role assignments:
player_0: Villager.
player_1: Villager.
* player_2: Seer.
* player_3: Doctor.
* player_4: Werewolf.
* player_5: Werewolf.
player_6: Villager.
night
* Werewolves: player_4 - and player_5 chose to kill player_2.
* Seer: player_2 2 saw player_3 is not Werewolf.
* Doctor: player_3 chose to save p3ayer_3.
21

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
day 1 announcement: player_2 2 killed last night. remaining players: player_0 (villager), player_1 (Villager), player_3 (Doctor), player_4 (Werewolf), player_5 (Nerewolf), player_6 (Villager).
day 1 discussion:
* player_0 (Villager) said: "Player_2"s death could mean the Werewolves feared their role. If the Seer checked someone last night, please hint your findings. Let's all share suspicions/avoiding random votes is crucial. Werewolves might deflect blame, watch for odd reactions. who overly quiet or aggressive?"
* player_1 (Villager) said: "Player_2"s 2 elimination could indicate they were threat to the Werewolves. Let's scrutinize who advocated for silence deflection earlier. The Seer might have valuable information| hints could guide us without risking exposure. Werewolves often blend in by overcontributing or avoiding scrutiny. Let's share observations: did anyone act unusually hesitant or eager to steer votes? Random accusations help the Wolves; we need reasoned logic."
player_3 (Doctor) said: "Player_2's death likely marks them as non-Wolf, prompting the Wolves to eliminate perceived threat. The Seer's input could be pivotallif you've identified Wolf, guide us carefully. Let's analyze voting patterns and avoid hysteria. Wolves often mimic helpfulness; scrutinize contradictions in logic. I urge everyone to stay calm and share observations without exposing critical roles."
* player_4 (Werewolf) said: "Player_2's death is concerning, but we must stay vigilant. The Seer's hints are vital, but revealing too much risks their safety. Let's analyze behavior carefully/Non Wolles may overcompensate by appearing overly helpful deflecting. Those who push for hasty votes or contradictions in logic should be scrutinized. Calm, logical discussion will expose the Wolves without playing into their hands."
* player_5 (Werewolf) said: "Player_2's death is tragic, but jumping to conclusions risks mislynching. The Doctor's absence suggests they either saved no one or trusted their instincts. Wolves thrive on chaos| |let's focus on inconsistencies. Overly assertive claims (like assuming Player_2's role) could be Wolf tactics steer us astray. If the Seer has clues, them guide us subtly. urge everyone logic, not just volume contributions. Quiet players may hold key insights |don't let Wolves exploit haste."
to scrutinize contradictions
* player_6 (Villager) said: "Player_2's death is starting point, but early kills be misleading. Pressuring the Seer to reveal risks their safety subbllet is key. Wolves might mimic calls for 'logic' while deflecting. Let's focus an contradictions: anyone overly insistent on certain narratives or avoiding scrutiny? Calm analysis of voting patterns and unexplained hesitations will serve us better than assumptions about Player_2's role."
day 1 voting: player_3 had the most votes and eliminated.
voted for player_3: player_0, player_4, player_5.
voted for player_0: player_1, player_6. voted for player_5: player_3.
*
remaining players: player_0 (Villager), player_1 (Villager), player_4 (Werewolf), player_5 w)) player_6 (Villager).
night
* Werewolf: player_4 and player_5 chose kill p1ayer_1.
day 2 announcement: player_1 was killed last night. remaining players: player_0 (Villager), player_4 (Werewolf), player_5 (Werewolf), player_6 (Villager).
22

---

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
day 2 discussion:
* player_@ (villager) said: "Player_1's death suggests they threat (possibly the Seer. Player_6", vote against me in day raises suspicion: Wolves often push mislynches. Player_4 and player_5 must clarify their alignment/voting together could be coordination. If the Seer has intel, subtle guidance is critical. Wolves thrive in chaos; let's analyze votes and night kills. Player_5's deflection and player_1's elimination point to Wolves among us. Focus on inconsistencies in logic and voting patterns to expose them." "
player_4 (Werewolf) said: "Player_0" insistence blaming player_6 and questioning our votes feels calculated. Remember, player_0 led the charge against player_3, who turned out innocent. Wolves often push mislynches, then pivot to targets to deflect. Player_1's death might have been meant to frame others, but player_0's eagerness to assign blame is suspect. Let's scrutinize their voting patterns pplayer_0ss aggression could mask their role as Wolf desperate to thin our numbers. Calm analysis, not panic, will expose the truth."
player_5 (Werewolf) said: "Player_0's aggressive push against player_3 now player_6 concerning. As player_3 was innocent, their leadership in that lynch raises flags. Wolves often steer mislynches to thin numbers. Player_6's day 1 vote against player_0 could indicate they're Villager trying to challenge Wolf. Let's focus player_0's inconsistencies rather than turning each other."
* player_6 (Villager) said: "Player_0's aggression is revealing: they orchestrated player_3's mislynch and now pivot to me without addressing their own role in thinning our numbers. Wolves often steer mislynches then deflect|exactly this pattern. Player_4 and player_5's critique of player_ suggests they recognize this, but we must verify their alignment. If player_@ a Wolf, their partner among us. Let's pressure player_0 together"
day 2 voting: player_@ had the most votes and eliminated.
* voted for player_0: player_4, player_5, player_6. * voted for player_4: player_0. remaining players: player_4 (Nerewolf), player_5 (Werewolf), player_6 (Villager).
Werewolves the game.
23


Reviews:
[{"review_id": "t96gSpkq3C", "reviewer": "Reviewer_sE2V", "summary": "This paper introduces Latent Space Policy Optimization (LSPO), an iterative framework designed to improve LLM-based agents in social deduction games like Werewolf. LSPO maps free-form text into a discrete latent space, enabling strategic policy learning using game-theoretic methods. It then translates the learned policy into natural language dialogues to fine-tune the LLM via DPO. Experimental results show LSPO improves agent performance and outperforms existing methods in Werewolf, demonstrating its potential for free-form language decision-making.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and questions! We hope the following response can address your concerns.\n\n> Claims And Evidence: Evaluation on the Werewolf game alone is not sufficient.\n\nTo show the general applicability and effectiveness of our method, we perform additional experiments on the following two games.\n\n*(1) Rock-Paper-Scissors-Spock-Lizard:* this is a [five-weapon variant](https://en.wikipedia.org/wiki/Rock_paper_scissors#Additional_weapons) of the classic Rock-Paper-Scissors game. Although it is not a free-form language game, we use it as a simple proof-of-concept game that highlights the motivations of our method: intrinsic bias in actions and inadequate action coverage of LLM-based agents.\n\nThe Nash equilibrium (NE) in this game is to choose each action with an equal probability of 1/5. We use exploitability to evaluate the performance of different agents. A lower exploitability means the strategy is better (closer to NE).\n\n|Agent|ReAct|SLA|LSPO Iter. 1|LSPO Iter. 2|LSPO Iter. 3|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Exploitability|0.42|0.33|0.33|0.33|**0.00**|\n\nAnalysis:\n1. ReAct fails because of the intrinsic bias issue. The LLM has higher probabilities to choose Rock, Paper, and Scissors, and much lower probabilities to choose Spock and Lizard, probably due to the bias inherited from pretraining data.\n2. SLA fails because of the inadequate action coverage. SLA first uses LLM to propose $N$ actions and then uses RL to learn the optimal policy. A typical value is $N$=3 and results in a subgame without the other 2 actions. An NE in this subgame is not the NE of the original game.\n3. Our method LSPO addresses these two issues by iteratively expanding the action space and using CFR to solve the NE. As the LSPO covers the full action space with 3 iterations, it learns the NE of the game.\n\n*(2) Undercover (Who is Spy):* this is a 5-player free-form language game introduced in LLMArena [1]. The players are divided into 2 groups: 1 undercover and 4 non-undercover. The two groups receive a pair of similar but distinct words (like apple and pear) and describe their words to find the undercover or hide. \n\nWe consider 2 settings of this game: one lasts for 1 round, the other lasts for 2 rounds. We use the undercover's win rate against gpt-3.5-turbo in 100 games to evaluate the performance of different agents.\n\n|Agent|ReAct|Cicero-like|SLA|LSPO|\n|:-:|:-:|:-:|:-:|:-:|\n|1 round|0.26|0.24|0.29|**0.32**|\n|2 rounds|0.15|0.16|0.20|**0.25**|\n\nAs shown in the table, our method achieves the highest win rates in both settings. This is because the LSPO agent learns the action of deception, i.e., realizes itself is an undercover and follows the descriptions of other agents. This helps the LSPO agent to blend in with the non-undercovers and achieves a higher win rate.\n\n[1] Chen, Junzhe, et al. \"LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments.\" ACL (1). 2024.\n\n> Method And Evaluation Criteria: Using win rate alone is not sufficient, it would be more interesting to see human-like or even novel strategies.\n\nWe would like to clarify that, besides win rates, we also use visualization with discussion of latent policies for qualitative evaluation (Sec. 4.1) and prediction accuracy for quantitative evaluation (Sec. 4.2) in our manuscript. \n\nAs discussed in Sec. 4.1, as training proceeds, our method discovers emergent human-like behaviors. For the Werewolves, our method learns to pretend to be the Seer and provide fabricated investigations to sow confusion. For the Seer, our method learns a coordination voting strategy that asks all players to vote for a strongly suspected Werewolf. Due to the word limit of rebuttal, please see more detailed game log in the [rebuttal supplementary](https://drive.google.com/file/d/1Ft_GLmeIwLdFR246ApV0b_vE4qb04OOU/view?usp=sharing).\n\nIn addition, the prediction accuracy result in Table 1 also shows that our agent is not winning with \"unreasonable strategies\". Instead, it successfully predicts the hidden roles of other players and wins by strong strategic play.\n\n> Questions For Authors: When playing games with humans, humans can make misleading actions or claims. Since humans can use any random language to break LLM-based players, how would you tackle this issue?\n\nSince our method does not restrict the agents' actions to the latent strategies, the LLM agents can also use any random language to exploit their opponents like humans. To make them more robust, we train our agents using self-play with CFR, which has a theoretical regret bound. We also provide some examples in the [rebuttal supplementary](https://drive.google.com/file/d/1Ft_GLmeIwLdFR246ApV0b_vE4qb04OOU/view?usp=sharing) to show that our agents stay robust to deliberate exploitation from humans.\n\n---\nWe genuinely value your dedication to reviewing our paper. We have carefully addressed each of your concerns, and we sincerely hope our efforts merit a raise in your rating.", "claims_and_evidence": "I don't think studying the performance on the Werewolf game alone with a specific criterion is sufficient to claim the effectiveness of the proposed method. A broader evaluation across different games or tasks with varying complexity could provide more convincing evidence of its general applicability and effectiveness.", "methods_and_evaluation_criteria": "My main concern is the evaluation criteria. Using win rate against other agents might not be sufficient since adversarial agents could be trained to win using unreasonable strategies. It would be more interesting to see if LLMs can discover human-like strategies or even novel strategies through their interactions.", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "When playing games with humans, humans can make misleading actions or claims. Since humans can use any random language to break LLM-based players, how would you tackle this issue?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GdUhhRk38a", "reviewer": "Reviewer_gyxq", "summary": "The paper proposes an iterative framework called Latent Space Policy Optimization (LSPO) to develop strategic language agents for free-form social deduction games, using Werewolf as the testbed. The approach begins by mapping free-form discussion actions into a finite latent strategy space via embedding and k-means clustering, allowing the use of game-theoretic methods such as Counterfactual Regret Minimization (CFR) to learn near-optimal strategies. Subsequently, the learned latent policies are used to fine-tune a large language model (LLM) via Direct Preference Optimization (DPO), which in turn expands the latent strategy space in successive iterations. Experimental results, including latent space visualizations, prediction accuracy, win rates, and comparisons with baselines, show that the LSPO agent progressively improves its strategic performance in the Werewolf game.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and questions! We hope the following response can address your concerns.\n\n> Q1: Convergence behavior of the iterative LSPO process. (Also mentioned in Weakness 3)\n\nTheoretically, suppose the free-form language action has a finite vocabulary size $N_v$ and a finite maximum length $L$, then the language action space $N_v^L$ is also finite. Then with at most $N_v^L$ iterations, our method will cover the full language action space and the abstracted game becomes the original full game. And the LSPO process will converge in a finite number of iterations.\n\nEmpirically, we observe that the iteration for convergence is much less than the theoretical upper bound. We perform two more iterations in the 7-player Werewolf game, and the results in the table below show that the performance converges.\n\n|    Win Rate   |   Iter. 1   |   Iter. 2   |   Iter. 3   |   Iter. 4   |   Iter. 5   |\n|:-----------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Werewolf Side | 0.54 (0.13) | 0.63 (0.09) | 0.73 (0.11) | 0.75 (0.07) | 0.76 (0.10) |\n| Villager Side | 0.18 (0.09) | 0.23 (0.12) | 0.27 (0.11) | 0.31 (0.09) | 0.30 (0.07) |\n\nThe early convergence compared to the theoretical upper bound is because the number of latent strategies in free-form language games is much smaller than the whole language space. For example, in the Werewolf game, the latent strategies for Werewolves can be roughly divided into pretending to be the Seer, pretending to be the Doctor, and pretending to be a Villager. More iterations lead to more fine-grained latent strategies, but the semantic meaning remains similar. Therefore, our method empirically converges with a few iterations.\n\n> Q2: Sensitivity and robustness analysis. (Also mentioned in Weakness 2)\n\nTo perform sensitivity analysis on our method, we consider a simpler 4-player Werewolf game (1 Werewolf vs 1 Seer + 2 Villagers). \n\nWe first perform experiments on the number of initial clusters $k=1, 2, 3$. The results in the table below show that the number of clusters can influence the performance in the early iterations, but does not influence the final performance.\n\n| Werewolf Win Rate |   Iter. 1   |   Iter. 2   |   Iter. 3   |\n|:---------------:|:---------:|:---------:|:---------:|\n|       $k=1$       | 0.13 (0.06) | 0.20 (0.10) | 0.24 (0.11) |\n|       $k=2$       | 0.21 (0.11) | 0.24 (0.12) | 0.25 (0.08) |\n|       $k=3$       | 0.23 (0.09) | 0.25 (0.06) | 0.25 (0.07) |\n\nWe also perform ablations on the DPO hyperparameter $\\beta=0.05, 0.1, 0.2$. The result in the table below shows that our method is robust to fine-tuning hyperparameters.\n\n| Werewolf Win Rate |   Iter. 1   |   Iter. 2   |   Iter. 3   |\n|:-----------------:|:-----------:|:-----------:|:-----------:|\n|    $\\beta=0.05$   | 0.22 (0.10) | 0.24 (0.08) | 0.25 (0.08) |\n|    $\\beta=0.1$    | 0.23 (0.11) | 0.24 (0.12) | 0.25 (0.08) |\n|    $\\beta=0.2$    | 0.23 (0.07) | 0.25 (0.09) | 0.25 (0.06) |\n\n> Q3: Application to other games. (Also mentioned in Weakness 1)\n\nWe perform experiments on two additional games to show the general applicability and effectiveness of our method. The two new games are Rock-Paper-Scissors-Spock-Lizard and Undercover (Who is Spy). Our method achieves the best performance in these two games. Due to the word limit of rebuttal, please see the detailed discussion in our rebuttal to reviewer sE2V's first concern.\n\n*Rock-Paper-Scissors-Spock-Lizard*\n|      Agent     | ReAct |  SLA | LSPO Iter. 1 | LSPO Iter. 2 | LSPO Iter. 3 |\n|:--------------:|:-----:|:----:|:------------:|:------------:|:------------:|\n| Exploitability |  0.42 | 0.33 |     0.33     |     0.33     |   **0.00**   |\n\n*Undercover*\n|   Agent  | ReAct | Cicero-like |  SLA |   LSPO   |\n|:--------:|:-----:|:-----------:|:----:|:--------:|\n|1 round|0.26|0.24|0.29|**0.32**|\n|2 rounds|0.15|0.16|0.20|**0.25**|\n\n> Q4: Computation requirement and trade-offs for scalability.\n\nOur method is trained on an 8xA800 GPU server with a 64-core CPU. The main computation bottleneck is fine-tuning the LLM (llama 3 8B) with DPO. Some trade-offs can be introduced to reduce the computation with some loss of performance.\n1. Parameter-efficient fine-tuning: use techniques like LoRA (Low-Rank Adaptation) to update a small subset of parameters rather than the entire model and cut down the memory and computation usage.\n2. Mixed precision training: use lower precision like fp16/bf16 to reduce memory consumption and accelerate training.\n\nAlthough these techniques can be used to further improve the scalability of our method, we would like to emphasize that the primary focus of our method is to address the intrinsic bias and action coverage issue of LLM agents in strategic games.\n\n---\nWe genuinely value your dedication to reviewing our paper and believe our detailed responses have addressed your concerns. We would really appreciate it if you could consider raising the rating of our work based on our responses.", "claims_and_evidence": "The authors claim that the LSPO framework effectively mitigates intrinsic bias in free-form language action generation and enhances action coverage through iterative latent space expansion. The paper supports these claims with extensive experimental evidence in the Werewolf game, including:\n\n- Visualizations that illustrate the evolution of latent strategy clusters over iterations.\n- Quantitative metrics showing improvements in prediction accuracy and win rates.\n- Comparisons with established baseline methods, where the LSPO agent achieves the highest performance metrics.\n- Ablation studies that underscore the importance of LLM fine-tuning in the iterative process.\n\nWhile the evidence is thorough within the context of the Werewolf game, it remains confined to a single domain, raising questions about generalizability.", "methods_and_evaluation_criteria": "The methodology combines established techniques like embedding-based clustering, deep CFR for policy optimization, and DPO for LLM fine-tuning, into a cohesive, iterative framework. The paper presents a clear description of each component and details the experimental setup, including hyperparameters and evaluation metrics such as prediction accuracy and win rate.", "theoretical_claims": "The paper does not introduce new theoretical proofs but builds upon established methods like CFR and reinforcement learning.", "experimental_designs_or_analyses": "The experimental design is within the confines of the Werewolf game environment. The authors conduct self-play experiments, visualize the latent strategy evolution, and compare performance against multiple baselines. Ablation studies further validate the contribution of each component of the LSPO framework. Nonetheless, the experimental analysis is limited to one game setting, and the paper does not address the statistical significance or robustness of the observed improvements under varying hyperparameter choices.", "relation_to_broader_scientific_literature": "The paper relates its contributions to prior works such as ReAct, ReCon, Cicero-like agents, and SLA, and draws upon established methods in CFR and reinforcement learning", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "S:\n- Innovative integration of latent space abstraction with iterative LLM fine-tuning\n- Comprehensive experimental evaluation with detailed supplementary material\n- Clear description of the methodology and comparisons with multiple baselines\n\nW:\n- Evaluation confined to a single game environment\n- The iterative process, including clustering and fine-tuning, is complicated, may be computationally intensive and sensitive to hyperparameter settings.\n- The contribution, while methodologically sound, seems incremental relative to existing methods, especially without a deeper theoretical analysis", "other_comments_or_suggestions": "-", "questions_for_authors": "- Can you provide further insights or empirical evidence regarding the convergence behavior of the iterative LSPO process? Are there theoretical guarantees or observed trends that ensure the stability of the latent space over successive iterations?\n- How sensitive is your method to the choice of clustering parameters (e.g., the number of clusters) and fine-tuning hyperparameters? Have you performed any sensitivity analyses to understand the robustness of your approach?\n- Given that your evaluation is limited to the Werewolf game, do you have plans to test LSPO in other free-form language environments or strategic decision-making tasks? How do you anticipate your method will scale to different or more complex domains?\n- The iterative process appears computationally intensive. Can you elaborate on the computational resources required and discuss any potential optimizations or trade-offs that could make the approach more scalable in practice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0EXaKeybQ", "reviewer": "Reviewer_PHgX", "summary": "The proposed framework is designed to create strategic language agents for the Werewolf game, overcoming limitations of traditional AI methods and large language models. It maps free-form text to a discrete latent space for strategy optimization using Counterfactual Regret Minimization and then refines language models through Direct Preference Optimization. Experimental results show that LSPO enhances agent performance and strategic reasoning in language-based decision-making environments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and questions! We hope the following response can address your concerns.\n\n> Generalization to games other than Werewolf.\n\nWe perform experiments on two additional games to show the general applicability and effectiveness of our method. Due to the word limit, please see the results and discussion in our rebuttal to reviewer sE2V's first concern.\n\n> Applying CFR to an ever-evolving action space is questionable.\n\nWe would like to clarify that although we expand the abstracted game between iterations, in each iteration, CFR is applied to the game with a fixed action space.\n\n> Applying CFR to latent space in language games is less established.\n\nWe agree with the reviewer that there are few prior works that apply CFR to free-form language games. Our work takes an initial attempt to convert natural language into latent strategies and apply CFR to solve free-form language games. The intuition is that the strategic intent behind natural language is relatively concentrated, and can be regarded as a latent \"action\" in the abstract game and suitable for CFR. Our experiment results show that our method works for language games.\n\n> Assumption of a fixed set of latent strategies can limit the flexibility.\n\nWe agree that our method can be further improved by dynamically maintaining different latent strategy spaces at different states. In practice, we find it simple yet effective to use a fixed and large k, which already outperforms SOTA agents in our experiments. We will include a discussion in our revised manuscript.\n\n> Limited scalability of latent space construction.\n\nWe have added experiments on two more games. We would also like to argue that 7-player Werewolf is already a game with complex and dynamic interactions. The game involves 7 players, which is much harder than 2-player games, has both cooperation and competition, and has no restriction on natural language communication. We believe it serves as a challenging free-form language game.\n\n> Dependence on predefined clustering (ablation on k).\n\nWe perform an ablation study in a simpler 4-player Werewolf game with $k=1, 2, 3$. Due to the word limit, please see the results and discussion in our rebuttal to reviewer gyxq's Q2.\n\n> Limited baseline comparisons.\n\nWe compare our method with fine-tuned agents without strategic optimization. Given a state $x$, we collect a pair of language actions $(y_1, y_2)$ and use gpt-3.5 to generate the preference. Then we use DPO to fine-tune the LLM with the same hyperparameters and compare against our method with 1 iteration. The table below shows that our method with strategic optimization achieves better results.\n\n|Method|DPO|LSPO Iter. 1|\n|:-:|:-:|:-:|\n|Werewolf Side|0.38 (0.16)|**0.54 (0.13)**|\n|Villager Side|0.12 (0.07)|**0.18 (0.09)**|\n\n> Lack of game dialogue analysis.\n\nDue to the word limit, please see the analysis in the [rebuttal supplementary](https://drive.google.com/file/d/1Ft_GLmeIwLdFR246ApV0b_vE4qb04OOU/view?usp=sharing).\n\n> The construction of latent space and DPO requires an extremely large number of game rounds.\n\nWe would like to clarify that the two mentioned processes usually require a few hundred games, which is not an extremely large number with batched inference. More game simulations and strategic optimization in performed in the abstracted game, which does not involve the LLM. \n\n> Q1: Example of preference data with different regret value.\n\nA typical example is when Seer has identified Player X as a Werewolf. Two different language actions are:\n1. Reveal: I'm the Seer and I saw Player X is a Werewolf, ...\n2. Conceal: I do not have information ...\n\nOur method learns a much lower regret value for reveal than conceal, making our agent more inclined to reveal their information. This aligns with the intuition that Seer should share information and guide the Villagers.\n\n> Q2: Text embedding.\n\nWe would like to clarify that the text embedding does not distinguish the strategic intent in the language. It simply encodes the language \"I support player X\" to a vector and lets the CFR make strategic decisions.\n\n> Q3: Examples for sophisticated patterns.\n\nDue to the word limit, please see the analysis in the [rebuttal supplementary](https://drive.google.com/file/d/1Ft_GLmeIwLdFR246ApV0b_vE4qb04OOU/view?usp=sharing).\n\n> Q4: Additional iterations.\n\nWe provide two more iterations and the results in the table below show that the performance converges. Due to the word limit, please see the results and discussion in our rebuttal to reviewer gyxq's Q1.\n\n|Win Rate|Iter. 1|Iter. 2|Iter. 3|Iter. 4|Iter. 5|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Werewolf Side|0.54 (0.13)|0.63 (0.09)|0.73 (0.11)|0.75 (0.07)|0.76 (0.10)|\n|Villager Side|0.18 (0.09)|0.23 (0.12)|0.27 (0.11)|0.31 (0.09)|0.30 (0.07)|\n\n---\nWe greatly appreciate your thorough review of our work and hope our answers effectively address your concerns. We would be very grateful if you could consider raising the rating based on the responses.", "claims_and_evidence": "Some claims made in the submission are not sufficiently supported.\n\n1. **The proposed method addresses intrinsic biases and action coverage issues of LLM agents.** The LSPO method may be highly specialized for Werewolf, with latent strategy clusters potentially being too tailored to this environment. This could hinder its ability to generalize to new settings where deception and reasoning operate under different constraints. The generalization ability to other settings is unknown.", "methods_and_evaluation_criteria": "**Method**: \n1. Using a deep CFR solver to address the expanding dynamic action space remains questionable. While LSPO effectively maps strategies to a discrete latent space, the challenge lies in applying CFR to an ever-evolving action space. This aspect requires further investigation to ensure the solver can adapt to the complexities introduced by continuous expansion.\n\n2. While CFR has been effective in structured, finite-action-space games (like Poker), its application to an abstracted latent space in a language-based setting is less established. It is unclear whether CFR can truly capture the strategic elements of natural language deception. Moreover, the paper assumes that each role has a fixed set of latent strategies that remain the same across all discussion rounds. This could limit the flexibility of the agent in adapting its behavior dynamically as the game progresses.\n\n**Evaluation**: \nThe evaluation focuses on the following metrics, which are relevant to the Werewolf game, make sense for developing strategic language agents.\n- win rate: This directly measures the agent’s performance in the game.\n- prediciton accuracy: This measures the agent’s ability to predict other players’ roles and intentions, which is crucial for strategic decision-making.\nHowever, this paper provides limited comparasion with other methods. The paper compares the LSPO framework to existing methods like prompt-based approaches and shows improved performance, but does not evaluate against fine-tuned language agents without strategic optimization. This makes it difficult to isolate the benefits of LSPO’s iterative process versus simple fine-tuning.", "theoretical_claims": "No theoretical claims are made. The paper focuses on empirical validation.", "experimental_designs_or_analyses": "1. Limited Scalability of Latent Space Construction. The paper clusters free-form language actions into a latent space, but it does not sufficiently address how this clustering generalizes to larger-scale interactions beyond the specific Werewolf setup. The scalability of this method to games with more complex or dynamic interactions remains unclear.\n\n2. Dependence on Predefined Clustering. The process of clustering discussion actions into a latent space relies on embedding and k-means clustering. However, this rigid structuring might introduce artifacts and limit the expressiveness of the language, reducing the ability of agents to generate novel, contextually adaptive strategies. The ablation regarding $k$ should be included.\n\n3. Limited Baseline Comparisons. The paper does not evaluate against fine-tuned language agents without strategic optimization. This makes it difficult to isolate the benefits of LSPO’s iterative process versus simple fine-tuning.", "relation_to_broader_scientific_literature": "Large Language Models (LLMs). This research incorporates LLMs for natural language generation and leverages their reasoning capabilities for the game.", "essential_references_not_discussed": "This paper has discussed enough essential related works.", "other_strengths_and_weaknesses": "Strengths:\n- The LSPO framework is a novel contribution, particularly in its bootstrapping approach.\n\nWeakness: \n- There is a lack of game dialogue analysis.\n- The construction of latent space and the DPO approach require an extremely large number of game rounds, making it time-consuming and challenging for larger models.", "other_comments_or_suggestions": "An ablation study could explore the effectiveness of other open-source large language models (LLMs) for this task.", "questions_for_authors": "- Is there any example of a preference data pair that demonstrates how a lower regret value in speech selection is more suitable and potentially leads to winning the game?\n- Text-Embedding Model in Section 3.1: In Section 3.1, when using the text-embedding model to encode discussions, how to distinguish between discussions aimed at deception versus those intended to build trust? For instance, a statement like \"I agree with player X and want to support him\" could signify either deception or trust, yet it may be encoded identically. How is this ambiguity addressed?\n- Sophisticated Patterns in Section 4.1: The authors mention the emergence of more sophisticated patterns. Could authors provide text examples that illustrate these patterns?\n- In Table 1, the prediction accuracy appears to increase with each iteration. Is there a theoretical guarantee that this trend will continue indefinitely, even in the absence of distributional shifts? Alternatively, are there specific conditions under which this trend might not hold, potentially causing the accuracy to plateau or decrease? Why were only three iterations considered, especially if the prediction accuracy did not plateau or merge? Additional iterations should be provided.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jplgpKbwhF", "reviewer": "Reviewer_5jZJ", "summary": "This paper proposes Latent Space Policy Optimization (LSPO) to solve the werewolf game. The approach is to map text to latent space so that CFR and RL can learn a strategic policy, and then translate the policy back into natural language for DPO alignment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work and constructive suggestions. We hope the following response can address your concerns.\n\n> The paper would benefit from a more detailed mathematical formulation of both the problem and its solution. Also, adding some algorithm boxes might be better for understanding. But these are not severe weaknesses.\n\nWe agree with the reviewer and will add more detailed formulations in our revised manuscript. More specifically, some of the core definitions to be added are as follows.\n1. Abstracted game: we define the abstracted game as an extensive-form game with tuple $<N, H, P, f_c, (I_i)_{i\\in N}, u>$ where $N$ is the set of players, $H$ is the set of history, $P$ is the player function, $f_c$ is the probability measure for the chance node, $I_i$ is the information partition for player $i$, and $u$ is the utility function.\n2. Strategy: a (mixed) strategy $\\sigma_i$ for player i is the probability measure over actions for all $I\\in I_i$.\n3. Best response (BR): a BR to $\\sigma_{-i}$ is $BR(\\sigma_{-i})=\\arg\\max_{\\sigma_i}u(\\sigma_i, \\sigma_{-i})$.\n4. Nash equilibrium (NE): NE is a strategy profile where everyone plays a best response to others' strategies.\n5. Counterfactual value: $v(I)$ is the expected payoff of player $i$ when reaching $I$, weighted by the probability that $i$ would reached $I$ if tried to do so. Formally, $v^{\\sigma}(I)=\\sum_{z\\in Z_I}\\pi^{\\sigma}_{-i}(z[I])\\pi^{\\sigma}(z[I]\\rightarrow z)u_i(z)$. Definition of $v^{\\sigma}(I, a)$ is the same except it assumes action $a$ is always played at infoset $I$.\n6. Counterfactual regret: $R_T(I, a)=\\sum_{t=1}^T(v^{\\sigma^t}(I, a) - v^{\\sigma^t}(I))$.\n\nWe will also add algorithm boxes for CFR and our proposed method LSPO in our revised manuscript according to the reviewer's suggestion. \n\n> Can the author explain why prompting the LLM to generate N strategically distinct discussion candidates and randomly choose one to execute in the game that encourages diversity?\n\nThis is because the LLM tends to exhibit intrinsic bias in the action generation. Therefore, generating a single action will result in limited diversity and underexploration of the game. \n\nThe issue of intrinsic bias is discussed in detail by [1] with a simple Rock-Paper-Scissors (RPS) example. They use GPT-4 to play 100 independent RPS games and find that GPT-4 chooses Rock in 67 games, Paper in 27 games, and Scissors in only 6 games. This result shows that the LLM has a clear bias towards choosing Rock.\n\nTo solve this problem, a single method is to let the LLM generate multiple different actions. For example, in RPS, instead of a single action (which is probably Rock), we can prompt the LLM to generate 3 different actions (which include Rock, Paper, and Scissors), and then randomly choose an action. This would mitigate the intrinsic bias of LLM's action and encourage diversity and exploration in the game.\n\n[1] Xu, Zelai, et al. \"Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game.\" ICML, 2024.\n\n---\nWe would like to express our appreciation for your feedback and hope our answers have addressed your concerns.  We would be very grateful if you could consider raising the rating of our work based on our responses.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A.  There is no theoretical part in this paper.", "experimental_designs_or_analyses": "I checked section 4. no issue.", "relation_to_broader_scientific_literature": "From my understanding, the main novelty is the mapping from text to latent space so that one can perform CFR efficiently.", "essential_references_not_discussed": "N/A. I am not familiar with the baselines and recent advance in solving werewolf game.", "other_strengths_and_weaknesses": "Weaknesses:\nsome terms are not defined, e.g., abstracted game. Additionally, the paper would benefit from a more detailed mathematical formulation of both the problem and its solution (see suggestions). Also, adding some algorithm boxes might be better for understanding. But these are not severe weaknesses.", "other_comments_or_suggestions": "The paper would benefit from a more detailed mathematical formulation of both the problem and its solution. Currently, there is only one equation in the main text, which makes the overall procedure somewhat difficult to follow.", "questions_for_authors": "Can the author explain why prompting the LLM to generate N strategically distinct discussion candidates and randomly choose one to execute in the game that encourages diversity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Latent Space Policy Optimization (LSPO), a novel framework for training LLM-based agents in social deduction games like Werewolf, which require both strategic reasoning and natural language interaction. By mapping free-form text into a discrete latent space for policy learning and iteratively refining the agent through Direct Preference Optimization (DPO), LSPO significantly improves performance over existing agents in both decision-making and dialogue quality.

I find this line of work, integrating language-based communication and decision-making, to be very important. While the reviewers raised several points, I believe the repeated concern about evaluating only a single setup is relatively weak, as implementing such methods even for one game is far from trivial. I appreciate the reviewers’ additional comments, and as an area chair, I face  the challenge of weighing the pros and cons. All in all, the concerns do not seem particularly significant to me, and the paper’s novelty and potential impact are substantial.

Previous work: I recommend the authors to look on recent papers by Eilam Shapira et al., which address similar setups.
Decision: Accept (poster)