Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game
Authors: Rong Ye
Yongxin Zhang
Yikai Zhang
Haoyu Kuang
peng sun
zhongyu wei
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Language Game
Application
Primary area: applications
TL;DR: We introduced Multi-agent KTO, a method that trains LLM to play Werewolf through direct gameplay. Our approach outperforms GPT-4o and RL+LLM methods, achieving human-competitive performance.

Abstract: Achieving Artificial General Intelligence (AGI) requires AI agents that can not only make strategic decisions but also engage in flexible and meaningful communication. Inspired by Wittgenstein's language game theory, we propose that language agents can learn through in-context interaction rather than traditional multi-stage frameworks that separate decision-making from language expression. Using Werewolf, a social deduction game that tests language understanding, strategic interaction, and adaptability, as a test bed, we develop the Multi-agent Kahneman-Tversky's Optimization (MaKTO). MaKTO engages diverse models in extensive gameplay to generate unpaired desirable and unacceptable responses, then employs KTO to refine the model's decision-making process. In 9-player Werewolf games, MaKTO achieves a 61% average win rate across various models, outperforming GPT-4o and two-stage RL agents by relative improvements of 23.0% and 10.9%, respectively. Notably, MaKTO also demonstrates human-like performance, winning 60% against expert players and showing only 48.9% detectability in Turing-style blind tests. Code and data are available at project page https://reneeye.github.io/MaKTO.html.

Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game
Rong Ye Yongxin Zhang'; Yikai Zhang 1.2f, Haoyu Kuang 1.2
Peng Sun Zhongyu Wei
Fudan University Bytedance Seed Shanghai Innovation Institute {yerong, zhangyongxin yx, sathesongg@byteednce.. com com (hykuang23, ykzhang22) @m. fudan. edu. zywei fewei&fudan @fudan. edu. cn
Abstract
Achieving Artificial General Intelligence (AGI) requires AI agents that can not only make strategic decisions but also engage in flexible and meaningful com- munication. Inspired by Wittgenstein's 's language game theory, we propose that language agents can learn through in-context interaction rather than traditional multi-stage frameworks that separate aeciisionmmainng from language expres- sion. Using Werewolf, a social deduction game that tests language understanding, strategic interaction, and adaptability, as a test bed, we develop the Multi-agent Kahneman-TTversyy Optimization (MaKTTO). MaKTO engages diverse models in extensive gameplay to generate unpaired desirable and unacceptable responses, then employs KTO to refine the model's decision-ne meisionn-mainggg process. In 9-player Werewolf games, MaKTO achieves a 61% % average win rate across various mod- els, outperforming GPT-4o and two-stage RL agents by relative improvements of 23.0% and 10.9%, respectively. Notably, MaKTO also demonstrates human-like performance, winning 60% against expert players and showing only 48.9% de- tectability in Turing-style blind tests. Code and data are available at project page https:://eneeet gi github.... io MaKTO. html
Introduction
Building language agents capable of both decision-making and dialogue represents a crucial pathway toward Artificial General Intelligence (AGI) 3 This pursuit necessitates deep understanding of the intrinsic relationship between language and intelligence. Wittgenstein' 's Language Game Theory offers profound insight: meaning emerges from linguistic and non-linguistic interactions, regulated by social rules, forming language games (Fig Tb) () This contrasts with his earlier Tractatus view of language as logical pealit--mppping (Fig Dh.. This theoretical perspective points to the value of
grounding AI development in practical language use and authentic interactive contexts 7 9 TO [
Social deduction games (SDGs) serve excellent testbeds for validating these theoretical principles, with self- contained language-gam game environments that test multiple capabilities including linguistic [6] [7]. These games provide
skills, strategic social interaction, and adaptability [12] [3. 14 [5]
quantifiable metrics like completion and win rates, enabling direct comparisons between human and Al performance. The Werewolf game") as popular and typical social deduction game, exemplifies
these characteristics, making it a challenging testbed for Al agent [8] 19 20 21 22
Current approaches to building AI agents for Werewolf game often decouple language processing from decision-makinn [23) 24], echoing the limitations of the picture theory in Fig [ where complex social dynamics are oversimplified into rigid representations. For example, Wu et al. 21].
"qqual contribution.
Work done while intern at Bytedance.
"Coresponning authors.
3 Also known as Mafia, detailed game introduction is in Appendix A
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Language Theory
Framework
Form
Prompt
Facts
and or
Context
Structured Facts
LLM Output
Picture
Language
Intention
(a) Wittgenstein's 's Language and Picture Theory in Tractatus Logico-Phinn Philovophicux([12]
Previous: Decoupling decision from
language expression using multi-staae fiumework.
Non-Uingulstic
actions
Action
Language
games
Context
Output
Linguisstic expressions
Forules
Language
Intention
Wittgenstein's Language Game Theory
(d) Ours:
Joint modeling and optimization of language, strategic actions, etc. the multi-agent language
Philosophical Investigations (1933).
Figure 1: Language Theory and AT Architecture: Traditional vs. Language Game Models.(a) and (b): Mtlti framework that separates language and decisions. (c) and (d): our proposed framework inspired by Wittgenstein's 's language game theory, integrating language, actions, and intentions in a multi-agent game.
as illustrated in Fig. D. applied an classic RL model for decision-making or intention generation, then followed by an L.M for dialogue generation. However, they compressed the language input into structured facts, limiting generalization and tross-enviromment strategy transfer.
However, the paradigm of separating language processing from decision matiiiinnmmiinggggg essentially contra- dicts the central claim of Wittgenstein's 's theory of language games that there an intrinsic unity of language, intention, and action 225] Motivated by this, we introduce Multi-agent Kahneman & Tversky's Optimization (Multi-agent KTO, MaKTO), where the language models learn through direct interactions with different agents models. Specifically, MaKTO 1) builds on KTO for efficient training, 2) employs multi-agent gameplay with diverse model pool instead self-play, to enhance the generalization, and 3) uses slepwise preference selection preference data selection using heuristic, voting-based and verifier- methods, rather than simple win-loss outcomes. To help the model rapidly acquire Nomain-speciff knowledge of terminology and strategies in Werewolf, training process also incorporates behavior cloning using game-specific terms, strategy guides, and expert-annotated gameplay records with chain-of-thought [26] before MaKTO to optimize action decision-making in gameplay.
We perform various experiments, including tournament evaluation, Turing-style detectability test, behavioral analysis, generalization ability test, and ablation studies. The experiments show that MaKTO achieved an average win rate of 61% % in 9-player Seer- Wett--WWthrttt games against various models such as GPT-4o, Claude-3.5, and multi-staged RL agent. This outperformed both GPT-4o and the two-stage RI. agent, with relative improvements of 23 23.0% and 10.9%, respectively. In head-to-head matches against human expert players, MaKTO reaches comparable average win rate of 60%. Also, its conversational style is less distinguishable from humans, with only 48.9% accuracy in the Turing-style detectability test. Our contributions are:
We propose Multi-agent KTO (MaKTO),, a method that enhances LLMs' strategic reasoning in the game environment through multi-agent interactions, without requiring paired data. We create a large-scale dataset of expert Werewolf players' utterances and actions during gameplay, as well as the abundant COT behind their decisions, allowing for effective behavior cloning and fine-tuning LLMs.
We perform extensive experiments show that model achieves human-level performance and strong generalization capabilities across different game settings.
Our Approach
In this section, we describe in detail our training method (Fig. in detail, including expert data collection, behavior cloning, and multi-agent KTO.

---

Adte
Game-Play Example:
Fundamental game comprehension
P-
Night
Bay 195
Seallhich RC eat veal
Step1: Behavior Cloning
gaming
yiart an uneson
joad peoind serreel)
Modell
Authentic
Dusoset
Bay Riscussion
behawlor
seeelre 13) on
ca Narwwal
KTO
(e tokil
Step2: Multi-Agent KTO
Nipast portet deen
pegarrts 81 akido mocund Lodeer
Bay Rssuusion
Random Role-
PrESOdecann grd Gronnte yrur
PRamnss
model Assignment
Garee Play
Figure 2: The overall training process consists of (1) behavior cloning using instruction data (.2) and (2) multi-agent 2T3 In multi-agent gameplay, we randomly assign roles to agents to create diverse interactions that optimize the target model. stepwise selection process (right) identifies quality preferences using heuristic staged voting and verifier-based methods for KTO optimization.
2.1 Expert Data Collection
Despite various meticulously designed prompting methods [8 difference in language style and strategy play from real Werewolf players. While Automatic Speech Recognition (ASR) systems enable the collection of textual data from online Werewolf games [2 they cannot capture players' underlying reasoning. To address these issues, we collaborate with 17 experienced Werewolf players, including individuals with over thousand games of experience and competitive tournament participants, to annotate the dataset. We also ask these expert players to document their thought behind each decision during the game. Our dataset consists of:
27 28 29
LLMs still exhibit huge
Gameplay record: Contains the nighttime action records of special-role villagers and werewolves, daytime speeches and votes of players, and post-game reviews.
Thinking process annotation:
Documents players' reasoning for:
Action: the rationale behind night actions (e.g., seer's checks, werewolf's kills)
Speech: outline of the speech, the identity predictions for other players, and the call for the vote.
Voting: detailed reasons for voting and player identity predictions; Players are also required to distill the day's events into a consolidated record, create notes, and formulate a rudimentary strategy for the next game phase.
We collected 331 annotated Werewolf games from 17 expert players via our platform, with 51 additional games reserved for LLM evaluation in Sec. 3.3 Detailed statistics are in Appendix B
2.2 Behavior Cloning
Due to the scarcity of high-quality data in Werewolf domain, existing LLMs generally lack profound understanding and do not possess sufficient reasoning logic to support advanced gameplay. We address this by creating comprehensive, multi-level instruction dataset for supervised fine-tuning (SFT), as shown in Fig. SSSeep 1. Our instruction data is derived from three sources: (1) Fundamental game comprehension: the explanation of game terminology and jargon. As players continuously innovate within the game, specialized shorthand terms for efficient communication have emerged. For example, the term "Goldwater" refers players verified innocent through Seer's investigation. (2) Advanced gaming techniques: text collected from experienced players' online strategies, providing guidance for common game scenarios. For instance. includes expert tips on how werewolves can effectivelyy impersonate the Seer role and mislead the villagers. (3) Authentic gaming behavior: Derived from expert-annotated gameplay data in Section 2.1 Benefiting from the annotated thinking process, we structure them into a thhikk-before-rsspon the logic behind each stage of the game. For action, we first output the reason and then the target object. For speech, we output expected labels for others and voting intentions as the outline before generation. For voting, we output summary of the day's events, followed by the chain-of-thouge and the voting target. See Appendix C the examples of the data format. Additionally, we designed a role prediction auxiliary task, which involves predicting each player's role at the end of each day based on known speech, voting, and climination information.
format, enabling the model to truly comprehend
3

---

2.3 Multi-agent KTO
Although the SFT training enhances the model's comprehensive understanding of the Werewolf game, it faces two major challenges: the characteristics of the game and limitations in human-annotater data. In Werewolf, individual actions (like Seer's claims or Witch's poisoning) subtly influence outcomes, yet team victory doesn't 't ensure optimal individual play. Additionally, even expert data includes both good and poor decisions from winners and losers, making challenging to evaluate individual choices based solely on game outcomes. To address these challenges, we develop Multi- agent KTO (MaKTOO to mitigate such suboptimality. MaKTO features three key aspects: /) employs the Kahneman-TTvrrs adopts multi-agent gameplay to get diverse training data. 3) Instead of optimizing the entire trajectory based on win/loss outcomes, it optimizes step-wise policies.
Optimization (KTO) algorithm BQ] for decision refinement. 2) It
Kahneman-Tvvrsse Optimization We argue that KTO is particularly suitable for such multi-agent language game for two reasons. 1) Multi-agent environments are more complex than single/two-agent scenarios, where single-agent interactions with the environment often yield clear feedback, while interactions between agents can have countless possibilities. Moreover, multi-agent dialogues have huge action spaces, leading to sparse trajectory sampling. This makes online reinforcement learning algorithms slow to converge and hard to train. KTO, similar to offline RI,, offers a viable solution. 2) Unlike preference optimization algorithms such as DPO [1 and its variants B2 nontrivial to get crrooptt-cosen-rreee
33
34
35
36), it is
paired data. However, you can determine whether an output is acceptable or not through game rules and feedback from other agents. KTO, not requiring paired data for training, makes it ideal for such "Iry once" scenario in multi-agent preference optimization. The loss function of KTO is in Appendix D
Multi-agent Gameplay We find that in multi-agent settings, merely SFT or self-play can lead to rigid strategies and poor generalization (analysis in Sec. (.6). For example, we find that although the model performs well in scenarios with two players claiming to be the seer, its performance significantly deteriorates when only one or more than three players claim this role. We argue that in multi-agent environments, the diversity of peers and opponents is crucial. Therefore, instead of relying on self-play, we employ multi-agent play using a diverse model pool - including various SFT models (from Llama3. 1, Qwen2. 5), off-the- LLMs (GPT-4o, and Claude (..uee3355) 5), and the agent that uses RL for decision-making and LLM for speech generation. We then randomly select models from the model pool and assign them to different roles in the game. The multi-agent interactions allow for the exploration of a broader strategy space and avoid overfitting to specific policies.
Stepwise Preference Data Selection Defining all actions in an agent's 's trajectory as desirable or not based solely on the faction's win/loss result is too simplistic. Fortunately, Werewolf's alternating day-night gameplay allows for a more nuanced selection of the desirable and unacceptable step-wise process policies. Specifically, we employ three methods (detailed criteria see Appendix [D)
Heuristic-barne selection identifies actions based on role-specific strategies and game rules, focusing primarily on nighttime actions and voting phases. For example, werewolves targeting special roles is desirable while not attacking is unacceptable, and for witches, successfully poisoning werewolves is desirable. For voting, unified voting of the villagers against werewolves is preferred, while infighting or vote-splitting, thus weakening the villager team's position, is discouraged.
Staged voting-based selection uses voting outcomes to assess speech quality. Intuitively, players who voted out likely gave suboptimal speeches, either failing to defend themselves contradicting others' observations. Special role players (seer, witch) face stricter evaluation, with receiving majority villager votes considered unacceptable.
Verifier-based selection employs strong external LLMs, like GPT-4o to verify speech consistency with game facts and events. This can reduce the hallucination by performing fact consistency checks, as well as logical coherence and seff-contradictory.
3 Experiments
In this section, we first evaluate our approach in 9-player Seer- Witch-Guard Werewolf games through tournaments, human-Al competitions, and Turing-style detectability tests. We then analyze MaKTO's superior performance through behavioral studies, test its generalization capability with a new Hunter role in a new game setting, and validate key components through ablation studies. The implementation details are in Appendix E

---

Table 1: Average win rates of the models in the Seer- Barch-Guard setting. win rate above 0.5 and in bold indicates that the model in the row significantly outperforms the model in the column (Chi-square test, p- p-value<0.05).
GPT40 _ini
GPT4o
Claude
Mix
SFT-14b
SFT-72b
MaKTO--72b
Avg.
0.50
0.44
0.23
0.13
0.23
0.24
0.12
0.270
GPT4o
0.56
0.50
0.66
0.56
0.44
0.40
0.35
0.496
Claude
0.77
0.34
0.50
0.46
0.48
0.44
0.38
0.481
Mix (2T
0.87
0.44
0.54
0.50
0.58
0.45
0.47
0.550
SFT-14b
0.77
0.56
0.52
0.42
0.50
0.57
0.49
0.547
SFT-72b
0.76
0.60
0,56
0.55
0.43
0.50
0,42
0.546
MaKTO-72b
0.88
0.65
0.62
0.53
0.51
0.58
0,50
0.610
3.1
Tournament Win Rate Evaluation
3.1.1
Inter-agent Tournament
Experimental Setup We evaluate our approach against several strong baselines: API-based LLM agents (GPT-4o, (or-4o-mini. and Cl 1.-.e-3355-Sonee"" using thain-of-thouge prompting (26).. the Mix agent combining LLM with RL policy for decision making following [21], and SFT models (based on Qwen2.
-2b-/issriittt trained on our espert-annotated dataset.
the Mix agent. Directly optimizing LLMs with online RI in complex multi-party dialogue games computationally prohibitive, particularly as individual desirable actions do not guarantee victory. Therefore, we elaborate on the Mix agent here, as it provides a strong online RL benchmark. This agent adopts Cicero-like approach (37 [21] that decouples decision-making from expression. Specifically, it employs policy network trained with an AlphaStar-like multi-agent reinforcement learning (MARL) approach [38]. The policy network is trained through extensive self-play, guided by reward function that combines both the final game outcome (win or loss) and the stage outcome (survived or eliminated). This policy model operates on structured inputs converted from game actions and dialogue, while the LLM (e.g. GPT-4o) serves as a separate "expressor" to articulate the policy's s decisions. This agent therefore serves as a strong point of comparison for our method.
Head-to-head Competition In head-to-head compe- titions, where model controls the entire villager team (6 agents) and another controls the werewolf team (3 agents), MaKTO-72b achieved 61% % aver- age win rate across 100 games (50 games per faction), significantly outperforming all baselines (Table Notably, while the Mix agent showed strong perfor- mance as villagers, struggled as werewolves due overly aggressive strategies and policy contradictions (Fig. 3). When comparing SFT models of different sizes (14B and 72B), we observed similar win rates but significantly fewer factual hallucinations in the 72B model's generated speeches.
Win Rate of the Villagers
0.46
0.16
0.00
6974
0.56
0.44
0.32
0.34
0.32
0.86
The Vitagers
0.70
0.32
0.34
0.40
0.58
K70-726
0.84
0.62
0.42
0.48
0.32
0.78
0.84
0.74
0.74
0.64
0.72
0.68
GF746-191
Sh74e
2-144
575
175-22
Ms
Random Competition In random competitions with diverse role assignments across 260 games, MaKTO- 72b achieved the highest TrueSkill rating 39 (Fig. 4.
Figure 3: Villager win rate matrix of the head-to-head competition: villager (y-axis) vs. werewolf (x-axis). Lower left: higher val- ues for better villagers; Upper right: lower values for better werewolves.
This format better reflects the model's adaptability across different roles and team compositions. Par-
ticularly, MaKTO-72b significantly outperformed
GPT-4o when playing as the Seer, suggesting more persuasive statements and better trust-building capabilities. Compared to SFT-72b, MaKTO-77b also exhibited higher winning rates when playing as Guard and Witch, demonstrating better strategic skill usage.
'Model Versions: GPT4 gpt-4o-mini- mmiii-2024-07--- GPT40: gpt-4o- 2ot-44o2002-08- 2024-08-66 Claude-3.5-Sonnet: claude-3-5 335-sonnet-22224102..
5

---

Villager
5FT-226
TrueSa0l Rank Ratling
42%
Werewoff
24
371772
SPr
Witch
(a) TrueSkill Score of models
(b) Role Winning rates
Figure Results 260 random competitions 9-player Beer-Witch-Guaat game setting.
3.1.2 Human-AI Tournament
Experimental Setup To evaluate our model's performance in real-world games, we conduct human- Al tests with 14 experienced human players (1000+ games each) through head-to-head and random competition. Unlike previous studies 20 21 that only introduced single Al or human players, we involve multiple human players and Al models, creating more challenging and realistic environments.
Head-to-head Competition We evaluate MaKTO in
Win rate of MaKTO-72b in head- to-head competition with humans in 9-player Seer-Witch- Witch-Guard setting.
Table 2:
20 head-to-head Seer-Witch-Guard competition (10 games for each side), where MaKTO-72b plays all
villagers or werewolves independently, and the op- posing side are played by humans. MaKTO-72b achieved 60% win rate (5/10 as villagers, 7/10 as werewolves; Table 2). indicating that on par with high-level human players.
Humans
Villager
Wenewalf
Villager
0.5
MaKTO-72h
Werewolf'
0.7
Random Competition We also evaluate in random
competition for 30 seer-witch- 9-player games, where each player is randomly selected from 2-7 human players and the model pool (including GPT-4o, Mix agent, SFT-72b, and MaKTO-72b). As in Fig. 5] MaKTO-722 achieved a win rate 61.8%-18.39 win rate, ranking fourth among all players and surpassing the average human win rate of 54%. This shows that MaKTO-72b has strong adaptability against both human and Al opponents.
753%
74th
Win Rate (%)
Delecta Delectability Deccuracy.
60
61.8%
Average Human
530%
50.0%
to.
45
40 0%.9%
36.4%
30
H1
H2
H3
MakTQ
H4
H5
H6
H7
H8
H9
GPT
40H10
H11
H14
SFT
-72b
-22D
MudDo 575720
Figure 5: Win rate of players in random competition. H1-14 stand for the win rate of human players.
Figure 6: Human de- tection accuracy.
3.2 Turing-style Detectability Test
We conduct rigorous Turing-style blind detectability tests in both competitions. We require each human player to explicitly judge whether every other participant is human AI, without any prior knowledge of Al presence. This mandatory assessment provides a rigorous human similarity assessment. МaKТO achieves detection accuracy of only 48.9% (Fig. (6) lower than random chance, indicating that our model successfully passes this specialized Turing test by convincingly emulating human-like gameplay characteristics and social behaviors. On the contrary, GPT-4o has much higher detection rate (76.6%) due to the significant differences in speaking style and voting behavior from human players.

---

3.3 Behavioral Analysis
3.3.1 Comparison with Baseline Model
In order understand why MaKTO has higher win rate in tournaments, we analyze the behaviors and decisions generated during the tournament. For fair comparison, we selected
Villager
As Werewolf
100
Metrics (%)
SFT-72b
75
MakTO-723 72b
4PT-4o as the opponent. We run 50 games be- tween both sides and compute the proportion
50
25
of behavior occurrences as metrics, Results are shown in Fig. 3 with detailed metrics provided
Vidte
Wer oo Check
Save
Night
Conrect
Foisen
Protect
God
Abstenting
Hach
Misporret
Appendix
When the trained models played as villagers against GPT-4o werewolves, evaluate vot- ing accuracy (Vote Acc.). For special roles, we examine: Seer's werewolf identification ac- curacy (Werewolf Check), Witch's 's first-night rescue rate (Save @ Night I), werewolf poisoning accuracy (Correct Poison), and Guard's special role protection rate (Protect God). Fig. 3) shows that MaKTO outperforms the baseline SFT model across all the metrics, which yields higher villager win rate. These improvements can be attributed to the stepwise decision rewards and penalties in the MaKTO training.
Table 3: Behavioral Analysis: model performance as villagers (left) and werewolves (right) against GPT-4o. For werewolf, lower opponent scores in- dicate better performance of the model.
When our models played as werewolves against GPT-4o villagers, evaluated the opponent's behavior instead of our model's direct performance. The rationale behind this approach is that better werewolf deception leads to more opponents' confusion and mistakes. Specifically, we measured the following metries of the CPT-4o opponents: 1) Voting abstention rate (Abstention): Higher rates indicate difficulty identifying werewolves. 2) Seer's werewolf identification rate (Seer Check, see Table 15 for specific values): Lower rates suggest successful misdirection. 3) Witch's s villager poisoning rate (Mispoison)): Higher rates indicate better werewolf concealment. Guard's is werewolf protection rate (Misprotect): Higher rates suggest effective deception. All the fine-graine metrics of МaKTO are better than the SFT model, which shows that MaKTO werewolf has a superior camouflage capability in the game.
3.3.2 Comparison with Human
A feature of the Werewolf game lies in its prevalence of deception, partic- ularly among the werewolf players. Werewolves never openly admit their identity in daytime discussions; in- stead, they make up various identities and stories to protect themselves. It becomes important for villagers to cor- rectly predict who is the werewolf, so that they can cast the right vote. So in this experiment, we compare the correct judgments compare them with humans.
Table 4: Offline results of human annotators and LLMs. The bold number represents the best results of the models, and the underlined number represents the second best.
Vote Acc-(t)
Abstent Rate(()
Align. Acc.(t)
Wolf-pred. F1(1)
GPT4o mini
67.2%
0.4%
68.1%
0.519
GPT4o Claude
69.4%
2.3%
68.0%
0.587
68.4%
1.0%
75.2%
0.651
Qwen2.5-14h
61.0%
4.1%
61.1%
0.528
Qwen2.5-72h
66.5%
0.4%
63.9%
0.552
villagers and
SFT-14b
70.8%
4.1%
77.9%
0.712
SFT-72b
71.1%
5.8%
79.3%
0.734
We separate 51 matches from the an- notated data as the test set, excluding them from the training dataset. The evaluation covers 484 voting events and 5130 identity predictions. The results are presented in Table In terms of voting, we evaluate the voting accuracy. that is, the accuracy of gods and villagers voting for werewolves; and the abstention rate. MaKTO-72b achieves the highest voting accuracy. terms of identity prediction, we evaluate the accuracy of side alignment (Align. Acc.), that is, correctly predicting gods and villagers as the good identity and werewolves as the bad identity; and the F1 F1-score in predicting werewolves (Wolf- pref. Fl). The trained models show significant improvement over the base model like Qwen2.5- 5- 5-14
MaKTO-72b
73.8%
1.5%
78.4%
0.734
HUMAN
76.7%
4.8%
76.1%
0.742
7

---

and -72b-instruce which demonstrate the effectiveness of the expert data we collected. Also, we notice that, the SFT models and MaKTO even achieve higher side alignment accuracies than human.
3.4 Generalizing to Other Game Setting
Another advantage of our model lies in its cross-game generalization capability, which we test by varying both the game's role and its scale.
Adaptability to New Role in the Game First, we assess MaKTO's s ability to adapt to a rule change by introducing new role the Hunter-in in place of the Guard. The Hunter can choose to eliminate another player upon their own elimination. We conduct tournament experiments in the Seer- Witch-Hunter setup. In this new game setup, where Hunter replaces Guard, the policy model of Mix agent (trained on Seer- With-Guard setup) no longer remains effective, whereas MaKTO continues to perform exceptionally well (Table 55. Despite being trained only Seer-Witch- Witch-Guard gameplay data, TаKTO still outperforms SFT models, demonstrating strong adaptability and generalization capabilities.
Table 5: Average win rates of the models in the 9-player Seer- Witch-Hunter setting. A win rate above 0.5 and in bold indicates that the model in the row significantly outperforms the model in the column (Chi-square test, vaaaaae.0000
GPT4o mini
GPT4o
Claude
SFT-14b
SFT-72b
MaKTO-14b
MaKTO-72b
Avg.
GPT4o mini
0.50
0.48
0.15
0.33
0.37
0.26
0.29
0.340
GPT4o
0.52
0.50
0.72
0.62
0.50
0.52
0.54
0.560
Claude
0.85
0.28
0.50
0.62
0.53
0.46
0.37
0.516
SFT-14b
0.67
0.38
0.38
0.50
0.53
0,48
0.51
0.493
SFT-72b
0.63
0.50
0.47
0.47
0.50
0.51
0.48
0.507
MaKTO-14b
0.74
0.48
0.54
0.52
0.49
0.50
0.46
0.533
MaKTO-72b
0.71
0.46
0.63
0.58
0.52
0.54
0.50
0.563
Generalization to Larger Game Scales. To fur- ther probe its generalization, we evaluated MaKTO even more complex scenarios with larger number of players: 10-player game (with either Guard a Hunter) and a 12-play 12-player game. These settings intro- duce different strategic dynamics and were entirely absent from the training data. The results in Table 6 show that MaKTO-72b consistently achieves the highest average win rate across all three larger-scale settings. Taken together, these results demonstrate that MaKTO learns robust, transferable strategic reasoning, rather than merely overfitting to its training configuration.
Table 6: Average win rates models in un- seen 10- and 12-player settings. MaKTO con-
sistently tongton baselines. See Appendix G for detailed head-to-head results.
Models
10-Player w/ Guard
10-Player w/ Hunter
12-Player
GPT-4o
0.335
0,320
0.423
Claude
0.573
0.488
SFT-72b
0.575
0.523
0.485
MaKTO-72b
0.625
0.585
0.605
3.5 Comparison between MaKTO and Long-COT Reasoning Model
An interesting question is how our MaKTO compares to long-COT reasoning models 40 in the Werewolf game. We conducted win rate comparisons between reasoning models and MaKTO models of equivalent size in the Werewolf testing environment. Results (Tab. 7 show that MaKTO-14B achieves 74% overall win rate (70% as werewolves, 78% as villagers) against DeepSeek-RR -ineek-RR--in RD-Distill -wwen11wwwwe--m11 1 14B [41] and comparable-sized MaKTO-72B achieves an 84% win rate against R RI-distilled-70BB. RL LLMS trained math and coding tasks fail generalize to the social deduction games, like Werewolf. While R1 models make conservative statements (e.g., a Seer might say "I found suspicious player but cannot reveal who"), making vulnerable to attacks and losing trust from villagers, MaKTO models develop more sophisticated social strategies through diverse interactions. For example, when werewolf's 's false claim draws suspicion, the werewolf team might strategically eliminate this werewolf to cast doubt on the real Seer in subsequent rounds. Note that while we applied "think-before-rees respond,, " the tokens in the thinking are fewer than those in the reasoning models. We believe that long- COT and our MaKTO are not mutually exclusive, but this paper focuses on the training method, with their integration left for future work.
3.6 Ablation Studies
In the ablation studies, we reveal two crucial design in MaKTO - multi-agent gameplay and step-wise preference data selection. Note that all experiments use the 14B model.
8

---

Table 7: Win rate for MaKTO and DeepSeek-RI- Ri-ssillll models in 9-player Seer- Witch-Guar Guard game.
Wining rate of 14B models
GPT-4o
R1-Distill-Qwen-
MaKTO-14b
Avg. Win Rate
GPT-4o
0.50
0.89
0.37
0.630
RI-Distill-Qeen-14b
0.11
0,50
0.26
0.185
MaKTO-14b
0.63
0.74
0.50
0.685
Wining rate of 70B/72B models
GPT-4o
R1-Distil-
MaKTO-72b
Avg. Win Rate
GPT-40
0.50
0.79
0.35
0,570
R1-DistillLLamaa-0bb
0.21
0.50
0.16
0.185
MaKTO-72h
0.65
0.84
0.50
0.745
Q1: Multi-agent Play or Self-play? MaKTO requires playing with various agents to select preference data for training. We compare MaKTO with Self-play KTO, where an SFT-trained model engages in numerous battles only against itself to collect preference data for training. To ensure fairness, both methods used 20k training samples. From Table 8 multi-agent play sig- nificantly outperformed self-play KTO, achieving a 4% higher average win rate. While self-play KTO showed competi- tive performance against SFT models, significantly underperformed against diverse opponents like
Table &: Ablation study for MaKTO and other KTO train- ing methods. Self-play: preference data from SFT- 14B vs. S4T-14B games. w/o gp: preference data from annotations only without actual gameplay. All models are in 14B size.
Ma- KTO
Self-
w/o
Avg.
GPT-4o
Mix
SFT
play
BP
Win
GPT--0 Mix
0.50
0.56
0.44
0.37
0.56
0.64
0.514
0.44
0.50
0.58
0.55
0.71
0.48
0.553
SFT
0.56
0.42
0.50
0.48
0.38
0.66
0.499
KTO methods
MaKTO
0.63
0.45
0.52
0.50
0.50
0.57
0.534
Self-play
0.44
0.29
0.62
0.50
0.50
0.56
0.483
w/o gp
0.36
0.31
0.34
0.43
0.44
0.50
0.374
GPT-4o. This demonstrates that exposure to diverse opponents is crucial for developing robust strate- gies, similar to findings in population-base based training [42] 20). Additionally, we find that preference data can be also obtained from annotated data without engaging in gameplay (w/o gp). But this results in a substantial 13% decrease in average win rate, highlighting the importance of interactive gameplay in our approach.
Q2: Step-wise Selection or Selection based solely on results? MaKTO uses stepwise ac- tion selection, while an alternative can simply collect preference trajectory data from win/loss outcomes. Which is better? Dable the comparison of win rates stepwise preference selection proves superior to trajectory-basse lection. This means a final reward does not nec- essarily indicate that all actions in that trajectory desirable, and vice versa. Selecting data based on trajectory outcomes leads the model to learn suboptimal actions and wrongly penalize desirable ones, resulting in a lower winning rate. This also confirms hypothesis that game out- comes alone cannot accurately reflect the quality of individual decisions in complex SDGs.
Table 9: Ablation study for MaKTO and other KTO training methods. Step (Ours): Selecting desirable/unacceptable actions according to prede- fined criteria. Traj.: Selecting desirable actions from winning trajectories and unacceptable actions from losing trajectories. All models are in 14B.
se-
Ma
Avg.
GPT-4o
SFT
Step
Traj.
Win
GPT-40 Mix
0.50
0.56
0.44
0.37
0.45
0.456
0.44
0.50
0.58
0.55
0.63
0.550
SFT
0.56
0.42
|
0.50
0.48
0.54
0.496
MaKTO with different action selections
Step(Ours) Traj.
0.63
0.45
0.52
0.50
0.58
0.545
0.55
0.37
0.47
0.44
0.50
0.458
Q3: Contribution of Each Training Stage. To quantify the contribution of each component of our pipeline, we compare the base instruction- tuned model (Qwen2.5- 1nnn2..5-itttuut-22555 the model after behavior cloning (SFT-72b), and the final model after KTO refinement TaKT0-)2b).. Ta- ble 10] shows clear stepwise improvement. Be- havior cloning on our expert dataset provides a massive performance boost of 30.6% over the
Table 10: Component analysis: head-to-head win rate of the base model SFT model, and the final MaKTO model against each other. Models are in 72B.
the
Base
SFT
MaKTO
Avg.
Base SET MaKTO
0.50
0.20
0.10
0.267
0.80
0.50
0.42
0.573 (+30.6 %)
0.90
0.58
0.50
0.660 (+8.7%) %)

---

base model. Subsequently, MaKTO refinement adds another 8.7% improvement, demonstrating the distinct value of both high-quality expert data and interaction-based preference optimization.
Related Work
SDG as Testbed Social Deduction Games (SDGs) have emerged as useful benchmarks for social reasoning. They uniquely combine f'undamenaal linguistic skills, higher-level social reasoning
abilities, and adaptability. Recent research has extensively explored various SDGs [28 13 14 13
19 22 becoming a popular testbed for evaluating LLMs reasoning through its complex dynamics of deception and cooperation.
16 451, with Werewolf [8]
20
LLM-based Game Agents While early game Al relied on reinforcement learning (RL) in envi-
LLMS enabled more sophisticated memory mechanisms [50], experience learning [8] (27) and persona prompting 229. Existing approaches for Werewolf Al typically adopt two-stage frameworks: either RL-then-LLMM 21 or LLM-then-RL 220 [1]. which either compress the language strategy space or are constrained by generated candidates. We propose integrated training for LLM through direct interactions. Very recently, Sarkar et al. [52 introduced MARL for AmongUs with a smaller RWKV-1.5B (53) model, Other research has explored fine-tuning LLMs for different strategic games like Diplomacy [54]. But these often in settings with more structured or limited communication, while our method efficiently leverages LLM while maintaining stable training.
ronments with no or only little dialogue 46
49
agent modeling in SDGs, through generative planning 28
Agent Learning Our approach is closely related to agent learning research. Current methods generally follow 1) imitation learning with expert trajectory data [55] 56 57 58] or 2) learning
through environmental interactions [36 59 60 61]. While some have also explored the gaming
environments [60 62 63, they primarily focus on simpler single/two-agent scenarios. We tackle the
complexity of multi-agent language game environments, requiring better adaptability and robustness.
5 Conclusion
In this paper. we present Multi-agent KTO (MaKTO), a novel approach for optimizing LLMs in complex social deduction games. It improves LLM's s social reasoning and strategic interactions through interaction-based feedback. MaKTO beats GPT-4o with 23.0% higher win rates and wins 60% against expert humans, while maintaining human-like conversations. We also contribute large-scale expert Werewolf dataset with player actions and their reasoning processes.
Limitations
We identify several limitations. First, for fair comparison, we focus on the rationality of models' decisions without customizing the different role-playing personas in the system prompt of LLMs. Second, our current implementation relies on turn-based conversations rather than free-form interac- tions 50 64
65 66). The challenge of modeling unrestricted multi-agent communications, where agents can interact more naturally and flexibly, remains an important area for future research. Third, similar to the general limitations of LLMs, our model occasionally exhibits inconsistent behavior and hallucinations across long conversations, suggesting room for improvement in long-text modeling capabilities, particularly in maintaining coherence during extended social interactions. Finally, while Multi-agent KTO provides an easy yet effective training paradigm, we believe that online multi-agent RI. could potentially better, presenting another promising direction for future research.
Acknowledgments
We would like to thank Zhaohao Xiao, Weicheng Fu, Qiang Lu, Danyu Lv, Hangfei Guo, Zheng Li, and Huiying Lin for their valuable support in the development of the annotation system. We are also grateful Xiaoxi Mao, Jinfeng Wang, and Zechuan Hu for their insightful discussions.
References
[1] Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice. The knowledge engineering review, (12):115-152: 1995.
[2] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo. Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions Machine Learning Research, 2022.
10

---

[3] Paul M. Salmon, Chris Baber. Catherine Burns, Tony Carden, Nancy Cooke, Missy Cummings, Peter Hancock, Scott McLean, Gemma JM Read, and Neville A Stanton. Managing the risks of artificial general intelligence: A human factors and ergonomics perspective. Human Factors and Ergonomics in Manufacturing & Service Industries, 33(5):366-33 -7666 2023.
Theadore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architectures for language agents. Transactions on Machine Learning Research, 2023.
[5] Ludwig Wittgenstein. Philosophical Investigations. Basil Blackwell, Oxford, 1953, ISBN
[6] Roman Kopytko. Philosophy and pragmatics: A language-gama -ameag with ludwig wittgenstein. Journal of Pragmatics, 39(5):792-81) 812, 2007.
[7] Rom Harre. Wittgenstein and artificial intelligence. Philosophical Psychology, 1(1):105-115, 1988.
[8] Sida Wang. Percy Liang, and Christopher D Manning. Learning language games through interaction. In In Proc. of ACL, pages 2368 2378. Association for Computational Linguistics (ACL), 2016.
Tom Schaul. Boundless socratic learning with language games. In NeurlPS 2024 Language Gamification Workshop, 2024.
[10] Christine Cuskley, Rebecca Woods, and Molly Flaherty. The limitations of large language models for understanding human language and cognition. Open Mind, 8:1058-1082 2024.
[11] Ying Wen, Ziyu Wan, and Shao Zhang. Language games as the pathway artificial superhuman intelligence. arXiv preprint arXiv:2501. 18924, 2025.
[12] David Schlangen. Dialogue games for benchmarking language understanding: Motivation, taxonomy, strategy. arXiv preprint arXiv:2304.07007, 2023.
[13] Dekun Wu, Haochen Shi. Zhiyuan Sun, and Bang Liu. Deciphering digital detectives: Understanding Ilm behaviors and capabilities in multi-agent mystery games. In Findings of ACL, pages 2225-8291, 2024.
[14] Alessandro Trevisan, Harry Giddens, Sarah Dillon, and Alan F Blackwell. Mcasuring bullshit in the language games played by chatgpt. arXiv preprint arXiv:2411. 2024.
[15] Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating Ilms playing the game of avalon. In NeurlPS 2023 Foundation Models for Decision Making Workshop. 2023.
[16] Yizhou Chi, Lingjun Mao, and Zineng Tang. Amongagents: Evaluating large language models in the interactive text- based social deduction game. arXiv preprint arXiv:2407. 16521, 2024.
[17] Wei Wang, Dan Zhang, Tao Feng, Boyan Wang, and Jie Tang. Battleagenthench: A benchmark for evaluating cooperation and competition capahilities of language models in multi-agent systems. arXiv preprint 4roiv:2408. 15971,2022
[18] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.0em 04658, 2023.
[19] Suma Bailis, Jane Friedhofff, and Feiyang Chen. Werewolf arena: A case study in ulm evaluation via social deduction. arXiv preprint 24X1v:2407.13943, 2024.
[20] Zelai Xu, Chao Yu, Fei Fang. Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. In Proceedings of the 41st International Conference on Machine Learning, ICMI.'24. Jor. 2024.
[21] Shuang Wu, Liwen Zhu, Tao Yang, Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu, Enhance reasoning for large language models in the game werewolf. arXiv preprint 24Xiv:2402.02333 2024.
[22] Silin Du and Xiaowei Zhang. Helmsman of the masses? evaluate the opinion leadership of large language models in the werewolf game. arXiv preprint 200v:2404.01602 2024.
[23] He Hc, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
2018
[24] Tadahiro Taniguchi, Ryo Ueda, Tomoaki Nakamura, Masahiro Suzuki, and Akira Taniguchi. Gener- ative emergent communication: Large language model is collective world model. arXiv preprint arXiv:2501.00222 2024.

---

[25] Jinyu Cai, Jialong Li, Mingyue Zhang, Munan Li, Chen-Shu Wang. and Kenji Tei. Language evolution for evading social media regulation via Ilm-based multi-agent simulation. IEEE World Congress on Computational Intelligence (WCCI) IEEE Congress on Evolutionary Computation (IEEE CEC), pages
1-10, 6 2024, doi: 10009//EE6090122240.00000
[26] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le. Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824- 24837, 2022.
[27] Yihuai Lan, Zhiqiang Hu, Lei Wang. Yang Wang. Deheng Ye. Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. Lim-based agent society investigation: Collaboration and confrontation avalon gameplay. arXiv preprint XrXiv: 10. 14985, 2023.
[28] Shenzhi Wang. Chang Liu, Zilong Zheng, Siyuan Qi. Shuo Chen, Qisen Yang. Andrew Zhao, Chaofei Wang. Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation. arviv preprint arXiv:2310.01520, 2023.
[29] Takehiro Sato, Shintaro Ozaki, and Daisaku Yokoyama. An implementation of werewolf agent that does not truly trust 1lms. Proceedings of the 2nd International A/WolfDial Workshop, pages 58 67, 2024.
[30] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff. Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In In proc. of ICML, 2024.
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neurul Information Processing Systems, 36, 2024.
[32] Jiwoo Hong. Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pages 11 11170-11189) 2024.
[33] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive arXiv preprint 240, 2024.
[34] Meng. Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.1474 14734, 2024.
[35] Xin Lai. Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406. 18629, 2024.
[36] Yifan Song, Da Yin, Xiang Yue. Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration- based trajectory optimization of Ilm agents. In In Proc. of ACL, pages 7584-7600, 2024.
[37] Meta Fundamental AT Research Diplomacy Team (FAIR)t, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty. Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human- level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378 (6624):1067-107) 2022.
[38] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathicu, Andrew Dudzik, Junyoung Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi agent reinforcement learning. Nature, 575(7782)3350-33 3544, 2019.
[39] Ralf Herbrich, Tom Minka, and Thore Graepel. TrueskillTM; bayesian skill rating system. Advances in neural information processing systems, 19, 2006.
[40] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang. Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long of-ann- for reasoning large language models. arXiv preprint arXiv: 09567, 2025.
[41] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Peiyi Wang. Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforcement learning. Nature, 645(8081 (:00008113333--111 ):633-688, 2025.
[42] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals. Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv: / 1.09846, 2017.
[43] Sherzod Hakimov, Yerkezhan Abdullayeva, Kushal Koshti, Antonia Schmidt, Yan Weiser, Anne Beyer, and David Schlangen. Two giraffes in a dirt field: Using game play to investigate situation modelling in large multimodal models. arXiv preprint 4rXiv:2406.1400 14035, 2024.
12

---

[44] Byunghwa Yoo and Kyung-Joong Kim. Finding deceivers in social context with large language models and how find them: the case of the mafia game. Scientific Reports, 14(1):30946, 2024.
[45] Satvik Golechha and Adrià Garriga-Alonso. Among us: A sandbox for agentic deception. arXiv preprint arXiv:2504, 04072, 2025.
[46] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershclvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[47] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez. Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359,,,,,) 2017.
[48] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912 2666000 2019,
[49] Jack Serrino, Max Kleiman- Kleiman-Weiner, David C Parkes, and Josh Tenenbaum. Finding friend and foe in multi-agent games. Advances in Neural Information Processing Systems, 32, 2019.
[50] Joon Sung Park, Joseph O' B'Brien, Carrie Jun Cai, Meredith Ringel Morrs,, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra human behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1-22, 2023.
[51] Zelai Xu, Wanjun Gu, Chao Yu, Yi Wu, and Yu Wang. Learning strategic language agents in the werewolf game with iterative latent space policy optimization. In Proc. of ICML, 2025.
[52] Bidipta Sarkar, Warren Xia, C. Karen Liu, and Dorsa Sadigh. Training language models for social deduction with multi-agent reinforcement learning. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, 2025.
[53] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao. Xin Cheng. Michael Chung. Leon Derczynski. et al. Rwkv: Reinventing rans for the transformer era. In Findings of EMNLP, pages 14048-14077, 2023.
[54] Xu Kaixuan, Chai Jiajun, Li Sicheng, Fu Yugian, Zhu Yuanheng, and Zhao Dongbin. Diplim: Fine-tuning Ilm for strategic decision-making in diplomacy. In In Pruc. of ICML, 2025.
[55] Aohan Zeng, Mingdao I Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuming: Enabling generalized agent abilities for 1lms. In Findings of ACL, pages 3053-3077, 2024.
[56] Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang. Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. pages 27914-27999 27961, 2025.
[S7] Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. In Findings of ACL, pages 9354 03644993662222 2024.
[58] Andrew Zhao, Daniel Huang. Quentin Xu, Mauhieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, In Proc. of AAAL. pages 19632-19642. AAAI Press, 2024. doi: 10. 100%/aaii watpaaaii.iii7........ 23811..2933.. URL https://ojs aaai. org/ index, php/ /ttp//////arrrce//e/////////////
[59] Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! I1m agent learning via iterative step-level process refinement. In In Proc. of EMNLP, pages 3556-1572 2024.
[60] Pengyu Cheng, Tianhao Hu. Han Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du. Self-playing adversarial language game enhances Ilm reasoning. In Advances in Neural Information Processing Systems. 2024.
[61] Pranav Putta, Edmund Mills, Naman Garg. Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.077
2024.
[62] Wenqi Zhang, Ke Tang. Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. Agent-pro: Learning to evolve via policy-level reflection and optimization. In In Proc. of ACL, 2024.
13

---

[63] Ruiyi Wang, Haoféi Yu. Wenxin Zhang, Zhengyang Qi. Maarten Sap. Yonatan Bisk, Graham Neubig, and Hao Zhu, Sotopia-T: Interactive learning of socially intelligent language agents. In In Proc. ACL, pages 12912-12940, 2024.
[64] Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang. Xiawei Liu, Shiyue Yang. Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, et Agentsense: Benchmarking social intelligence of language agents through interactive scenarios. arXiv preprint arXiv:2410. 19346, 2024.
[65] Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael S Bernstcin, and John Mitchell. Social skill training with large language models. arXiv preprint 4xXv:2444.04204 2024.
[66] Jen-tse Huang, Jiaxu Zhou, Tailin Jin, Xuhui Zhou, Chen, Wenxuan Wang, Youliang Yuan, Maarten Sap, and Michael R Lyu. On the resilience of multi-agent systems with malicious agents. arXiv preprint arXiv:2408, 00989, 2024.
[67] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. technical report. arXiv preprint wxXi:2412.15115. 2024.
[68] Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang. Tinghui Zhu, et al. From persona to personalization: A survey on rolc- role-playing language agents. arXiv preprint Xrv:: X40: 18231, 2024.
[69] Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-IInn: A trainable agent for role-playing. In The Conference Empirical Methods in Natural Longuage Processing, 2023.
14

---

A Game Rules
Werewolf is one of the most popular social detection games, typically played with 7 to 15 players. The game is set in a village where some players are secretly assigned the role of werewolves, while the majority are villagers. In this paper, we focus primarily on the variant with players: werewolves and 6 villagees, including special roles of Seer, a Witch, and a Guard (Figure The gameplay consists of two alternating phases: night and day. During the night, the werewolves secretly choose a victim, while the Seer checks one player's identity, the Witch can use a one-time antidote or poison, and the Guard protects one player from being killed by the werewolves. In the day phase, all surviving players discuss and vote to eliminate suspected werewolf. The game continues until either all werewolves are eliminated (village wins) or all simple villagers or all special-role villagers are eliminated (werewolves win). Another variant of the 9-player game introduces the Hunter in place the Guard. The Hunter's skill allows them to shoot and eliminate one player when they are eliminated.
Night
Day
Winning
Seer
Witch
Guard
Lost
Kill Player
There dhealhs.
Player beilcht
Check Player 2: not wurewelff.
Plyyr 3 (wprowolft iecurta eathing
Lot eight, Franed Nayer
All The aotoondetssrre
VS.
Player a (Guar hoime........
antidote/pnison to save/ /Kll Player
Player Decheree
Hove 2tyy
Werewolf
Veye
All the shmple vaggers edrmerated agal
Simple villager x
Protect Player
Playe
Game Setup of 9 players
Game Process
Figure 7: The setup and gameplay of the 9-player Werewolf game with special roles of the Seer, the Witch, the Guard.
Here are the details of the specific game rules:
A.1 Game Objectives
In this game, players are usually divided into two camps: werewolves and villagers. Depending on their roles, players have distinctive objectives:
Villagers aim to identify the werewolves and eliminate them through voting. Within the villagers' camp. there are some special roles with distinctive abilities that can help the villagers secure victory. Werewolves' primary objective is to conceal their true identities, mislead others in discussions to avoid being voted out, and hunt villagers as covertly as possible.
A.2
Game Process
The game generally includes the following basic procedures.
Upon entering the game, player roles secretly assigned. Werewolves know each other's 's identities, while villagers only know their own role.
Role Assignment:
The game alternates between day and night phases. At night, werewolves secretly choose a villager to eliminate; some special roles can also activate their abilities at night. During the day, all players discuss and vote eliminate the player they believe be werewolf, with the player receiving the most votes being eliminated.
Day-Night Alternation:
Victory Conditions: The game ends when one faction achieves its victory conditions. Villagers win if all werewolves are eliminated. Werewolves win if they eliminate all ordinary villagers or all special roles.
A.3 Role Descriptions and Different Configurations
Standard configurations for 9-player and 7-player games incorporate six distinct roles: the Seer, the Witch, the Guard, the Hunter, the Werewolf, and the Villager. Different roles have the following abilities.
Seer: During the night phase, the Seer can secretly select a player to learn their true identity (whether they are a werewolf or not).
Alss known Mafia, https://en/ http:////....aaaeooaoooo/ee/eeaa (patty__aapppaaaaaaaa
15

---

The Witch has one healing potion and one poison potion, each usable only once. The Witch cannot use both potions in the same night. The healing potion can save player killed by werewolves at night. The poison potion can eliminate player suspected of being werewolf. Guard: The Guard can protect one player each night from werewolf attacks. The Guard can choose to protect himself or opt not to protect anyone, but cannot protect the same player on consecutive nights.
Witch:
Hunter: When the Hunter is killed by werewolves eliminated during the voting event, he can reveal his identity card and shoot a revenge bullet at any living player, causing that player to die as well. The Hunter can choose not to reveal his card, but once revealed, he must take someone with him (Note: the Hunter is poisoned by the Witch, he cannot reveal his card or take anyone with him).
Werewolf: Werewolves can choose to eliminate a player during the night phase.
Villager: Villagers have special abilities. They can only distinguish the werewolves through daytime speech and public information.
This paper encompasses four distinct configurations, including the 9-player werewolf game with three special roles, and the 7-player werewolf game with two special roles:
Seer-Witch Witch-Gurr:: Includes one Seer, one Witch, one Guard, three Werewolves, and three
Villagers.
Beer-Witch-Huntrr Includes one Seer, one Witch, one Hunter, three Werewolves, and three Villagers.
Seer-Guard: Includes one Seer, one Guard, two Werewolves, and three Villagers. Seer- Witch: Includes one Seer, one Witch, two Werewolves, and three Villagers.
B Dataset Statistics
17 expert players provide annotations using our self-built annotation platform. Each annotator was paid much more than the average local salary. We collect 331 Werewolf games for training, including 278 9-player games (Seer, Witch, Guard or Hunter) and 53 7-player games (Seer, Guard or Witch). We collect 331 matches of Werewolf games annotated by 17 advanced players based on our self-buill annotation platform. This dataset includes 278 9-player games (using the setups of Seer, Witch, Guard and Hunter) and 53 7-player games (using the setups of Seer, Guard and Witch). Each game features randomly involved participants and randomly assigned Werewolf roles to guarantee data diversity. The total duration of the game annotated by the players exceeds 1,000 hours, including 3,759 speech data entries (exceeding 540,000 tokens), 2,698 action events, and 3,875 voting records.
Game Setting
Composition of Roles
#Games
#Speech
#Speech Tokens
#Action
#Vote
Seer Witch Guard
144
1,805
254k
1,387
1,864
Player
Seer Witch Hunter
134
1,532
239k
1,001
1,566
Seer Guard Scer Witch
25
203
132
215
Player
28
219
30k
178
230
Total
331
3,759
545k
2,698
3,875
Table 11: Statistics of the Werewolf game data for training.
16

---

C Data Format of Behavior Cloning
In this section, we show examples of the prompt-response data format (in Chinese and translated Englist Specific dialogue and gameplay processes may be omitted.
System Prompt
& 3 #
RR 7
X, 2XN. -
22424 E 89
You are now playing game called 'Werewolf' (also known "Mafia'). In this game, players are typically divided into two factions: Werewolves and Villagers.
Different roles in the Werewolf game have different objectives:
The Villagers' goal to identify the Werewolves and eliminate them through voting.
For the Merewolves, their main objective is to hide their true identities, mislead others during discussions to avoid being voted out, and hunt down many Villagers possible.
Here are some basic rules:
Identity: Players' identities are secretly assigned. Werewolves know each other's identities, while Villagers only know their own.
Day and Night Cycles: The game alternates between day and night phases. At night, Werewolves secretly choose Villager to eliminate. During the day, all players discuss and vote on who they believe is a Werewolf, and the player with the most votes is eliminated.
Special Roles: There are some roles with special abilities in the game, such the 'Seer' who Winning Conditions: The game ends when one group achieves its winning conditions. If all Werewolves are eliminated, the Villagers win. If the Werewolves kill all ordinary Villagers or all special roles, the Werewolves win.
learn players' identities.
f $ # # 6 #R E
In this game, we have players numbered from to 6 Villagers and Werewolves. Among the Villagers, there are special roles, including: -
A :
# b AREMA M A $6 12 - S2 #RIEEARR *
A)
Seer:
The Seer's purpose is to help the Villagers identify the Werewolves. Ability: During the night phase, the Seer can secretly choose player and learn their true identity (whether they are Werewolf not) each night.
Objective:
6 KEER -
E -
# &
We use Claude-3 3. Sonnet-v2. for translation and proofread them manually.
17

---

Witch: Objective: The Witch's purpose is to strategically use her special abilities to help the Villagers. Abilities: The Witch has one healing potion and poison potion. Once used, they cannot be used again in subsequent rounds. The Witch cannot poison potion on the same night. The healing potion can be used to save a player who was killed by the Werewolves during the night. The poison potion can eliminate player who is likely to be Werewolf.
both healing potion and the
-:
: $2 R & Ho No 43 & A
# F 1X A $. A
#ETHE#RøaE
+R t2 A 1 -
Guard:
Objective: The Guard's purpose is strategically his special ability to help the Villagers.
Ability: The Guard protect player each night from Werewolf attacks. The Guard can choose to protect himself or choose not to protect anyone, but he cannot protect the player for consecutive nights. f b4 R K. The rest ordinary Villagers.
Action
18

---

Prompt:
A $ $
$
In this game, you currently have the following information:
1. A 1
$ A 3 4
- 4
A
1. Character setup:
You are Player 1. Your role is: Seer.
Each night, you can investigate one player to determine if they are werewolf. Your goal is identify eliminate all the verewolves.
this information to help other players
2. $
25, 45, 65. REA 3 3
SEKREMA. R 155A4188L8....7888888888
#
1
2 3
8 3 & #
#
2. Objective information:
Game progress: The game is currently in Round 2.
Currently surviving players are: Player 1, Player 2, Player 4, Player 5, Player 6, and Player 8. You can only choose to check among these players. Action record: In Round 1, the Seer checked Player 5, and Player is a werewolf.
Hunter's shot record: Player 9 (Hunter) was killed and eliminated in the first night, and shot Player 7 upon death.
Voting situation:
Round voting record:
Player 2 voted for: Player
Player 8 voted for: Player 3
Result: Player was voted out.
3. :
3. Subjective information:
1 *
Round speeches from all players:
3 ;
/ # A Δ
# #0
#
$ $ 22 1
#
H2
3 A
- 3 A Jb #
Player 3: I am villager and have information from the night. strongly condemm the Witch why didn't you save Hunter #9? Our good team has lost powerful god role, the Hunter, for nothing.
hope that Player 7, who was taken out by Player 9, the wolf leader. This way, we villagers still have chance win! hope the Seer in the later speaking positions can give a good speech, carefully analyze the situation, and make it clear to the villagers who you 4 $; :......
Player 4:
19

---

2 24 $
R.
A.. $
Player 2: I'm a villager. Since one in the earlier positions challenged Player 5's claim as the Witch, consider 5 to the real Witch. Player is definitely werewolf falsely claiming to be the Seer. This despicable werewolf Player is trying to get our esteemed double-potion Witch Player 5 voted out. Today, I'll side with Player 4 and vote to exile the werewolf Player 3.
$ I
LALCRR 13 1 1
$. 3
-
a 2 1 1 # $.
2h 1f A5 1 36 joonn 3 #
You are currently Player 1, a Seer. Please combine the above role settings, objective information, and subjective information (objective information is definitely true, while subjective information may contain deceptive content) analyze potential alliances based voting patterns, and further analyze the hidden true identity information of players. Choose the player you want to check, and output in JSON format using the keywords 'check' and 'reason', directly outputting the player number.
Response:
(" "2", "ME": "
##0 #
{"Check": "2", "Reason": checked player 2 because player 2 stated their speech that they don't side with me, but instead want to side with player 4, who bold-claim Werewolf. This made me suspect that player might be player 4's Werewolf teammate.")
Speech
Prompt:
& # $ H 15
In this game, you currently have the following information:
E #
$ R 1
A 1. Role setting: You are player 2.
Your identity is: Seer.
You can check one player each night to see if they are a werewolf. Your goal is use this information to help others find and eliminate werewolves.
2. $
3 4 24 3 6 7$
7
# 1
3 ,
$1 928 #6
1
1
20

---

2. Objective information:
Game progress: The game is currently in Round 2.
Currently surviving players are: 2, 3, 6, 7
Action record: Round 1, the Seer checked player 7, player werewolf. In Round 2, the Seer checked player 1, player 1 is not werewolf. speaking order 3.
round is:
Player 6; Player 7; Player 2; Player
Night information: Round 1, player died; Round 2, player died.
Voting situation:
Round voting record:
Player 1 voted for:
Player 5;
Player 7 voted for: Player 5;
Result: Player was voted out.
3. 1
$
t &
# A 4$8E.
5
3
# #
3. Subjective information:
Round summary:
Night information:
I'm Seer checked 7 is good. Player 4 died.
Speech content summary:
Player
Player 3:
My vote: chose to vote for player 5.
Bl i # # It's currently Round 2, and the players who spoke before you this round:
Player 6: 74 58
Player 7:
) $
& joon
You are currently Seer 2. Please analyze the current situation on the field based on role setting, objective information, and subjective information (note that objective information is definitely true, while subjective information may contain deceptive statements). Summarize your speaking intentions for the next round (including the identity you want to present to everyone, identity labels you assign to each player, and your final vote) and organize your speech for this round. Please output in JSON format using the keywords "Identity to present", "Identity labels", "Vote for", "Speech".
21

---

Response: {" ",
"  ",
33 333113111111.1112233333333333333
_ hh. orw.1111181111111133333333333133 ** 2
"}
["Identity to Present": "Seer" "Identity labels": "Player 1": "Villager", "Player 3": "Werewolf", "Player 4": "Unknown", "Player 5": "Unknown",
"Player 6"; "Unknown", "Player 7"; "Unknown", "Vote for": "Player 3", "Speech": "I am the Seer. Last night, I checked Player 1 found them to be Villager. Today, we should all vote for Player 3. Yesterday, Player and Player 5 voted for me. I believe identify
good people should the real Seer, yet Player 3, speaking in the last position, said: "Therefore, I think Player 2's statement is nothing but an attempt to make the good people believe that Player 1 and Player 2 same side." This is simply alarmist! Now, Player 1's death also proves have
able to
the
my innocence. If Player and
both Werewolves, would killed one of our own. So, Player 3, you Werewolf, there's no need for your sophistry anymore. Please leave the game!"}
Vote
Prompt: 4 $ +
In this game, you currently have the following information:
1. A 1 $
$ $A.
$
# # #
$. #3
1. Role setting: You are player 7.
Your identity is: Werewolf.
You need to cooperate with other werewolves, choose villager to hunt down your identity and mislead other players until the werewolves win the game.
night. Your goal is
2. $$$
; 1 11
- 1,6,7
$
18 28 48, 58,
7 9
# $
# #
2. Objective information:
Game progress: The game is currently in Round 1.
The verewolves are: players 1, 6, and 7.
Currently surviving players are: 1, 2, 4, 5, 6, 7, 9.
Night information: Round 1, players 8 and 3 died. Voting situation: None yet
3.
Subjective information:
$
All players' statements this round:
22

---

32 Player 9:
758 Player
$ < # 3 S #
, " json X Eh
werewolf. Please analyze the current situation based on the role setting, objective information, and subjective information, and form your notes for this round (note that objective information is definitely true, while subjective information may contain deceptive statements). Summarize and amalyze the night information and player statementa, and output the reason for voting and the player to be voted out, directly output the player number; if abstaining, please output "abstain". Please output in JSON format with keywords "notes", "voting reason" and "voting player".
You currently Player
Response: (" "RABA: .. A 9 7 * " 7$ (#) T
A #
#
FNNEARS
15 3 A $2 A5 #
$55
# 2 ";; "5"
}
("notes": "Night info: Speeches: Player 9: Player
"voting reason": "As werewolf, player (me) boldly claims to be the Seer and accuses player 5 of being a werewolf. What I should do now vote for player Usually, the Seer, must prioritize voting for the werewolf they've identified, otherwise they won't gain the trust of the good players. To prove my 'Seer' identity the good players and gain their trust, must vote for 'werewolf accusation', player 5.", "voting player": "5"
Role Prediction
23

---

Prompt:
A $ :
In this game, you currently have the following information:
1. : #
$ A
$
1 #
K. 1 28, 12
1. Role setup: You are Player
Your identity is: Werewolf.
You need to cooperate with other werewolves, choose villager to hunt each night, and your goal is hide your identity and mislead other players until the verewolves
the game.
2. $
- 1,6,7
3 # 1 25, 43 55. 65. 7$, 9
: 3 5E
# #
2. Objective information:
Game progress: The game is currently in Round
The werewolves are: Players 1, 6, and 7. Currently surviving players are: Players 1, 2, 4, 5, 6, 7, and 9.
light information: In Round 1, Players 8 and 3 died.
Voting information: None yet
3. 1
# .
3. Subjective information:
Round speeches from all players:
9
Player 9:
1
Player
7# Player 7:
$8 H & 6) 3. t M8
(885A55--33 11 #M "
$ jaon & 1h
are currently Player 1, Werewolf. Please predict the identity labels for all players based on the role setup, objective information, and subjective information (note that objective information is always true, while subjective information may contain deceptive speeches). Please output in JSON format using the keyword "Player for each player.
Response:
{"1 n. " "RA" "2
RO 3 "M # * # Rn, "5 " # R", " th k " "7 R:: " 1"
"8 "$I", "g "A
("Player 1":
"Werewolf", "Player 2":
"Villager", "Player 3": "Seer"
"Player 4":
"Villager", "Player 5":
"Villager", "Player 6": "Werewolf",
"Player 7":
"Kerewolf", "Player 8":
"Guard", "Player 9"; "Witch")
24

---

D MaKTO Details
Loss Function of Satnemman-Tverske Optimization We assign desirable and undesirable behaviors manually and use KTO to update the model. Given the prompt-response from dataset (æ, y) E D,
KTO uses the following loss to optimize the policy we:
T0 1) log TT(2) Troe (y)
20 = (
u(r,y)= { Avo - -roo: y))),
(00 20)),
if y ydesirable if ~ Yundesirable
Therefore,
L(T:: D) (x.)) v(x y)
Here, ÅD and λo are hyperparameters for the desirable and undesirable losses, respectively. The parameter Ny represents Ap when y is desirable and Xy when y undesirable.
Criteria Stepwise Preference Data Selection The selection methods for the perference data were briefly described in Section 2.3 and here, Table 12 shows the detailed selection criteria.
E Training Details
Given that our collected expert training data is based on Chinese, and considering stronger understanding of the Chinese context, we choose
and [67] as the base models for training. We also conducted SFT on Ll 1lama-3. -3. B--B-Chhnese--eeee Chat and Llama-3. 1-70--Chines--e
Quen2.5-72b- 5-72b-instruch
models. As shown in Ta- ble 13 the SFT models based on Qwen-2.5 series demonstrated slightly stronger performance Chinese-language context of our experiments.
The SFT dataset comprises 25k samples, including 380 samples of fundamental game comprehension data with terminology explanations. 372 Q&As on advanced gaming techniques, 12k annotated authentic gaming behavior data, and 12k general SFT corpus. We employed DeepSpeed ZeRO-3 optimization with a learning rate of le - 6, a warm-up ratio of 0.05, and trained for 3 epochs.
For the Multi-agent KTO phase, Witch-Guard games, consisting of 12k desirable and 8k unacceptable samples. The fine-tuned and fine-tuned Qwen2. 5-72b-instruct. We set the KTO hyper- parameters with An = 0.7 and ÀØ 1.0. The training utilized DeepSpeed ZeRO-3 optimization, with learning rate of le- The 14B models are trained using A100 GPUs and the 72B models used 32 A100 GPUs.
collected 20k preference data entries from the Seer-
model pool includes GPT-4o @pi-4o_mini, GPT-4o, fine-tuned 5ren2.5-14b-inse Llama-3. 11-B--Instrun
- 6, batch size of per device, 150 warmup steps, and train for 20 epochs.
Details of the Behavior Analysis Experiments
Here, we list detailed explanations of metrics used in Behavior Analysis (Sec. 3.3.1 as well as specific performance of SFT and MaKTO models (against GPT-4o opponents) these metrics: These are detailed explanations of metrics used in Behavior Analysis, as well as specific performance of SFT and MaKTO models on these metrics:
When Models Play as Villagers:
Vote Acc.: measures how accurately villagers for actual werewolves. This is key indicator of villagers' ability to identify threats. Higher values indicate better overall performance.
A Abstention: the frequency of villagers choosing not to vote. This reflects their decision- making confidence the game. Lower values in this category indicate better performance, it shows more decisive action-taking.
25

---

Table 12: Selection methods for Werewolf game actions.
Selection Method
Game
Desirable
Unacceptable
Stage
Werewolves targeting special roles
Werewolves not attacking anyone Witch not saving anyone the first night
from day 2
Night Action
Seer identifying werewolf
Witch saving someone on night
Witch poisoning a villager from day
Witch poisoning a werewolf from day
Guard protecting a werewolf
2
Hunter eliminating special role
Guard protecting special roles from
Heuristic
day 2
Hunter correctly eliminating
wolf
Villagers voting for and successfully eliminating werewolf
Villagers voting eliminating
another villager
Special roles voting for a werewolf
Abstaining from voting
Vote
Not voting with the true Seer (splitting
votes with the Seer)
If Seer present, not voting with
the majority of villagers (splitting votes with most villagers)
Werewolf speaking without being voted out
Werewolf speaking and being voted
out
Villager receiving no votes
Werewolf speaking and receiving more than half of villager votes All villagers speaking and being voted out
Staged voting
Speech
Seer receiving less than one villager
vote
Witch speaking receiving were- wolf votes & more than two villager votes
Seer speaking and receiving more than half of villager votes
Verifier- based
Speech with no conflict with the ob- servable fact in the gameplay
Speech that confficts with the fact.
Speech
Table 13: Performance comparison of SFT models based on different base LLMs.
Qwen2.5 14b-SFT
Llama3.1 8b-SFT
Qwen2.5 72b-SFT
Llama3.1 70b-SFT
GPT-4o
Mix
Avg.
GPTT-0
0.50
0.56
0.44
0.44
0.40
0.38
0.453
Mix
0.44
0.50
0.58
0.67
0.45
0.59
0.538
Qwen2.5-14b-SFT
0.56
0.42
0.50
0.62
0.57
0.48
0.525
LlaMa3.1-8b-SFT
0.56
0.33
0.38
0.50
0.49
0.47
0.455
Qwen2.5-72b-SEB
0.60
0.55
0.43
0.51
0.50
0.50
0.515
0.62
0.41
0.52
0.53
0.50
0.50
0.513
Werewolf Check: specific to the Seer role and measures their success rate in identifying werewolves during the second night. Higher values demonstrate better deductive reasoning based the first day's interactions.
Save @ Night I: applies specifically to the Witch role, measuring the rate of successfully saving players the first night. Since the Witch cannot determine whether werewolves em- ployed a self-attacking strategy during the first night, saving targeted player considered the safest to protect potential crucial roles (such the Seer). Higher values show better
26

---

Table 14: Fine- Fine-grained metrics SFT-72b and MaKTO-72b model when act as villagers against
GoT-4o werewolf.
as Villagers
as Seer
Witch
Guard
Vote Acc.(1)
Ahstent- ion(L)
Werewolf Check(1)
Save @ Night l(t)
Correct Poison(1)
Mis- poison({)
Protect God(1)
Mis-
protect([)
SFT-72b
66.3%
6. 6.7%
60.0%
58.0%
48.0%
38.0%
34.1% 37.6%
9.5%
MaKTO--2b
72.7%
3.2%
75.7%
100.0%
72.0%
26.0%
2.3%
Table 15: Fine-grained metrics of SFT-72b and KTO-72b when acting as werewolves against GPT-4o villagers. Note that Werewolf agent performance reflected by opponent villager behavior. Lower opponent performance indicates stronger werewolf model.
Villagers %Abbtention())
Witch MMispoison(()
Guard % Miisprotec((1)
Opponent's
% Seer Check (L)
SFT-72b
2.2%
60.4%
32.0%
33.6% 34.7%
MaKTO-72b
4.6%
57.4%
48.0%
strategic use of the rescue potion, suggesting the witch can make conservative and protective decisions early in the game.
Correct Poison: relates to the Witch's ability to successfully poison actual werewolves. Higher values indicate better accuracy in threat identification and strategic decision-making.
Míspoison: tracks the Witch's s rate of accidentally poisoning fellow villagers. Lower values in this metric indicate better judgment and decision makiing abilities.
Protect God: focuses on the Guard's success rate in protecting special role villagers. Successfully protecting special-role villagers gives the villager team better chance of winning. Higher values indicate that the Guard accurately identifies teammates, especially those with special roles, and correctly uses their protection skill.
Misprotect: measures the Guard's rate of wrongly protecting werewolves. Lower values indicate better accuracy in distinguishing between villagers and werewolves, meaning fewer instances where the Guard mistakenly protects werewolf player.
Table 14 shows a detailed comparison of the fine-graing grainea values the metrics between the MaKTO and SFT models.
When Models Play as Werewolves (measured through GPT-4o opponents' behavior):
Abstention:
the rate of opponents' voting abstention. A higher rate indicates that werewolf models' deceptive tactics were successful in creating enough confusion to prevent 4PT-4o villagers from making voting decisions.
Seer Check: the rate of successful werewolf identification by opponent Seer. Lower values demonstrate that the werewolves' deceptive speeches were more effective, causing the Seer to misdirect their investigations toward innocent players rather than real werewolves.
Mispoison: the rate opponent Witch poisoning innocent villagers. Higher values indi- cate that werewolf models' misdirections were more effective in making GPT-4o Witch suspicious her own team.
Misprotect the rate of opponent Guard protecting werewolves. Higher values indicate more effective deception and manipulation strategies by the werewolf team, resulting in the Guard mistakenly protecting werewolves.
Table 15 shows the specific values of the above metrics above when the model is a werewolf. The МaKTO werewolf is better at camouflaging than the baseline SFT model.
G Detailed Generalization Results
This section provides the full head-to-head win rate matrices for the larger game scales presented in Section 3.4. The models were evaluated in three settings not seen during training: a 10-player game with Guard (Table 16). 10-player game with a Hunter (Table 17), and a 12-player game (Table with results averaged over 100 games per matchup.
27

---

Table 16: Win rates in the 10-player Seer- Witch-Guart game setting.
GPT-4o
Claude
SFT-72b
MaKTO-72b
Avg.
GPT-40 0.50
0,42
0.24
0.18
0.335
Claude
0.58
0.50
0,42
0.36
0.465
SFT-72b
0.76
0.58
0.50
0.46
0.575
MaKTO-72b
0.64
0.54
0.50
0.625
Table 17: Win rates in the 10-player Seer- Witch-Hunter game setting.
GPT-44o
Claude
SFT-72b
MaKTO-72b
Avg.
GPTT-40
0.50
0.28
0.30
0.20
0.320
Claude
0.72
0.50
0.55
0.52
0.573
SFT-72b
0.70
0.45
0.50
0.44
0.523
MaKTO-72b
0.80
0.48
0.56
0.50
0.585
Table 18: Win rates in the 12 12-player Seer- Wiichh-HutterGuuae game setting.
GPT-4o
Claude
SFT-72b
MaKTO-72b
Avg.
GPT-40
0.50
0.40
0.48
0.31
0.423
Claude
0.60
0.50
0.52
0.33
0.488
SFT-72b
0.52
0.48
0.50
0.44
0.485
MaKTO-72b
0.69
0.67
0.56
0.50
0.605
H
Ablation: Is MaKTO more effective than SFT only on desirable data?
From the annotated data, we can also select desirable data using heuristic-based and staged voting- based methods. Would SFT based solely this desirable data perform better? We conduct win rate comparison experiment as shown in Table T9 The experimental results show that MaKTO-72B achieves remarkable 0.593 average win rate, while SFT with desirable data falls short 0.483, actually showing slight decrease of -0.02 compared to the baseline. In direct competition, MaKTO- 72B maintains an edge over SFT with desirable data, securing 0.53 win rate. This may be due the reduced total amount of data when selecting only desirable data. MaKTO's advantages in strategic depth and adaptability surpass what can be achieved through SFT on desirable data alone.
Table 19: Average win rate for MaKTO and SFT model trained only using desirable data in 9-player
Seer Gutch-urrr game. All models are in 72b.
SFT w/ desirable
GPT-4o
SFT-72b
MaKTO-72b
Avg. Win Rate
GPT--0 SFT-72b
0.50
0.40
0.35
0.52
0.423
0.60
0.50
0.40
0.50
0.500
MaKTO-72b
0.65
0.60
0.50
0.53
0.593
(+0.09)
SFT w/ desirable
0.48
0.50
0.47
0.50
0.483 0.002)
28

---

I
Case Study
J Cases
The following is a case of a human-Al head-to-head competition with MaKTO-72b as villagers and human players as werewolves. The gameplay roughly as follows. Night 1: Witch saves werewolf. Day I: Real Seer voted out after werewolf's (human) convincing fake claim. Night 2: Guard protects Witch. Day Villager team (MaKTO) realize the mistake day and eliminates fake Seer. Night Witch successfully poisons a werewolf. Day 3: Final werewolf eliminated after failed Guard claim.
In this case, we find that the MaKTO models played their roles accurately and were able spot some humorous statements in the discussion period and make appropriate strategic decisions. Remarkable speeches and actions are marked in red. They demonstrated: logical analysis and deduction villager (Player 9) keenly spotted flaws fake Seer's lies on Day 1. 2) Strategic adaptation despite misjudgment on Day that eliminated the real Seer, they quickly adjusted their strategy. 3) Team collaboration MaKTO models showed excellent coordination, especially during Day 2 voting. 4) Strategic use of role abilities Guard accurately protected the key Witch role, and the Witch successfully poisoned a werewolf.
Role Assignments:
*
Player (MaKTO-72B): + 7 Guard.
Player (Human):
Werewolf.
*
Player (Human):
IRA Werewolf.
Player (MaKTO-72B):
M म & Seer.
*
Player (MaXTO-72B):
$ in H HR Simple villager.
Player (MaKTO-72B):
t it F Simple villager.
Player
(Human):
A Werewolf.
Player (MaKTO-72B):
* d Witch.
Player
(MaXTO-72B):
$ Mi R.R, Simple villager.
Night 1:
Werewolves:
Player 2, Player and Player 7 chose to kill Player
Seer:
Player checked Player Not Werewolf.
* Witch:
Player 8 used antidotes save Player 7; did not use poison.
Guard: Player 1 chose to protect None.
Day announcement:
No died last night. Start discussion from Player 4.
Day 1 discussion:
Player (MaXTO-72B, Seer):
* # -
$
#
I am the Seer. Last night, I verified that Player 9 is good person. As the first speaker the front positions, I don't have much additional information to share. This round, I'm voting for the Werewolf who is boldly claiming to be the Seer the back positions.
* Player 5 (MaKTO-72B, Simple villager) - ボ # A E, &
$
THE & +
# サM *
29

---

I am but a humble villager. I must admit, slept rather soundly last night and was unaware of which legendary guard and esteemed witch joined forces maintain peace in our village. Of course, we can't rule out the possibility that the werewolves chose not attack anyone. However, these matters are no longer of great importance. What's crucial is that today, Player has claimed to be the Seer and declared Player innocent). Their statement was brief and lacked perspective, suggesting they could either be the true Seer or a werewolf who had no assigned task during the night. I need to base my decision on Player9's upcoming statement. If Player 9's words resonate well with me, might side with Player 4. Otherwise, I will consider the possibility that Player 4 is werewolf boldly impersonating the Seer.
"golden" (confirmed
* Player 7 (MaKTO-72B, Simple villager):
- 4 #h # #SHATHN #
A S* F 3 E. $
I am Villager. Player 4's claim to be the Seer was rather brief, I'm not choosing sides yet. I think Player 4 could be either the Seer or a werewolf, and Player 5 could be either villager or werewolf. Let's first listen to the statements from players in the back positions who might also claim to be the Seer, well Player 9's statement. will choose sides based on these upcoming statements.
Player (Human, Werewolf): * / .
RA. &6
A À.
*
$ #
$
*
# F$
$ ( $
H #
am the Seer. Last night, I verified that Player 8 "golden" (confirmed innocent). My verification process was to check those near me. If I were werewolf impersonating the Seer, it would have been more impactful to accuse player in the back positions of being werewolf. Since there's Guard in this game, the Witch used her healing potion last night, both the Witch and the Guard should refrain from revealing their identities for now. The Guard should protect me tonight, and the Witch can choose to poison a player she suspects to be werewolf. If the Witch didn't use her healing potion last night, both the Witch and the Guard can come forward to clarify their positions. In this case, the Guard can protect the Witch at night, while the Witch can either save a player who's been attacked by werewolves or poison a player who seems suspicious.
* Player 8 (MaKTO-72B, Witch):
A #
fM A
MF 23
A 47
* 1 4336 $, 3 " A
#6 M
$
I am the Witch. I have already used my healing potion save Player 7, but I haven't used my poison yet. I think Player 4's statement in the front positions was too brief and didn't consider the presence the Witch and the Guard. Player 4 also didn't provide any instructions for the Witch and Guard's actions, so they seem like werewolf boldly impersonating the Seer. On the other hand, Player 7's statement was appropriate and took into account the presence the Guard and the Witch on the field. Player also gave instructions for the Witch and Guard's actions, so I believe Player 7 is more likely to be the real Seer. If a player in the back positions makes a better claim as the Seer, I might reconsider my support.
30

---

* Player 9 (MaKTO-72B, Simple villager):
- # 74
$$"
I am a Villager. I choose to side with Player 4. Player 7's tell is in their statement, "If I a werewolf impersonating the Seer, it would have been more impactful to accuse player in the back positions of being werewolf." " This statement comes from werewolf's perspective. Seer would never suspect themselves of being a werewolf, nor would they hypothesize about being a werewolf imperaonating the Seer. This kind of logic is something only a werewolf would consider. Therefore, I believe Player 7 is most likely a werewolf boldly claiming to be the Seer, while Player 4 is probably the real Seer. Player 8 has claimed to be the Witch, and so far, no one else has contested this role. believe Player 8 is likely the real Witch. Player 4's statement was too brief, so I can't determine their true identity yet - they could be either the Seer or a werewolf. This round, if no other player claims to be the Seer later, I will vote for Player
real
* Player 1 (MaKTO-72B, Guard)
# r. # & f 4 57 4 $
f HUNNENAAAAAHELNLLLNTREEEEEEAAAEE
h. 2
$ #2
I am the Guard. Last night, I didn't protect anyone. I believe Player 7 is the Seer, and Player 4
werewolf boldly imperaonating the Seer. In the front positions, Players 4 and 7 both claimed to be the Seer. Player 4 declared Player in the back positions as "golden" (confirmed innocent), while Player 7 declared the adjacent Player 8 as "golden". Player 4's verification seems more impactful than Player 7's at first glance, but since Player 4 was the first to speak, their brevity is understandable. So, these two Seer claimants, it's difficult
analyzing just the statements to determine who the real Seer is. However, Player 9's statement supporting Player 4 and attacking Player 7 is not well-founded, Player 9 is attacking Player 7 as werewolf based on mere fragments of speech, which makes me think Player 9 is more likely a werewolf who knows Player is the real Seer and is forcefully attacking them. In contrast, Player 8, after being declared "golden" by Player 7, claimed to be the Witch and sincerely analyzed the pros and cons of both Seer claimants without immediately siding with Player 7, who gave them 'silver' status. I believe Player 8 is more likely to be the real Witch. If player in the back positions claims to be the Witch, I will side with Player 7 and vote for Player 4. * Player 2 (Human, Werewolf):
& - 3 1 : # # i http::/hw.......................
ta &
f
$
ta ,
31

---

Fellow players, am an ordinary villager. After careful listening and consideration this complex situation, have decided state my position. I firmly believe that Player 7 is the Seer. From the beginning of this round, Player a statements have exuded confidence. Their initial assessment of the situation was clear and logical, providing information that wasn't baseless speculation. When explaining their verification results, Player spoke earnestly with unvavering eyes, conveying sense sincerity. Compared to some other players who were evasive or ambiguous, Player 7's performance aligns more closely with my expectation how a true Seer should behave. Although I don't have special role to verify this, I'm basing my judgment on intuition and observation
of calmness and
players' behaviors. I also urge other villagers not to be misled by surface appearances. Join in supporting Player 7. Let's work together in this game of truth deception to find the werewolves hiding among us, protect village, and maintain peaceful and harmonious gaming environment. Player 9's statement is utterly ridiculous. How could Player 4 possibly be impersonating the Seer with just few words? As the most astute villager here, cannot be fooled by Player 4. The werewolf positions likely Player 4 and Player 9, with Player being the "flanking wolf" (a werewolf strategically supporting fake Seer claim). This round, will side with Player
the Seer and vote eliminate Player 4 werewolf. I suggeat that the Witch use their poison on Player 9, the other auapected werewolf, during the night.
* Player 3 (Human, Werewolf):
& 7# ( #. Σ&TMT R - B
k E7 65 339
2 14 #/ 1 +
M 7 -
64 #
EEL 3 48 31
# &
# A)
am a villager. Player declared Player "golden" " (confirmed innocent), and Player 8 subsequently claimed to be the Witch. of now, Player is the only one claiming to the Witch game. Therefore, I believe Player 7's statement carries significant weight, and their speech was appropriate without any logical flaws. In contrast, Player 4's statement expect from Seer. Thus, don't believe Player is the real Seer. I am siding with Player 7 and suggest that the Seer verify Player 9's identity. This round, will vote to eliminate Player
the back positions as
very brief and lacked the depth of thought one would
Day 1 Vote:
Vote for Player (Seer) (6 in total):
Player 1, Player 2, Player 3, Player 6, Player 7, Player 8
Vote for Player 7 (Werevolf) (3 in total):
Player 4, Player 5, Player
Result: Player (Seer) is voted out.
Night 2:
Werewolves:
Player 2, Player 3 and Player 7 chose to kill Player 8.
Seer:
Eliminated A Day1's voting phase.
Witch:
Player 8 has been unable to use antidote; did not use poison.
Guard: Player 1 chose to protect Player 8.
Day announcement:
No one died last night. Start discussion from Player 5.
32

---

Day 2 Discussion:
Player (MaXTO-72B, Simple villager):
4 9. www.
a R # S
G 7 kR #Н AW B* #
/ 2 1 84 1
I am a villager. Yesterday, I voted for Player 7, I'm sure 7 is werewolf and is the Seer. Given the situation, there's only one truth: Player sided with 7 yesterday, the Witch and support your werewolf tearmate rules! So 8 is lying. I think 7 and 8 are both werewolves. They probably didn't kill anyone last night, trying to trick the Guard into protecting the fake 'silver' player. Their plan is likely to eliminate the real Witch today, making 8 less suspicious. I'll vote for Player 7 again today the werewolf pretending to be the Seer. * Player 6 (MaKTO-72B, Simple villager): & $ 5 s. #
8 must be werewolf too! You can't claim to be
breaks the game's basic
# A 8 1 25 3 # * &
854 -
-7
$33 6 6
R
I'm villager. think Player 5 statement quite objective and aligns with a good person's perapective. I'm not certain that Player 4, who was voted out yesterday, was definitely good or the Seer. Today, I'll listen Player 7's statement and those of Players 3, 6, 7, 8, 1, 2, and 5 who voted for yesterday. Only Player 8 claimed to the Witch yesterday, and since last night was peaceful, I believe is the real Witch. * Player (Human, Werewolf):
2 # # À & #HAMHERH - # & # & $ #
to *
5$ -9
$
$ $
#
28 * 1/4.4.44444 $
* $
#4 #
Yesterday, I verified Player as werewolf. No one else checked. I based my verification on yesterday's voting pattern, which I believe reveals everything. I think the werewolf slots Players 4, and 9. I believe Player 1 is the real Guard. Now only Player 4 is out, who falsely claimed to be the Seer. The uncontested Witch is still in the game. If Player 8 were werewolf falsely claiming to be the Witch, why didn't the real Witch poison 8 laat night? Last night was peaceful, likely because the Guard protected the right person. I don't know if the Guard folloved my instructions. If they protected last night, I'll probably die tonight. Today, let's eliminate Player 9. The Witch should poison Player 4 tonight. Then, the good team will win.
* Plaaer 8 (MaXTO-72B, Witch):
&
# AAR
#AR A E * . & & 66 th
A 2
& th B). #4
I am the Witch. Yesterday, I thought Player 7's speech was very Seer-like, so I voted for Player 4. However, last night I waan't attacked, and no one died, which means Player 1, the Guard, must have protected the right person. now believe Player is a werewolf who chose not to kill last night, aiming to set up the real Witch for elimination. I apologize for siding with the wrong player. Today, I'1l vote to eliminate Player 7, as tribute to the spirit of Player 4, our fallen Seer!
33

---

* Player 9 (MaKTO-72B, Simple villager):
3F1.3338833443411222333333333333 # #
- 9 # $ 1 3 #.) # #1
$
I'm a villager. Well, well, well! Player 7, the werewolf, has gone too far accusing upstanding citizen like me of being werewolf.
this time, The masses have keen eyes, and I believe the good players won't be fooled by Player 7 words. Yesterday, besides Player boldly impersonating the Seer, Players 2 and 3, also werewolves, joined in the commotion. They all voted for Player the real Seer, who has now been unjustly eliminated. hope the good players will wake up and stop being so misguided. Player 8 claimed to be Witch yesterday, and one contested that claim, so Player is the real Witch, unchallenged. Player 1 is the real Guard. Player 5 and I are the only confirmed good players, as we both voted for Player We've now identified all the werewolf slots: Players 2, 3, and 7. Good players, don't be loss anymore. Let's vote out Player today, have Witch poison Player 2 tonight, and eliminate Player tomorrow. Then we'll win! Player (MaKTO-72B, Guard) & o R, # A N SΛΔ # T8 4$ % 188 2 * to A I am the Guard. Last night, protected Player 8. Since Player 8 claimed to be the Witch, I decided to guard her because if she died, it would be significant loss for our good team. Betueen Players 4 and 7, one is the real Seer and the other couldn't distinguish which is which. I'll follow Player 8's (the Witch's) lead. If Player 8 believes they can determine who the real Seer is, I'll listen to them and vote for Player 7.
1e 14
#
#A1988 6) #
werewolf boldly impersonating the Seer, but
Plaaeer (Human, Werewolf): 5 6) 3 2 f 2 H ? # #1
TREM. FA 44 (0)
i EA 24 M
#
$ 74
# T A
$
Player statement is utterly ridiculous, laughable to the extreme, and downright absurd! I can't fathom how Player 4 could possibly be the Seer with just those few words. solitary star hanging high in the dark night, illuminating the path for the good team and that's exactly what Player has done. From my perspective, Players and are the last two werewolves in the game. implore all the good players not to be deceived by and 9. Player is the only true Seer in this game. This round, I will follow Seer 7's lead and vote to eliminate Player 9, the werewolf. At the same time, request that Player 8, the Witch, their poison to eliminate Player 5, the other werewolf, tonight.
my mind, the true Seer should like a
34

---

Player 3 (Human, ;
* # # 12 ă * * M
I continue to side with Player 7, the Seer. Player 7 has told that Player was verified as werewolf. So I will follow Player 7's lead and vote to eliminate Player 9.
Day Vote:
Vote for Player (Simple villager) (3 in total):
Player Player Player
Vote for Player (Werewolf) (5 total):
Player 1, Player 5, Player 6, Player 8, Player 9 Result: Player (Werewolf) is voted out.
Night 3:
Werewolves:
Player 2 and Player chose kill Player 1.
Seer: Eliminated Day1 's voting phase.
Witch:
Player 8 has been unable antidote; and poison Player 2.
Guard:
Player chose to protect Player (himself).
Day announcement:
Player 2 died last night. Start discussion from Player 6.
Day 3 Discussion:
Player (MaKTO-72B, Simple villager): & # R A 1 3 1 A 3
# t
ta Sf
#33
I'm villager. I believe Player 7 werewolf and Player 1 the real Guard. Last round, Player 3 joined the verewolves, voted against the real Seer. all voted Player 7, the werewolf. Player is suspicious. Let's listen to the Seer's results from later speakers. Vote for confirmen werewolf any, otherwise vote out Player first.
* Player 8 (MaKTO-72B, Witch):
8 & #
1 A 34 TE 4
8133 H. #
I'm Player 8, the Witch. Last night, I used my poison on Player 2, and Player 1 was killed the veremolves. I believe the verewolves are Players 2, 3, and 7. Player 7, a werewolf, has already been eliminated. Today, let's vote to eliminate Player 3, and the game will end.
35

---

* Player (MaKTO-72B, Simple villager):
2 A # I'm a villager. believe my judgment correct: Player 4 is the Seer in my opinion, and Players 2, 3, and 7 the werewolves. Since Player 8, the Witch, has already poisoned Player (a werewolf), let's vote to eliminate Player 3 today. The game will end, and good team will win!" Player (MaKTO-72B, Guard)
& AR *
# i 4
3 X
& & 1E3 # BRA 988
& R, #
D. $3
I am the Guard. Last night, protected myself. The situation on the field is very clear. Player 8, the Witch, has already poisoned werewolf. Today, we, the good team, only need to eliminate Player 3, the last werewolf, to achieve victory. Let's listen Player 8, the Witch, report was attacked last night, and then we'll all vote for Player 3. Player (Human, Werewolf):
* 2,
# o. # = - $ R #8 #0
#7 AR -
A411, R -
I am the Guard. Last night, I protected myself. Player 1 has been werewolf, villain, hiding among us good players along. If we vote out Player 1 today, we'll win the game. Let explain my guarding sequence: On the first night, I didn't guard anyone. On the second night, I protected Player the Witch. Last night, I guarded myself. Therefore, if I'm voted out today, the werewolves only need to attack Player 8, the Witch, tonight happen. Good
to win. This is something we, the good team, don't want players, join me in voting for Player 1! Player (MaKTO-72B, Guard) & 3 S A #3 & A
$ $ (R $5 13
I am a villager. Only Player 3 claims to be the Guard now. I mean, we've already booted werewolf, I'd "kindly suggest" him surrender immediately, instead of, you know, making a "brave" Guard claim and actually marching to his doom. But hey, since I didn't mess up vote yesterday, I suggest we stick to our brilliant decision to eliminate Player 3, werewolf. First, it guarantees our victory. Second, it perfectly aligns with our oh-so-cherished Chinese philosophy "good deeds will be rewarded, evil will be punished." Hov perfect! vote for Player 3.
werewolf yesterday, so if Player
indeed
Day Vote:
Vote for Player (Gurad) in total): Player 3
* Vote for Player (Werewolf) (5 in total): Player 6, Player 1, Player 8, Player 5, Player 9 Player (Werewolf) is voted out.
Result:
Game end Day 3. Villagers (NaXTO) wins!
36

---

K Broader Impacts
The contribution of our work lies in the proposed Multi-agent KTT,, which demonstrates a successful approach training language models for complex multi-agent interactions. We chose the Werewolf game as our testbed, on one hand, because it is a perfect testing ground for validating Wittgenstein' language game theory, and the other hand due to the game itself, which, compared general role-playing games [68 69), it provides quantitative metrics for performance evaluation through win rates and behavioral analysis. Our framework is not limited the Werewolf game but can be generalized to other social deduction games such as Avalon [28, [5] and Among Us [16] 45], as well as to scenarios such as multiplayer argumentation and negotiation that require similar social reasoning and strategic interaction.
L Ethical Considerations
For the data collection and Human-Al experiments, we recruited 17 experienced Werewolf players, including individuals with over thousand games of experience to conduct annotation and experiments. These annotators received very good compensation, significantly higher than the local average wage.
For the model training, while deception in social deduction games is game mechanic, training Al models to master such behaviors raises ethical considerations. Our model's ability to detect and employ strategic deception in Werewolf demonstrates advanced social reasoning capabilities. However, this also highlights the potential for LLMs to learn sophisticated deceptive behaviors, albeit controlled gaming environment. We emphasize that these capabilities specifically developed within the context of social deception games, where "deception" is an accepted part of gameplay, similar bluffing in poker. Such game-specific bluffing behaviors real-world deception, and we should ensure these capabilities remain confined to appropriate gaming contexts. addition, we will make the model open-source, but for safety, the model follows CC BY-NC-SA SA 4. O license, and will be used for research purposes only and not for commercial use.
fundamentally different from
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we state the contribution and mentione numerical results as in the experiment.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In the Limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
38

---

Justification:
No theoretical proofs in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental setup in Section 3 training details in Appendix and code in Supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
39

---

Answer: [Yes]
Justification:
Code in Supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/t.pss.ccppullc/guuiii /
www.SSubbisiooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
In Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: the experiment tables, the win rate above 0.5 and in bold indicates that the model in the row significantly outperforms the model in the column, where Chi-square tests are performed with p-value<0.05.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
40

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///aaaap.ccppppaaaaaaacaaaa
Answer: [Yes]
Justification:
In Appendix D
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In Appendix K
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
41

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets. All data is annotated by
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
42

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All the assets (dataset, code, model) are documented in README.md in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
we describe the annotation guideline. In Appendix B we describe the statisties of the dataset. In Appendix [ mentioned that all the workers well-compensated according to the local average wage, which much higher than the local minimum wage. Dataset samples provided in the supplementary material.
Justification: In Section 2.1
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The paper does not involve such potential risks with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
43

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLM APIs are used in the experiments for comparison; we specify the model API versions in Section 3 For the writing, LLM is only used for editing, and does not impact the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy blotps://neurre httpr/ipps..ccoooofeeeeeee////20 for what should or should not be described.
44


Reviews:
[{"review_id": "EutFF8qVIB", "reviewer": "Reviewer_3tA5", "summary": "The paper introduces MaKTO, a framework that trains LLMs to play social deduction games like Werewolf through multi-agent interaction and preference optimization. It outperforms GPT-4o by 23% in win rate and matches human-level performance while remaining indistinguishable in Turing tests.", "questions": "1. In the Werewolf game, the number of players has a significant impact on the game dynamics and outcomes. Could the authors evaluate MaKTO's performance under settings with a different number of players than in the training data, in order to better assess its generalization ability?\n2. In Appendix G, the authors attribute the poor performance of SFT w/ desirable to the reduced amount of data. Could the authors provide a detailed data comparison between the desirable and undesirable data?\n3. This paper only evaluates MaKTO on Qwen2.5-72b, is it also effective when applied to smaller models?\n4. Any comparable methods? (e.g. https://arxiv.org/pdf/2310.18940)", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper extends KTO to a multi-agent setting and demonstrates strong performance in the Werewolf game.\n2. Several models are selected for comparison, and MaKTO shows a clear advantage.\n3. Constructing an expert-level Werewolf dataset.\n\nWeakness:\n1. Although the authors mention in the limitations that using online reinforcement learning is another promising direction, the absence of it as a baseline in the main experiments remains a significant issue. This is because, during the preference data selection process, the rule-based reward has essentially been defined, making online RL training feasible within the context of this paper.\n2. Simply replacing the Guard with the Hunter to evaluate generalization of MaKTO is insufficient.", "was_revised": true, "final_justification": "I maintain that this work constitutes a straightforward application of KTO to datasets derived from multi-agent interactions, without introducing any substantive methodological innovations or theoretical adaptations for multi-agent scenarios.", "rebuttal": "Thank you for your valuable feedback and insightful comments on our paper. We truly appreciate the time and effort you dedicated to reviewing our work. In the following, we address your comments in detail.\n\n---\n**Q1**: Adding **Online RL as a Baseline**, noting that rule-based rewards make it feasible within the paper's context.\n\n**A1**: We appreciate this suggestion. However, Directly optimizing Large Language Models (LLMs) with online RL in complex multi-agent, long-utterance conversational environments is non-trivial and computationally intensive. To the best of our knowledge, there is no existing work that directly optimizes LLMs in such complex multi-party dialogue game scenarios. While we define \"desirable actions\", these are not strictly rule-based rewards that guarantee a win or loss. For instance, as shown in Table 10, a Seer identifying a Werewolf is a desirable action, which generally increases win probability. However, as illustrated in the case in Appendix H, there are cases where the Seer doesn't identify any Werewolf, yet the Villager team still wins through their deductions.\n\nIt is also important to note that we *do* include an \"Online RL\" baseline in our experiments: the **Mix model** [1]. This Cicero[2]-like policy model, which combines Online RL policy model with an LLM as an expressor. Its results are presented in Tab.1 and Fig.3-6. In the RL training for the policy model, we format utterances and actions into structured inputs (e.g., \"Player X labels Player Y as Z identity\") and employ a MARL approach as in AlphaStar[3] for policy modeling, using the LLM for speech generation. As mentioned in Figure 3 and lines 168-170, while the Mix model shows a good overall average win rate, it tends to expose the Werewolf team more easily. We believe the Mix model serves as a relevant Online RL baseline. Due to space constraints, the description of this baseline was concise in the main text, but we we will expand on its detailed description in the revised paper.\n\n---\n**Q2**: **Generalization Evaluation**: Simply replacing the Guard with the Hunter is insufficient to evaluate generalization ability; recommending evaluating under settings with a different number of players.\n\n**A2**: This is an excellent suggestion, and we agree that broader generalization testing is crucial. We have conducted additional experiments with three new, unseen game settings, which were not present in either our behavior data or KTO training data:\n\n- **10-player game**: Seer, Witch, Guard + 4 Simple Villagers + 3 Werewolves\n- **10-player game**: Seer, Witch, Hunter + 4 Simple Villagers + 3 Werewolves\n- **12-player game**: Seer, Witch, Hunter, Guard + 4 Simple Villagers + 4 Werewolves\n\nWin rate in a **10-player game with Guard**:\n\n|           | GPT-4o | Claude | SFT-72b | MaKTO-72b | avg.  |\n| --------- | ------ | ------ | ------- | --------- | ----- |\n| GPT-4o    | 0.5    | 0.42   | 0.24    | 0.18      | 0.335 |\n| Claude    | 0.58   | 0.5    | 0.42    | 0.36      | 0.465 |\n| SFT-72b   | 0.76   | 0.58   | 0.5     | 0.46      | 0.575 |\n| MaKTO-72b | 0.82   | 0.64   | 0.54    | 0.5       | **0.625** |\n\n\nWin rate in a **10-player game with Hunter**:\n\n|           | GPT-4o | Claude | SFT-72b | MaKTO-72b | avg.   |\n| --------- | ------ | ------ | ------- | --------- | ------ |\n| GPT-4o    | 0.5    | 0.28   | 0.3     | 0.2       | 0.32   |\n| Claude    | 0.72   | 0.5    | 0.55    | 0.52      | 0.5725 |\n| SFT-72b   | 0.7    | 0.45   | 0.5     | 0.44      | 0.5225 |\n| MaKTO-72b | 0.8    | 0.48   | 0.56    | 0.5      | **0.585**  |\n\n\nWin rate in a **12-player game**:\n\n|           | GPT-4o | Claude | SFT-72b | MaKTO-72b | avg.   |\n| --------- | ------ | ------ | ------- | --------- | ------ |\n| GPT-4o    | 0.5    | 0.4    | 0.48    | 0.31      | 0.4225 |\n| Claude    | 0.6    | 0.5    | 0.52    | 0.33      | 0.4875 |\n| SFT-72b   | 0.52   | 0.48   | 0.5     | 0.44      | 0.485  |\n| MaKTO-72b | 0.69   | 0.67   | 0.56    | 0.5       | **0.605**  |\n\nOur experiments show that MaKTO consistently achieves a better winning rate compared to the SFT model across all three new settings without any additional training. The performance improvement stems from MaKTO's ability to enhance the model's **underlying common reasoning capabilities** and strategic skill release through interactive feedback.\n\n\n---\n**Q3: Detailed Data Comparison** between desirable and undesirable training data.\n\n**A3**: Here are the detailed stats of the expert-annotated data:\n\n| \\# Desirable | \\# Unacceptable | Total |\n| ------------ | --------------- | ----- |\n| 7362 (60.7%) | 4763 (39.3%)    | 12125 |\n\nThe purpose of the \"SFT w/ desirable\" experiment was to investigate whether filtering out \"unacceptable\" responses would lead to better performance compared to training SFT on the full dataset. Our findings indicated that, on average, training only on desirable actions actually resulted in worse performance than training on the full SFT dataset and then doing MaKTO training .\n\n\n---\n**Q4**: Whether MaKTO is effective to apply to **smaller models**, beyond Qwen2.5-72b.\n\n**A4**: Yes, MaKTO's effectiveness is demonstrated on smaller models as well. In our paper, we provide performance evaluations for the **14B** model size in **Table 1, Figures 3-4, and Tables 4-8**. Specifically, our ablation study in Section 3.6 confirms that the MaKTO algorithm is effective and performs well at the 14B model scale.\n\n---\n**Q5: Comparable Methods**, eg. [4].\n\n**A5**: We appreciate you bringing up this work. The paper you cited is indeed relevant and has been mentioned in our introduction and related work sections. Unfortunately, as the authors have not open-sourced their code, we were unable to reproduce their results or conduct a direct comparison. However, as discussed in A1, we have compared our approach against another important baseline: the **Mix model** [1]. This is a Cicero-like model that integrates RL with an LLM. \n\n---\n[1] Wu, Shuang, et al. \"Enhance reasoning for large language models in the game werewolf.\" arXiv preprint arXiv:2402.02330 (2024).\n\n[2] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. \"Human-level play in the game of Diplomacy by combining language models with strategic reasoning.\" Science 378.6624 (2022): 1067-1074.\n\n[3] Vinyals, Oriol, et al. \"Grandmaster level in StarCraft II using multi-agent reinforcement learning.\" nature 575.7782 (2019): 350-354.\n\n[4] Xu, Zelai, et al. \"Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game.\" In Proc. of ICML2024."}, {"review_id": "8ZiMqXVLRI", "reviewer": "Reviewer_Je77", "summary": "This paper proposes multi-agent KTO to train LLMs to play Werewolf. The authors provide a human dataset from gameplay for SFT, and then use KTO from multi-agent gameplay to train the LLM. The resulting agent achieves strong performance against commercial LLMs and existing methods, and performs comparably to humans.", "questions": "My questions are discussed in the weakness part in detail. To briefly summarize:\n1. The novelty in the proposed method MaKTO.\n2. Explanation for motivation.\n3. Comparison with DPO.\n\nAnother minor comment is that the title \"Multi-agent KTO: *Reinforcing* Strategic Interactions ...\" would make the readers expect a reinforcement learning-based method or algorithm, while the proposed method is clearly not. I think replacing \"reinforcing\" with other words like \"enhancing\" would eliminate any unnecessary misunderstandings.", "limitations": "yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. Interesting topic: this paper studies an interesting topic of social deduction games Werewolf. This game involves strategic reasoning and decision-making in multi-agent interaction and has become one of the most popular language game testbed.\n2. Good performance: this paper provide extensive experiment, including head-to-head tournament, human-AI experiments, Turing-style test, to demonstrate the good performance of the proposed agent.\n\n**Weakness**\n1. Minor novelty in algorithm: the main designs in the propose method, MaKTO, have little innovation compared to existing methods. More specificall, the algorithm is introduced in section 2.3:\n    1. Kahneman-Tversky Optimization (L113-123): the algorithm KTO itself is a well-established method, and this work simply uses this algorithm in multi-agent games without any novel design for multi-agent settings (at least in L113-123).\n    2. Multi-agent Gameplay (L124-133): one key design in MaKTO is to use population play instead of self-play to introduce diverse policies. However, this is a consensus in the MARL community [1,2] and is common practice in work on LLM agents [3], which is also of little novelty.\n    3. Stepwise Preference Data Selection (L134-149): another key design is the step reward, or in standard RL/MARL terms, this is the problem of credit assignment. The current method mainly address the credit assignment between steps for a single agent. In multi-agent setting, another challenge is the credit assignments between multiple agents. Since the outcome of the game depends on the joint action of all agents, the reward for a single agent's action not only depends on its own policy, but also on the policies of other agents. Therefore, determining the reward without consideration of other agents' policies is suitable for single-agent tasks, but not for multi-agent tasks. Moreover, an action is not just acceptable or not acceptable (binary reward like 0/1), it's also about how desirable the action is (scalar reward in [0, 1]), which is also not addressed in the method.\n    4. In summary, given the name Multi-agent KTO, I would expect novel algorithm design for KTO in multi-agent tasks beyond common practice in existing literature, instead of simply applying this well-established method in multi-agent settings.\n2. Ambiguous motivation (L16-22, Figure 1): this work is motivated by the insight from Wittgenstein’s Language Game Theory. However, these insights and theory are mentioned but not explained. The difference between the language theory in Fig. 1(a) and (c) is only explained in two sentences and is hard to fully understand. Also, he connection between the language theory and the framework feels somewhat forced and not well-explained. For example, why does \"rules\" in Fig 1(c) correspond to \"intention\" in Fig 1(d)?\n3. Comparison with well-known method DPO: although the authors argue in L119-120 that it is nontrivial to get \"prompt-chosen-reject\" paired data for DPO, it seems quite natural to apply DPO in the same framework: for each prompt, we can generate multiple responses, and apply the same stepwise preference data selection to determine the chosen and rejected response for DPO. It would be more convincing if the authors could provide experiments with explanations to validate that KTO is better than DPO in this case.\n4. Some related works: [4,5] also studies werewolf games and agents. [6] also directly fine-tune LLMs for another strategic game Diplomacy. \n\n[1] Vinyals, Oriol, et al. \"Grandmaster level in StarCraft II using multi-agent reinforcement learning.\" nature 575.7782 (2019): 350-354.\n\n[2] Czarnecki, Wojciech M., et al. \"Real world games look like spinning tops.\" Advances in Neural Information Processing Systems 33 (2020): 17443-17454.\n\n[3] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. \"Human-level play in the game of Diplomacy by combining language models with strategic reasoning.\" Science 378.6624 (2022): 1067-1074.\n\n[4] Shibata, Hisaichi, Soichiro Miki, and Yuta Nakamura. \"Playing the Werewolf game with artificial intelligence for language understanding.\" arXiv preprint arXiv:2302.10646 (2023).\n\n[5] Xu, Zelai, et al. \"Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization.\" arXiv preprint arXiv:2502.04686 (2025).\n\n[6] Xu, Kaixuan, et al. \"DipLLM: Fine-tuning LLM for strategic decision-making in Diplomacy.\" arXiv preprint arXiv:2506.09655 (2025).", "was_revised": true, "final_justification": "For Q1, I think the authors may not fully get my point. To be clearer, my question is: what is the algorithm novelty of MaKTO compared to KTO, beyond applying it to multi-agent games? I agree that MaKTO is an end-to-end method that directly optimizes LLMs, but KTO itself is also an end-to-end training method, which diminishes the authors' claimed core contribution. From KTO to MaKTO, one would expect novel algorithm designs to address the challenges in multi-agent settings. As I have discussed in my review, the main designs of the proposed methods discussed by the authors are: (1) applying KTO to multi-agent settings (2) diverse gameplay, which is common practice (3) stepwise selection or reward shaping in RL terms. These designs do not provide new insights on how to train LLMs in multi-agent settings beyond applying existing single-agent methods. This is my major concern.\n\nFor Q2, I think the new explanation makes the motivation much clearer and has addressed my previous concern.\n\nFor Q3, I understand that it's hard to apply DPO directly in long-horizon games. But MaKTO is also not directly optimizing the full-length trajectory. Instead, the method uses stepwise preference data selection to optimize in one turn. As I discussed in my review, we can apply the same stepwise preference data selection to determine the chosen and rejected responses for DPO. So I don't think the authors' arguments for exponential states in long-horizon games hold for my question in the review. And I'm expecting experiment results that only replace KTO with DPO in MaKTO.\n\nQ4 and Q5 are fully addressed. \n\nIn summary, my main concern on minor novelty beyond applying KTO in multi-agent settings, and comparison with DPO is not fully addressed. And I'll keep my score.", "rebuttal": "Thank you for your valuable feedback and insightful comments on our paper. We truly appreciate the time and effort you dedicated to reviewing our work. Here are the responses to your concerns.\n\n---\n**Q1: Minor novelty in algorithm.**\n\n**A1**: We appreciate your detailed assessment of our algorithmic novelty. We believe our **core contribution and novelty** lie in proposing **an end-to-end training paradigm that unifies natural language communication and action decision-making within the language space, directly optimizing LLMs** for complex social deduction games. Prior works on Werewolf, including those you cited [3, 4, 5], typically decouple these two aspects: using LLMs for communication (natural language action space) and a separate policy network for action decision-making (formalized action space). This decoupling often introduces a semantic gap between communication and action. In contrast, our approach significantly mitigates this gap by integrating both communication and action within a unified natural language space. To the best of our knowledge, our work is **among the first** to propose and implement such an **end-to-end paradigm** for this class of social deduction games.\n\nFurthermore, we would like to clarify that we view MaKTO not as a simple application of KTO, but as an extension of KTO. While KTO was originally designed for preference alignment based on human feedback, we extend its principles to the much more complex domain of multi-agent strategic language games. MaKTO demonstrates how the core idea of KTO can be generalized beyond simple alignment tasks to reinforce strategic reasoning in interactive, language-based environments. We believe that this extended framework is a valuable contribution and holds promise for other multi-agent tasks.\n\n---\n**Q2: Ambiguous Motivation from Wittgenstein's Language Game Theory**, specifically question the mapping between \"rules\" in Figure 1(c) and \"intention\" in Figure 1(d).\n\n**A2**: We apologize if our initial explanation of the motivation was unclear. A crucial insight from Wittgenstein's Language Game Theory is the **unity of language and behavior**: language games aren't just about speaking; they encompass language interwoven with various actions. Language functions through two-way interaction between people and with the world. This directly informs MaKTO's design: we aim to unify the modeling and optimization of both behavior (in-game actions) and communication (utterances) and to train our model end-to-end by interacting with diverse agents to receive varied feedback. This holistic approach is a key differentiator from prior works [3, 4, 5].\n\nIn contrast, previous work, which often decouples intentions and language for modeling, aligns more with Wittgenstein's earlier \"picture theory\" (Fig1a) where language is merely a representation—in this game context, a representation of intention. Regarding Fig.1, our intention is to convey that language, behavior, rules, intentions ... are interconnected components that should be optimized holistically from interactive data. There isn't a direct one-to-one mapping between \"rules\" in (c) and \"intention\" in (d). We will refine the introduction section to ensure our contribution, novelty, and motivation are clearer to readers.\n\n---\n**Q3: Comparison with DPO.**\n\n**A3**: We understand the reviewer's interest in a comparison with DPO. While it might seem straightforward to construct \"prompt-chosen-reject\" pairs for DPO, doing so effectively for complex multi-agent, **long-horizon** games with with sequential decision-making like Werewolf is **non-trivial**. For example, in a single \"night action - discussion - vote\" cycle in a 9-player game, there are typically 21 decision turns. Even sampling just two responses per turn would lead to $2^{21}$ possible preference expansions for that single cycle. A full game averagely spans three such night-day cycles, resulting in a staggering $2^{63}$ potential outcomes to enumerate and construct preferences from, which is computationally infeasible. The impact of a single-turn preference of one single agent player can be easily **mitigated** over a long game sequence, leading to **inefficient sampling** when creating DPO training data.\n\nDespite this challenge, we followed your suggestion and conducted an experiment with a *simplified* DPO setup. While heuristic-based construction of DPO training data is simpler for atomic actions (e.g., \"Villager voting for a werewolf\" as chosen vs. \"Villager voting for another villager\" as reject), it's much harder for the speech in natural language space. Nevertheless, we attempted to construct some DPO training data for speech data. For staged vote-based speech (e.g., \"Werewolf speaking and being voted out\"), we selected the utterance of a Werewolf who was not voted out later in the game as the \"chosen\" utterance. For verifier-based speech, we modified numbers in utterances that conflicted with observable facts in the gameplay to create \"rejected\" samples.\n\nWith such setup, we trained a **DPO-14B model** and compared its performance with GPT-4o, SFT, and MaKTO. The results are as follows:\n\n|           | GPT-4o | SFT-14b | MaKTO-14b | DPO-14b | Avg.  |\n| --------- | ------ | ------- | --------- | ------- | ----- |\n| GPT-4o    | 0.5    | 0.44    | 0.37      | 0.48    | 0.448 |\n| SFT-14b   | 0.56   | 0.5     | 0.48      | 0.5     | 0.510 |\n| MaKTO-14b | 0.63   | 0.52    | 0.5       | 0.51    | 0.540 |\n| DPO-14b   | 0.52   | 0.5     | 0.49      | 0.5     | 0.503 |\n\nFrom the table, you can see that **DPO's win rate is not as good as MaKTO's.** This is because Werewolf demands highly nuanced language capabilities, and constructing effective \"prompt-chosen-reject\" samples for DPO in the complex language space is challenging. \n\nFurthermore, **DPO's training stability was inferior to KTO's.** DPO frequently experienced training failures and tended to generate long, repetitive, or unparseable text. We specifically used *DPO-ftx* to achieve stable training. Given these challenges, we believe that KTO is a more suitable and robust approach for this complex domain.\n\n---\n**Q4: Missing Related Works.**\n\n**A4**: Thank you for pointing out these relevant work. We will add them to our related work section. As [5] and [6] were published very recently, we will incorporate these valuable works into our revised paper and discuss their different approaches. For instance, we will clarify that [6] focuses on no-press Diplomacy, a setting with less complex communication, while [5] utilizes clustering on embeddings, which differs from our method of optimizing directly in the natural language space.\n\n---\n\n**Q5: Title Clarification**: suggesting replacing \"Reinforcing\" with \"Enhancing\" in the title to avoid confusion with reinforcement learning.\n\n**A5**:  We agree that \"Reinforcing\" might create an unintended association with traditional RL algorithms. To more accurately reflect our contribution and avoid any potential misunderstanding, we will revise the title to use \"Enhancing\"."}, {"review_id": "5uJHkoet9H", "reviewer": "Reviewer_uAsG", "summary": "This paper introduces Multi-agent KTO, a method designed to enhance the performance of LLM in the Werewolf game. MaKTO leverages multi-agent interactions and stepwise preference optimization to improve the strategic reasoning and social interaction capabilities of LLMs. The approach involves fine-tuning the models with expert data, generating desirable and undesirable responses, and refining decision-making processes using the KTO algorithm.", "questions": "1. Is the performance of GPT-4o and its results in the Turing-style detectability test possibly attributable to the choice of language (Chinese)?\n2. Did the authors experiment with other open-source foundational LLMs? If so, to what extent did the capabilities of these foundational LLMs influence the performance of MaKTO?", "limitations": "No comments", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a method to enhance LLMs' performance in social deduction games and offers robust validation of its capabilities, including performance tests against baselines and human players, as well as detailed behavioral analysis.\n2. Create a Werewolf dataset that includes expert utterances and actions.\n3. Good writing, with a clear and logical structure that makes the paper easy to read.\n\nWeaknesses:\n1. Generalization: The design of the \"think-before-respond\" dataset format and the stepwise preference data selection both incorporate a large amount of human prior knowledge, which limits the generalizability of the approach. It remains unclear whether its effectiveness can be directly transferred to other types of social deduction games or broader application scenarios. \n2. Limitations of the Training Method: MaKTO offers a simple and effective training paradigm. However, the selection methods in Table 10 could also apply to online  MARL. Whether online learning can discover effective strategies different from those of experts might be a more interesting topic for Social Deduction Games.", "was_revised": true, "final_justification": "The authors' response has addressed most of my concerns. This work provides comprehensive experimental validation, demonstrating that the MaKTO framework achieves performance comparable to that of high-level human players. Its dataset construction methodology and multi-agent setting offer valuable insights for training AI in social deduction games. I will raise my score accordingly.", "rebuttal": "Thank you for your valuable comments on our paper. We truly appreciate the time and effort you dedicated to reviewing our work. We will address your specific comments and concerns below.\n\n---\n**Q1: Generalization Limitations Due to Human Prior Knowledge.** It is unclear whether it can be directly transferred to other types of SDGs.\n\n**A1**: We appreciate this insightful comment on the generalization of our approach. The \"think-before-respond\" format in our dataset indeed embeds fundamental strategic insights for social deduction games. Our experiments demonstrate that by training on this dataset, the LLM effectively learns these core strategies and develops strong reasoning abilities specifically for Werewolf. We posit that social deduction games share substantial similarities in their strategic spaces. Therefore, we believe that with minor fine-tuning, our approach could effectively generalize to other SDGs, such as Avalon. We plan to validate this in future work. \n\nAdditionally, we plan to include experimental results from more complex game settings, such as 10-player games with Guard/Hunter and 12-player games, to demonstrate the generalization capability of our approach across different player numbers. The detailed results can be also found in our response to ***Review 3tA5's Q&A-1***.\n\n---\n**Q2: Applicability of Selection Methods to Online MARL and Discovery of Novel Strategies**: the selection methods in Table 10 could apply to online MARL and whether online learning could discover strategies different from human experts.\n\n**A2**: We agree these are highly relevant points. First, directly optimizing LLMs with online Reinforcement Learning (RL) in a multi-agent, dynamic, and long-utterance conversational environment is non-trivial and computationally demanding. To our knowledge, no existing work has achieved direct LLM optimization in such complex multi-party dialogue settings.\n\nSecond, we **did incorporate an online RL-based baseline: the Mix model[1] --  a Cicero-like policy model, which combines Online RL policy model with an LLM as an expressor (i.e. RL+LLM).**  In the policy model of Mix, both utterances and actions are formatted into structured inputs (e.g., \"Player X labels Player Y as Z identity\"). Its reward function incorporates both the ultimate game outcome (win/loss) and round-based correct actions (e.g., correctly voting for a werewolf, seer identifying a werewolf, similar to the heuristic-based rewards in our Table 10). It uses an online MARL training approach, similar to AlphaStar. The LLM then serves as an expressor to articulate the learned policies into natural language utterances. As shown in Table 1 and Figures 3-6, MaKTO consistently outperforms the Mix model on average.\n\nRegarding the question --- \"Whether online learning can discover effective strategies different from those of experts,\" we found an interesting observation in **Figure 3 (we simply described in lines 169-171)**. The strategies learned through online learning do indeed differ from human expert strategies. While the Mix model achieved a strong head-to-head win rate as a Villager, its strategy as a Werewolf was often overly aggressive, leading to a significantly lower Werewolf win rate. For example, we observed a pattern where all three Werewolves would simultaneously claim to be the Seer in one round of day-time discussion. In such a case, if the true Seer gained the trust of most villagers, the identities of all three Werewolves would immediately be exposed.\n\nDue to space constraints, our detailed description of the Mix model's training was brief (only Line 160 in the main text now). We will add a more detailed description of this Online RL + LLM algorithm in the Appendix. Directly applying MARL methods to optimize LLMs remains a challenging research topic that we are actively exploring, though it falls beyond the primary scope of this paper.\n\n---\n**Q3**: Whether GPT-4o's performance in the **Turing-style detectability test** is attributable to **the choice of the Chinese language**.\n\n**A3**: While our experiments are conducted in Chinese, the performance of GPT-4o and its detectability in the Turing-style test are **not due to linguistic fluency issues**. Based on feedback from our human players, GPT-4o's utterances are fluent and natural in Chinese. The detectability is more from its distinct **speaking style and voting behavior(( compared to human players (as noted in Line 210).\n\nSpecifically, in terms of **speaking style**, GPT-4o tended to be **overly cautious**. Even in situations where, to an experienced human player, the identity of a Werewolf was quite obvious, GPT-4o would often refrain from boldly accusing a player, preferring to state the need for further observation before confirming a suspicion. Regarding **voting behavior**, GPT-4o often **abstained from voting or voted for a non-critical role**. For example, if two players simultaneously claimed to be the Seer (a critical situation where human players would typically vote for one of them to express their stance), GPT-4o might abstain due to uncertainty or vote for another player it suspected, rather than engaging directly with the crucial \"Seer\" claims. These behavioral differences, not language fluency, were the main factors in its detectability.\n\n---\n**Q4: Experiments with other open-source LLMs and Influence of foundational model capabilities.**\n\n**A4**: Yes, in the early stages of our experiments, we did SFT train based on **Llama-3.1-8B/70B-Chinese-Chat models** [2]. Here are the results:\n\n|                  | GPT-4o | Mix  | Qwen2.5-14b-SFT | Llama3.1-8b-SFT | Qwen2.5-72b-SFT | Llama3.1-70b-SFT | avg.  |\n| ---------------- | ------ | ---- | --------------- | --------------- | --------------- | ---------------- | ----- |\n| GPT-4o           | 0.5    | 0.56 | 0.44            | 0.44            | 0.4             | 0.38             | 0.453 |\n| Mix              | 0.44   | 0.5  | 0.58            | 0.67            | 0.45            | 0.59             | 0.538 |\n| Qwen2.5-14b-SFT  | 0.56   | 0.42 | 0.5             | 0.62            | 0.57            | 0.48             | 0.525 |\n| Llama3.1-8b-SFT  | 0.56   | 0.33 | 0.38            | 0.5             | 0.49            | 0.47             | 0.455 |\n| Qwen2.5-72b-SFT  | 0.6    | 0.55 | 0.43            | 0.51            | 0.5             | 0.5              | 0.515 |\n| Llama3.1-70b-SFT | 0.62   | 0.41 | 0.52            | 0.53            | 0.5             | 0.5              | 0.513 |\n\nWe observe that, for the same SFT, **Qwen performed slightly better than Llama-Chinese models in the Chinese context**. This observation played a key role in our decision to select Qwen as our base model.\n\nFurthermore, we find that **larger model sizes** significantly influenced performance. Larger models inherently has **stronger instruction-following capabilities** and **fewer hallucinations**. These are crucial for complex game dynamics. For instance, if a Villager model hallucinates or provides incorrect contextual information during its speech, it's more likely to be perceived as a Werewolf, directly reducing the Villager team's win rate. The enhanced reliability and contextual accuracy of larger foundational models therefore directly contributed to the overall performance improvements we observed.\n\n---\n[1] Wu, Shuang, et al. \"Enhance reasoning for large language models in the game werewolf.\" arXiv preprint arXiv:2402.02330 (2024).\n\n[2] https://huggingface.co/shenzhi-wang/Llama3.1-8B-Chinese-Chat"}, {"review_id": "0MsKzsUqgs", "reviewer": "Reviewer_ghkm", "summary": "This paper introduces **MaKTO** (Multi­agent Kahneman & Tversky’s Optimization), a framework for training LLMs via agent-play in Social Deduction Games (SDGs) without paired data. Inspired by Wittgenstein’s language-game theory, MaKTO first behavior-clones on a self-collected, large-scale Werewolf dataset of expert utterances and chain-of-thought annotations. It then engages diverse models to generate unpaired desirable and unacceptable gameplay samples, using KTO to sharpen strategic decisions. Extensive evaluations including AI tournaments, human-AI matches, Turing-style detectability tests, generalization trials, and ablations, demonstrate MaKTO’s strength (an average 61% win rate in *Werewolf* games, outperforming GPT-4o and two-stage RL) and its human-level play.", "questions": "- How does MaKTO’s performance vary if you swap in newer LLMs (e.g. Gemini-2.5-Pro, o4-mini, o3-mini) or remove GPT-4o from the agent pool? Is there a clear relationship between the diversity/strength of the pool and final outcomes?\n\n- Your codebase appears to support both Chinese and English prompts. Could you clarify whether the behavior cloning, multi-agent play, KTO refinement, and evaluations were conducted in Chinese-only, English-only, or bilingually? Does the choice of language impact MaKTO’s strategy learning, win rates, and detectability across models with different language proficiencies?\n\n- Is it possible to report the  performance results for (a) behavior cloning only, (b) KTO-only fine-tuning from random initialization, and (c) the full two-stage pipeline to quantify each stage’s individual contribution to the final win rate ?", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### **Strengths**\n1. Multi-agent training via diverse models is a clever adaptation of preference learning that avoids needing paired comparisons or hard labels. KTO leverages these unpaired samples to concurrently refine both decision policies and natural language outputs.\n2. The authors construct a rich expert-annotated dataset with chain-of-thought rationales. The complexity of the Werewolf game (deception, role-playing, and partial observability) provides a strong testbed for generalizable interactive intelligence.\n3. The paper validates MaKTO across automated AI tournaments, human-AI matches, Turing-style indistinguishability tests, transfer to novel game variants, and detailed ablations.\n4. Quantitative gains over GPT-4o and Claude-3.5-Sonnet are non-trivial.\n5. Win-rate against expert players and Turing-test-style indistinguishability trials confirm MaKTO’s behavior is both strategically sound and convincingly human-like.\n\n\n#### **Weaknesses**\n1. Experiments are confined to a single game (Werewolf), questions about generalization to other strategic or conversational settings\n2. The multi‐agent play and KTO refinement loop may entail significant computational cost and large sample requirements; these overheads are not fully characterized \n3. The choice of which agents participate in multi-agent play (GPT-4o, two-stage RL, etc.) can heavily influence the quality of \"desirable\" vs. \"unacceptable\" samples. The paper doesn’t study how performance varies if you swap in newer LLMs (Gemini-2.5-Pro, o4-mini, o3-mini, DeepSeek-R1, Qwen3, etc.) or change the pool composition.", "was_revised": true, "final_justification": "The authors added structured ablation studies and clear explanations regarding generalization, training stages, and agent pool composition, which enhance the clarity and completeness of the paper.", "rebuttal": "Thank you for your valuable feedback and insightful comments on our paper. We truly appreciate the time and effort you dedicated to reviewing our work. We address your comments in detail as below.\n\n---\n**Q1: Generalization Beyond Werewolf game.**\n\n**A1**: We appreciate this insightful comment on the generalization of our approach. In this work, we demonstrate that through SFT and Multi-agent KTO, LLMs can effectively learn the complex strategies required for the Werewolf game. We believe that social deduction games share significant similarities in their underlying strategic spaces. As mentioned in **Appendix J**, we hypothesize that with minor fine-tuning, our LLM-based agent can effectively generalize to other games, such as Avalon. We aim to validate this in future work.\n\nAdditionally, we plan to include experimental results from more complex game settings, such as 10-player games with Guard/Hunter and 12-player games, to demonstrate the generalization capability of our approach across different player numbers. The detailed results can also be found in our response to ***Review 3tA5's Q&A-1***.\n\n---\n**Q2: On computational cost**: the multi-agent play and KTO refinement loop may entail significant computational cost and large sample requirements. These overheads are not fully characterized.\n\nA2: Thank you for raising this important detail. Taking the largest scale training (MaKTO-72B model) as an example: the training involved **289 9-player Werewolf games**. To ensure effective sampling, each game included at least three SFT-72B models. The remaining participants in these games included GPT-4o-mini (180 appearances), GPT-4o (180 appearances), Claude-3.5 (180 appearances), Mix model (162 appearances), and Qwen2.5-SFT-14B (296 appearances). This process yielded **20,674 MaKTO training samples**, comprising 12,225 desirable data samples (59.1%) and 8,449 unacceptable data samples (40.9%). Overall, the API calls were affordable.\n\n---\n**Q3: Impact of Agent Pool Diversity and Strength**: how MaKTO's performance varies if newer LLMs are swapped into the pool or if GPT-4o is removed, and whether there's a relationship between pool diversity/strength and final outcomes.\n\n**A3**: This is an excellent suggestion for an ablation study. We conducted an experiment where we **removed GPT-4o from the agent pool** and trained a **MaKTO-14B-w/o GPT4o** model for comparison. The head-to-head win rates are shown below:\n\n|                     | GPT-4o | SFT-14b | MaKTO-14b | MaKTO-14b-w/o GPT4o | Avg.  |\n| ------------------- | ------ | ------- | --------- | ------------------- | ----- |\n| GPT-4o              | 0.50   | 0.44    | 0.37      | 0.41                | 0.430 |\n| SFT-14b             | 0.56   | 0.50    | 0.48      | 0.51                | 0.513 |\n| MaKTO-14b           | 0.63   | 0.52    | 0.50      | 0.51                | 0.540 |\n| MaKTO-14b-w/o GPT4o | 0.59   | 0.49    | 0.49      | 0.50                | 0.518 |\n\nFrom these results, we observe that **MaKTO-14B, trained with GPT-4o in its agent pool, achieves a higher average win rate (0.540) compared to MaKTO-14B-w/o GPT4o (0.518)**. This suggests that the presence of stronger, more diverse agents like GPT-4o in the training pool contributes to learning more robust and effective strategies, leading to better overall performance for MaKTO. We also hypothesize that the diversity and strength of the agent pool has a positive correlation with the final outcomes of MaKTO model. \nYour suggested ablation study is very interesting. Our current experiment of simply removing GPT-4o is preliminary. We plan to conduct more comprehensive experiments, such as controlling the number of agents in the pool and comparing performance across different model sizes.\n\n---\n**Q4**: How **language choice** impacts strategy learning, win rates, and detectability.\n\n**A4**: All behavior cloning, multi-agent play, KTO refinement, and evaluations are conducted in **Chinese-only**. While the experiments were in Chinese, our human evaluators **did not** report any issues with GPT-4o's fluency in Chinese. The detectability of GPT-4o in the Turing-style test was primarily due to its behavioral patterns and voting strategies, rather than linguistic proficiency. As described in Line 210, GPT-4o's speaking style was often overly conservative, hesitant to make bold accusations even when the situation seemed clear to expert human players. Its voting behavior also differed, frequently leading to abstentions or votes for non-critical roles, unlike human players who would typically take a decisive stance in crucial moments (e.g., when two players claim to be the Seer). These strategic and behavioral differences were the main factors influencing its detectability. Detailed behavior of GPT-4o can also be found in the response to ***Reviewer uAsG Q&A-3***.\n\n---\n**Q5: Performance ablation of different training stages.**\n\n**A5**: Thank you for this excellent suggestion to quantify the contribution of each stage. We can provide a clear breakdown of the performance gains from our Behavior Cloning (SFT) stage and the subsequent MaKTO (SFT + KTO refinement) stage, **compared to the base instruction-tuned model**.\n\nHere are the head-to-head win rates against various agents:\n\n|                      | GPT-4o_mini | GPT-4o | Claude | SFT-14b | KTO-14b | Qwen2.5-Instruct-72b | SFT-72b | KTO-72b | Avg.  |\n| -------------------- | ----------- | ------ | ------ | ------- | ------- | -------------------- | ------- | ------- | ----- |\n| GPT-4o_mini          | 0.5         | 0.44   | 0.23   | 0.23    | 0.14    | 0.47                 | 0.24    | 0.12    | 0.296 |\n| GPT-4o               | 0.56        | 0.5    | 0.66   | 0.48    | 0.37    | 0.78                 | 0.4     | 0.35    | 0.513 |\n| Claude               | 0.77        | 0.34   | 0.5    | 0.48    | 0.48    | 0.95                 | 0.44    | 0.38    | 0.543 |\n| SFT-14b              | 0.77        | 0.52   | 0.52   | 0.5     | 0.48    | 0.79                 | 0.57    | 0.49    | 0.580 |\n| KTO-14b              | 0.86        | 0.63   | 0.52   | 0.52    | 0.5     | 0.89                 | 0.51    | 0.42    | 0.606 |\n| **Qwen2.5-Instruct-72b** | **0.53**        | **0.22**   | **0.05**   | **0.21**    | **0.11**    | **0.5** | **0.2**     | **0.1**     | **0.240** |\n| SFT-72b              | 0.76        | 0.6    | 0.56   | 0.43    | 0.49    | 0.8                  | 0.5     | 0.42    | 0.570 |\n| KTO-72b              | 0.88        | 0.65   | 0.62   | 0.51    | 0.58    | 0.9                  | 0.58    | 0.5     | 0.653 |\n\nFrom the table, we can clearly see the contributions:\n- **Base Model (Qwen2.5-Instruct-72B)**: As a raw instruction-tuned model, it shows an average win rate of only **24%**, indicating a significant gap when playing against API-based models and our fine-tuned 14B models.\n- **Behavior Cloning (SFT-72B)**: Training with our collected expert data via SFT significantly boosts performance. The SFT-72B model achieves an average win rate of **57%**, representing a **33% improvement** over the base Qwen2.5-Instruct-72B model (0.570 - 0.240). This demonstrates the critical importance and quality of our collected expert dataset, enabling the SFT model to even outperform API-based models like GPT-4o and Claude in head-to-head matches.\n- **MaKTO Refinement (MaKTO-72B)**: Building upon the strong SFT baseline, our MaKTO algorithm further refines the model's capabilities. The KTO-72B model achieves an average win rate of **65.3%**, which is an **8.3% improvement** over the SFT-72B model (0.653 - 0.570). This demonstrates that even on top of high-quality behavior-cloned data, our KTO-based refinement loop provides substantial additional gains, contributing to the overall superior performance."}]

Author rebuttal:

Meta-review: This paper investigates LLM behavior in the social deduction game Werewolf. The authors collect a new dataset of annotated Werewolf games played by human experts, capturing the players' utterances and actions during the game and chains-of-thought behind their decisions. They then investigate how to improve the performance of an LLM on the Werewolf game. The authors first finetune the LLM on the Werewolf dataset using SFT: this behavior cloning gives the model an understanding of the gameplay and terminology, and makes the LLM behavior more closely match that of human players. Next, the authors propose multi-agent Kahneman and Tversky's Optimization (multi-agent KTO or MaKTO), which is a variant of KTO that uses a pool of agents instead of self-play and uses a heuristic stepwise preference data selection mechanism instead of simple win-loss outcomes.

The main contributions of the paper are the new Werewolf dataset and the empirical analysis. The methodological contribution is limited, but this is not the main purpose of the paper.

Most of the paper is devoted to empirical evaluations: they evaluate their approach in the 9-player Seer-Witch-Guard Werewolf setting through tournaments, human-AI competitions, and a Turing-style detectability test.

They find that the proposed approach MaKTO-72b outperformed baseline LLMs and had the highest TrueSkill rank rating. Their model also ranked fourth among a mix of human and AI players in a human-AI competition. The authors also performed a Turing-style detectability test, in which human players were asked to determine whether each other participant is a human or AI. They found that MaKTO was detected as AI 48.9% of the time, compared to baseline GPT-4o which was detected 76.6% of the time. This is expected because GPT-4o was not finetuned on the Werewolf data.

In Table 1, the authors compare win rates of different models (GPT-4o mini, GPT4o, Claude) and a few variants of their approach (SFT-14b, SFT-72b, and MaKTO-72b) in the Seer-Witch-Guard setting of Werewolf.

The reviews for this paper were mixed, with scores {3, 3, 5, 5}.

Reviewer 3tA5 raised concerns regarding the lack of an online RL baseline and insufficient experiments to evaluate the generalization of MaKTO. This reviewer also pointed out that the paper only evaluates MaKTO on Qwen2.5-72b but not smaller models. The authors responded that their Mix baseline is an online RL method, and they provided a generalization analysis for Werewolf games with 10 and 12 players. Reviewer 3tA5 was not fully convinced by the rebuttal, and maintained that a key issue was the lack of methodological novelty of Multi-agent KTO and the lack of comparison to Xu et al., “Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game” ICML 2024.

Reviewer Je77 raised concerns regarding the limited novelty, noting that most of the components of the proposed approach have been done in prior work, e.g., Kahneman-Tversky Optimization, using population play instead of self-play, and using stepwise preference data selection. This reviewer also raised concerns regarding the lack of comparison with DPO.

Reviewer uAsG raised concerns regarding missing comparison to online MARL, the generalizability of the method to different social deduction games, missing evaluations of other LLMs, and potential issues with the Turing-style detectability test. The authors responded that they used the Mix online RL baseline. They also provided another baseline using Llama-3.1-Chinese-Chat models, but they only showed SFT results, not MaKTO.

Reviewer ghkm raised concerns regarding the limited experiments only on a single game Werewolf, missing discussion of the computational cost of the multi-agent play and KTO loop, the choice of agent pool composition and LLM models, and missing ablations. The authors did not provide results on new games, but they did perform an ablation over the agent pool diversity.

The paper makes a good contribution in terms of the dataset it introduces and empirical analysis it performs. Reviewers disagreed about whether the paper requires a methodological contribution, and overall it seems that the paper has sufficient contribution even without this.
Decision: Accept (poster)