DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
Authors: Kaixuan Xu
Jiajun Chai
Sicheng Li
Yuqian Fu
Yuanheng Zhu
Dongbin Zhao
Venue: ICML 2025 poster (2025)
Keywords: LLM-based agent
Diplomacy
fine-tuning
Primary area: reinforcement_learning->multiagent

Abstract: Diplomacy is a complex multiplayer game that re- quires both cooperation and competition, posing significant challenges for AI systems. Traditional methods rely on equilibrium search to generate extensive game data for training, which demands substantial computational resources. Large Lan- guage Models (LLMs) offer a promising alterna- tive, leveraging pre-trained knowledge to achieve strong performance with relatively small-scale fine-tuning. However, applying LLMs to Diplo- macy remains challenging due to the exponential growth of possible action combinations and the intricate strategic interactions among players. To address this challenge, we propose DipLLM, a fine-tuned LLM-based agent that learns equilib- rium policies for Diplomacy. DipLLM employs an autoregressive factorization framework to sim- plify the complex task of multi-unit action assign- ment into a sequence of unit-level decisions. By defining an equilibrium policy within this frame- work as the learning objective, we fine-tune the model using only 1.5% of the data required by the state-of-the-art Cicero model, surpassing its per- formance. Our results demonstrate the potential of fine-tuned LLMs for tackling complex strategic decision-making in multiplayer games.

DipLLM: Fine- Tunnng LLM for Strategic Decision-making in Diplomacy
Kaixuan Xu 12 Jiajun Chai
Sicheng Li
Yugian Fu
Yuanheng Zhu? 21
Dongbin Zhao
Abstract
Overview of DipLLM
Diplomacy is a complex multiplayer game that re- quires both cooperation and competition, posing significant challenges for AI systems. Traditional methods rely on equilibrium search to generate extensive game data for training, which demands substantial computational resources, Large Lan- guage Models (LLMs) offer promising alterna- tive, leveraging pre-trained knowledge to achieve strong performance with relatively small-scale fine-tuning. However, applying LLMs to Diplo- macy remains challenging due to the exponential growth of possible action combinations and the intricate strategic interactions among players. To address this challenge, we propose DipI.I.M, a fine-tuned LLM-based agent that learns equilib- rium policies for Diplomacy. DipLLM employs an autoregressive factorization framework sim- plify the complex task of multi-unit action assign- ment into a sequence of unit-level decisions. By defining equilibrium policy within this frame- work as the learning objective, we fine-tune the model using only 1.5% of the data required by the state-of-the-art Cicero model, surpassing its per- formance. Our results demonstrate the potential of fine-tuned LLMs for tackling complex strategic decision-making in multiplayer games.
Human-like Model
Game Data
Fine-tuning
(Learning Objective)
Autoregressive Factorization
Environment
Equilibrium Search
LLM-based Agent
(Diplomacy)
(DipLLM) A
Data Collection
Fine-tuning
Agent Performance & Dataset Size
Training Games(I))
Agent Score(%)
23.0
20.8
60
20
45
15
30--
10
6.6
15--
5
1.8
DipNet
DORA
Cicero
Ours
Figure I. An overview of agent, DipLLM and performance of various agents in a population.
action space that can exceed 10 possible choices per turn. This complexity stems from the game's mechanics, where each player controls up 34 units, each with around 26 possible actions. The exponential growth in possible action combinations, coupled with the intricate interactions among players, makes Diplomacy an challenging environment for advancing AI in strategic decision decision-making. Traditionally, success in Diplomacy has relied on equilibrium search meth- ods generate equilibrium policies (Zhu & Zhao, 2022; Jacob et al., 2022; Wongkamjan et al., 2024; Lu al., 2025). While effective, as illustrated in Figure 1, these methods re- quire large amounts of game data' to train models, limiting their scalability and applicability to other domains.
1. Introduction
Multiplayer games have long been cornerstone of Al re-
search, with classic examples like chess (Silver et al., 2017), Go (Silver et al., 2016; 2017), and poker(Moravčík et al.,
2017; Brown & Sandholm, 2018) with at most thousands of actions per state. In contrast, Diplomacy presents a sig- nificantly more complex challenge, with a combinatorial
In parallel, Large Language Model (LLM)- based agents have gained significant attention in Al research due their powerful general reasoning abilities. These agents have shown strong performance ing personal assistants (Li et al., 2024), robotics (Chai et al., 2025; Chen et al., 2025b) and games (Jin et al., 2024: Guan
'Equal contribution 'School of Artificial Intelligence, Univer- sity of Chinese Academy of Sciences, Beijing. China "State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. Correspondence to: Yuanheng Zhu Syuaanhnn.zhuuiia .n....en>.
variety of domains, includ-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'The data requirements for DORA, DNVI, and Cicero during training are estimated based on GPU usage, as their respective papers not provide exact numbers.

---

DipLLM: Finc-Tuning LLM for Strategic mecision-making in Diplomacy
al., 2024). Typically, LLM-based agents rely on prompt engineering, where the input text is adjusted to enhance per- formance for specific tasks without modifying the model's weights. Although this approach can yield short-term perfor- mance gains, it remains constrained by the inherent limita- tions of the foundational model, often leading to suboptimal decision- making in complex scenarios (Xu et al., 2024). Recent advancements (Zhai et al., 2024; Bai al., 2024) suggest that fine-tuning L.LMs small, domain-specific datasets can significandly improve their performance by op- timizing policy distributions. While this method has proven effective in environments with smaller action spaces, it strug- gles in complex multiplayer games like Diplomacy due to the exponential growth of action combinations and intricate strategic interactions among players. This raises a criti- cal question: Can we fine-tune an LLM-based agent to learn equilibrium policies that outperform traditional methods in Diplomacy?
player mixed cooperative-competiive dynamics, simultane- ous moves, and vast action space ranging from 10 to 1041 The no-press variant of Diplomacy, which prohibits com- munication between players, demands implicit coordination through in-game actions, posing significant challenges for AI in strategic planning. Paquette et al. (2019b) introduced DipNet, the first deep learning-based Diplomacy agent, uti- lizing imitation learning on large-scale human gameplay data. Subsequent studies in no-press Diplomacy build upon similar architectures, incorporating reinforcement learning methods (Anthony et al., 2020; Bakhtin et al., 2021; 2022b). Recent advances predominantly rely on training agents us- ing equilibrium search methods to approximate Nash equi- librium (Gray et al., 2020; Bakhtin et al., 2022a). While effective, these approaches require extensive computational resources to generate large-scale game data. For instance, Cicero utilizes the Coshar-piKL equilibrium search method to train policy and value network, requiring up to 448 GPUs for gameplay rollouts. In contrast, our approach lever- ages the general reasoning capabilities of LLMs, enabling fine-tuning on a relatively small dataset while outperforming domain -specfff models.
To address this challenge, we propose DipLLM, a fine-tuned LLM-based autoregressive factorization agent that learns equilibrium policies for Diplomacy (Figure 1). Our agent leverages an LLM-based autoregressive factorization frame- work to decompose the complex task of assigning actions to all units into sequential sub-tasks, each focusing on sin- gle unit's action. This decomposition significantly reduces the combinatorial complexity of the action space, offering a managcable foundation for fine-tuning. Within this au- toregressive factorization framework, we define a learning objective that approximates the Nash equilibrium and pro- vide theoretical analysis to establish its equivalence and optimality. To align the agent's ' policy with this objective, dataset
LLM--baed Agent. LLM-based agent have gained gener- alized reasoning abilities from vast amounts of human lan- guage data, enabling their application to decision-make making tasks. One line of research focuses prompting techniques (Sahoo et al., 2024) for enhancing the decision decision-making capa- bilities of large foundation models, with notable approaches including React (Yao et al., 2022), AutoGen (Wu et al., 2023), and Reflexion (Shinn al., 2024). These prompt- based techniques have shown success in various scenar- ios, such as personal assistants (Li et al., 2024), robotics (Cheng et al., 2024; Chen et al., 2025a) and games (Gallotta et al., 2024; Xu et al., 2023; Guan et al., 2024). However, prompt-based techniques offer only short-term performance improvements and are inherently constrained by the capabil- ities of the underlying foundation models. Another line of research focuses specific data to enhance their decision-making performance (Pan et al., 2024; Zhai et al., 2024). These fine-tuning ap- proaches have achieved promising results, though primarily in relatively simple decision decision-making environments. Unlike these methods, we fine-tune L.LMs specifically for complex multiplayer games, Diplomacy, enhancing their strategic decision-man making capabilities.
we fine-tune the model using Diplomacy-specn structured in autoregressive form and carefully designed loss function. This fine-tuning process enhances the agent's strategic capabilities, enabling it to outperform a diverse range of opponents, including tfate-of-the-ee art models such as Cicero, in Diplomacy.
whole LLMs with domain-
Our contributions can be summarized as follows: 1) we introduce DIp..LM, an M-baed autoregressive factoriza- tion agent for Diplomacy that simplifies complex decision- making and fine-tuning. 2) we formally define the equi- librium policy within the factorization framework and fine- tune DipLLM to align with this objective using a specially designed loss function and a collected dataset. 3) we demon- strate that our agent outperforms previous domain-specine models, including Cicero and other I.L.-based approaches, this with only 1.5% of
in no-press Diplomacy- the data required by Cicero.
3. Preliminary
3.1. Markoy Games
2. Related Work
We model Diplomacy as a multiplayer Markov game. For- mally, an n-player Markov game A defined as a tuple:
AT in Diplomacy. Diplomacy has become a prominent benchmark for multi-agent games, known for its seven-
A = (S, Ai,..., An, ri, p) where S is the state space, A; is the action space of player (i = n)....).
2

---

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
TextDiplomacy
Text State Prompt 5, Unit 2 Task of
Text State Prompt sg
Unit Task of
Unit Task af
[Your Power]: France
Env: Diplomacy
round, next your first action.
raund, actions you have generated
In this round, the actions
[Game Time and Phase]:
you have generated are [amyy in BUR moves BEL, army moves SPA.... Your next action is that
1901 Fall: Diplomacy
Your action is that army BUR
are (army in BUR to BEL]. Your next action that army MAR
[Board State]:
Your Power Unit:
BAB
Other Power Unit ...
PAR
Areas Without Unit:
BLB
fleet in MAD
Occupied Supply Center...
GAS
Not Supply Center:
[Last Move]:
DipLLM
Your Power Actions
Other Power Actions:
Joint Text Action
amy in BUR moves to DEL
Action To Game
Unit Action ot moves to BEL
Unit Action of moves to SPA
Unit D Action oz
army MAR SPA
BUR BEL, MAR SPA.... F MAO - POR
moves to POR
fleet MAD moves POR
Figure 2. The inference process of our -ased autoregressive factorization agent. DipLLM sequentially decides the action for cach unit by prompting the Text State and unit Task, and finally sends the joint actions to the enviromment for execution.
T' S A1 x....... A, An + R is the reward function for player i, p Sx x A1 x
spaces, constrained by maximum token lengths m and n, respectively (Zhai et al., 2024). The input text ein ym represents the state E S = Vm, while the output text your E ym corresponds to the action E A. An LLM policy, parameterized by é, maps inputs to outputs as T& V + y The probability of generating an output ouur given an input in is denoted by (""""l""")
A. - S is the transition function. The objective of each player is to select policy m.(s): : + AA; that maximizes their expected reward, given the policies of the other players. In each state ev- ery player i simultaneously selects an action ag from their action set A, The actions of all other players, excluding i, are represented as a-;- Players may select a probability distribution over actions, where the probability of action aş is denoted TT (a)
In the context of Diplomacy, the input di encodes the game state, and the output u Fa (v") represents the LLM- generated action, which is parsed into specific moves for the player i's units. By establishing the correspondence between wi in and s, and y and a; the policy probability simplifies to T game states to actions.
3.2. Equilibrium Search
piKL-Hedge (Jacob et al., 2022) is a typical equilibrium search algorithm that iteratively converges to equilib- rium. The core idea behind PiKL-Hedge is to apply KL- divergence constraint to guide the policy search, ensuring that the resulting policies stay close to human strategies. In this approach, each player aims to maximize their expected reward while also remaining "close" to a fixed anchor pol- icy T.. When player selects action a, while other players choose actions a-i. the utility u. ua(ai, (u(a,aa.)) is computed us- ing a pre-trained value function derived from reinforcement learning. The average reward for action o, up to iteration is given by: On each iteration i of piKL-Hedge, the policy xf is set according to
TT (a.(s), providing a direct mapping from
4. DipLLM: A Fine-tuned LLM-based Agent
DipLLM is LLM-based agent fine-tuned to learn equilib- rium policies for Diplomacy. To reduce the complexity of joint units decision-making, we introduce an autoregressive factorization framework. Within this framework, we define learning objective that approximates the Nash equilibrium, and the LLM fine-tuned on a collected dataset align with this objective.
4.1. LLM-based Autoregressive Factorization
(a,1s) exp ( ((.l(()( + (s)
(1)
In Diplomacy, a player can control up to 34 units, each with approximately 26 possible actions resulting in exponen- tial growth in possible action combination. To tackle such combinatorial challenges, many traditional methods employ factorization techniques (Farnoosh et al., 2021; Chebotar et al., 2023; Zhu et al., 2025). Building on this, we design an LLM-based autoregressive factorization framework, which decomposes the task of selecting joint actions into series of smaller, sequential sub-tasks.
Where Q (s, a.) E. (-s) [tg (s,a,, ag a))) The pa- rameter 3 controls the relative influence of the reward func- tion and the anchor policy.
3.3. --seed Decision- Making Policy
We define V as the finite and discrete token vocabulary, with Tm and n representing the input and output text
3

---

DipLLM: Finc-Tuning LLM for Strategic Decision-making in Diplomacy
The policy π [s) in Diplomacy determines the joint action of all D units controlled by player based on the board state s. We decompose this policy into sequence of sub-policies n². where each sub-policy decides the ac- tion of a single unit d. Each sub-policy of considers the board state s and the sequence of actions chosen for previ- ous units, denoted as (-1,,, at..... ot 1). This structure ensures that decisions for each unit depend on both the board state and the context established by earlier decisions. To align this formulation with the next-token prediction framework used in language models, the sequence of pre- viously selected actions {a} ..... a context a 1. Consequently, the probability of selecting at written as n (a at ). ). This factorization reduces the combinatorial action space into manageable sub-action spaces, effectively overcoming the challenges posed by com- plex environments like Diplomacy.
for each individual unit action at conditioned on prior unit actions, is defined as:
Q a, !:d-1 ...) 2 log
Σ
exp {Q, di a
(+1:
ai 2++1
+ B log T (a. 1:d of o(t)))ss))
(3)
Specifically, when d < D, Enttur denotes the summa- tion over all possible combinations of subsequent actions af .a d+2 ......""" When d = D, this term is defined to be 1, since there are no remaining actions to marginalize. The / controls the trade-off between the expected cumu- lative reward of action of and anchor policy, Our defined unit-level state-action value function Q (s, a a2:dd1,,00 aq) cap- tures the trade-off between the expected cumulative reward of selecting action af and the behavior under an anchor pol- icy. Based on this formulation, the policy learning objective for autoregressive factorization is defined as:
d- } is treated as the
Building on this, we can now describe the inference process for the LLM agent within this autoregressive factorization framework. As shown in Figure 2, TextDiplomacy module first converts the raw board state from environment into a text-based representation s. This representation serves as the foundational context for the agent. For each unit, task-specific prompt is generated that includes the actions of previous units, a; !:d-1, as well as the unit's 's identity. Us- ing this combined prompt, the LLM generates the current unit's s action a. such as moves to POR, following the policy of x[(af] |s, a 1. After generating actions for all units, the post-processing module maps these actions to their corre- sponding units, forming the complete joint action of e.g. ...fleet in MAO moves to POR. This final joint action is then formatted according to the game rules and sent to the environment for execution c.g. ...F MAO POR.
A exp ( a, lid-1 af)) Eag af exp { (2 (s, a
-. (a/ s, 1
(4)
lid-1 , {a
To elucidate the properties of the proposed objective in game settings, we provide a theoretical analysis.
Theorem 1 (Objective Equivalence). The joint policy de- rived using the learning objective in autoregressive factor- ization form, I n. d.* (af]s, a; ddd-1), is equivalent to the original policy distribution π..
Theorem 1 demonstrates that the joint policy derived from autoregressive factorization is theoretically equivalent to the original policy. This equivalence offers a unit-based optimization approach, allowing fine-tuned autoregressive factorization agents to effectively approximate the original Nash equilibrium. By breaking down the complex action space, the learning objective ensures that the LLM makes de- cisions within a manageable sub-action space for each unit, while guaranteeing that the resulting joint policy remains equivalent to the original policy.
4.2. Learning Objective in Autoregressive Factorization
LLM-based agents that are not fine-tuned for equilibrium strategies struggle to handle complex strategic interactions in Diplomacy. To guide the fine-tuning process, we first define a learning objective for the equilibrium policy within the autoregressive factorization framework. Inspired by the final iterative policy in pedLe 1)., we rewrite it for clarity and then use this refined version define the equilibrium policy in our autoregressive factorization framework.
Theorem 2 (Optimality of objective in 2p0s games). In two- player zero-sum games, when both players update their poli- cies using the learning objective in autoregressive factoriza- tion over T iterations, their average policies converge to a (maz==1.235.)) (000=::-a.B.)apppooimma approxxmmate Nash equilibrium as T - co. Here, 8; is defined as PPxa log (1/T, (d):D))
T (c!:Pl)) x exp (3log(T.(a.:11)))) + (s, (2)
Theorem 2 establishes the optimality of the proposed learn ing objective in two-player zero-sum games. Extensive research (Bakhtin et al., 2021; 2022b) indicates that that thcoretical insights from these settings can generalize effec- tively to multiplayer games. This supports the applicability of our approach to complex environments like Diplomacy. Full proofs are provided in the Appendix C.
where a. represents the joint action of D units for player i, T; serves as the numan-imitative anchor policy, and Q; represents the reward for the joint action.
To enable autoregressive factorization, we decompose the joint V-vuuee into a sequence of conditional Q-values, defin- ing separate Q-value Q for each unit d. The Vavalue
4

---

DipLLM: Fine-Tuning LLM for Strategic mecision-making in Diplomacy
Data Collection
Fine-tuning
et al., 2019a) and the Diplomacy environment, This initial dataset forms the foundation for subsequent training. Next, we apply the equilibrium search algorithm, piKL-Hedge, us- ing DipNet as a human- like anchor policy тN This process generates joint action Q-values Q,, which represent their expected rewards.
Equilibrium Search
Joint Q-value
Unit Data (or, (e1.01)
piKL-Heddg
Q
Action of to GAL Valargg
Raw Data
Autoregressive Factorization
Action- A BUD SSn
Gradient
Anchor Prob i 70%
A BUD
VIV
TRI
LLM Unit Action
To enable autoregressive policy training, we decompose the joint Q-values into unit-level training samples, denoted as Q Directly computing the Q-value defined in Equation 3 requires applying the log-sum-exp function over the entire action space of a, which is computationally expensive and inefficient. To improve efficiency, we approximate this Q-value by using a tight lower bound based on the sampled values of Q, a!: + 3llg (((: where a::D π (-|s). The derivation of this approximation is provided in the Appendix C. After compute unit Q-value, we convert actions into text format and split them to extract unit-specifie task prompts a; 1:d-1 and corresponding ground truth actions of The unit-specific task prompts of are then combined with the text board state s from TextDiplomacy to serve as the LLM's input. Finally, the processed data is stored as autoregressive factorization transitions in the form(s, (s, a. d.:dd1, at Q+),, where:
Human Model DigNetr
Action
Textuelization & Split
DipLLM +
Raw State s,
Prompt
Unit Task of
Prompt
In mund, onders generated BUD SER).
Viour Prover)
France
(Same Phoset... Statc)--.
Your order that In VIE
(Last Movol
Diplomacy
Raw data
TextDiplomacy
Figure 3. Pipeline for fine-tuning the LLM- autoregressive factorization agent. Raw data is collected by interacting with the environment via DipNet. Q-values Hedge search, and the data is stored in prompt form. The LLM is then fine-tuned using designed loss function.
generated through piKL-
4.3. Fine-tuning for Equilibrium Policy
8: textual board state for player i,
The fine-tuning process, depicted in Figure 3, involves op- timizing loss function using game-specific data structured in an autoregressive factorization format.
a: !:d-1, : a task prompt containing the actions of the previous d-1 units,
at: a textual representation of the ground truth action for the d-th unit,
Loss Function. The objective of fine-tuning is to align the policy of L.LM-based agent with the equilibrium policy defined by the optimization objective in Equation 4. This alignment is achieved by minimizing the KL divergence between the LLM's policy and the target equilibrium policy. Formally, the objective is expressed as:
Q the Q value corresponding to the d-th unit's action
5. Experiments
We conduct comprehensive evaluation of DipLLM across various scenarios to assess its effectiveness. First, we evalu- ate its performance against a pool of baseline opponents, in- cluding the SOTA Cicero. with all agents operating without equilibrium search techniques for a more time-efficient as- sessment. We further compare DipLLM with the enhanced agent, Cicero with equilibrium Search, for a deeper evalua- tion of its strategic decision-making capabilities. Addition- ally, we examine the benefits of fine-tuning by comparing the performance of the fine-tuned LLM agent with that of a domain-spenn -saaific model, highlighting the advantages of our approach. Finally, we conduct ablation studies analyze the contributions of the autoregressive factorization and the fine-tuning process.
min E, (a.aa'd 1NND DKL (aft (-|s, 1:d 1| (. a. (1T,((1),)))) Tip
1
(5)
Through derivation, we find that this objective is equivalent to maximizing:
(n.a (n - log ns (a s. a 1
max 85
exp { (s, a 1:d-1 [aa)]] (6)
This loss function comprises two key components. The term log na (a is, a..'d- !id-1) corresponds to the supervised fine-tuning (SFT) component, aligning the LLM's output actions with those in the collected dataset. The remaining terms act as weights, ensuring that the learned policy ap- proximates the equilibrium policy by incorporating the unit value estimates Q,
5.1. Experimental Setup
Evaluation Method. We evaluate DipLLM by conducting numerous no-press Diplomacy games against baseline mod- els. In each game, one agent controls a single power, while the other six powers are controlled by copies of the opposing
Data Collection. To fine-tune our LLM- based autoregres- sive factorization agent, we collect raw data through interac- tions between the domain speciffc model DipNet (Paquette
5

---

DipLLM: Finc-Tuning LLM for Strategic Decision-making in Diplomacy
Agent
SoS Scorest
Win Ratet
Most SC1
Survived
Defeated
DipLLM (Ours)
22.3% +0.2%
29.3 1.22%
50.3 10.7%
27.4% 10.1% 10.1%
Cicero (Bakhtin al., 2022a)
20.3%
20.5%++0.2%
28. 7..2%
50.1 +..5%%
29. 29.4%::0..2
DNVI (Bakhtin et al., 2021)
6.6%..11%
4.3%
5.0%610.1%
35.8%
59.9%:1..7%
DORA (Bakhtin et al., 2021)
5.7%
4.1%%
3.9%%+00.2%
33.0%%+0.64
62.3%
DipNet (Anthony et al., 2020)
1.8%61..1%
1.1%
1.1%
30.1% 50.4%
68.8% 08.8%%10.7%
Table 1. Performance of different agents in population of various agents. The + indicates one standard error.
SFT Prompt
12.5
0.6
0.3
0.4
0.3
0.4
16.7
16.4
15.7
15.3
(0.7)
(0.1)
(0.1)
(0.1)
(0.1)
(0.1)
15-
14.3
17.6 (1.1)
0.7
0.4
0.3
0.3
SoS Score (%)
(0.13
(0.1)
(0:1)
(0.13
(0.1)
10-
Ours Clcoro DNWI DORA DipNet
83.3
70.1
0.1
4.6
0.9
0.7
(0.3)
(0.1)
(0.1)
(0.1)
(0.1)
(0.1)
Agent
88.7
76.3
11.8
7-1
2.0
1.9
(1.1)
(0.1)
(0.2)
(0.13
(0.1)
(0.1)
89.0
6.4
2.1
(1.3)
(1.3)
(0.3)
(0.1)
(0.1)
(0.1)
2
4
8
Equilibrium Search Iterations
87.9
86.5
24.1
20.6
25.5
13.4
(1.1)
(1.1)
(0.1)
(0.1)
(0.2)
(0.3)
Figure 5. Performance comparison of our agent playing against six Cicero agents, with each Cicero agent incorporating different numbers of equilibrium search iterations. The purple dashed line indicates equal performance, i.e., a win rate of 1/7 v 14.3%.
89.0
88.9
29.4
21.0
25.0
16.7
(1.4)
(1.3)
(0.2)
(0.1)
(0.2)
(0.2)
Prompt
DipNet
DORA 6 Agents
DNVI
Cicero
Ours
any power without SCs considered defeated, while powers still controlling least one SC are considered survived.
Figure 4. SoS scores various agents (y-axis) when competing against six identical copies of another agent (x-axis). The values in parentheses represent one standard error. Note that equal perfor- mance corresponds 1/7 = 14.3%.
5.2. Experimental Results
Baseline Agents We evaluate DipLLM's performance against five open-source speeiic models and two LLM-based decision-making agents employing either prompting supervised fine-tuning (SFT) techniques:
agent, forming 16 competition (Bakhtin et al., 2021). For computational efficiency, agents directly output their actions during the inference phase without employing additional equilibrium search. This approach is practical, as equilib- rium search modules are typically resource- -ntturcc-itteivveeeeee and cime-consuming. Importantly, this approach ensures a fair evaluation, as all models have the potential to incorporate search-based based enhancements in future deployments.
DipNet (Anthony et al., 2020): A 0.3B parameter model trained via imitation learning on a large-scale dataset of human expert demonstrations. It serves as a baseline agent for comparison and is also used to interact with the environment for raw data collection. DORA (Bakhtin et al., 2021): Built on the DipNet architecture, DORA incorporates an equilibrium search module to train both policy and value networks from scratch using reinforcement learning.
Evaluation Metrics.
We evaluate the agents using two key metrics. The first the sum-of-squares scoring (SoS), which is commonly used in previous studies (Bakhtin et al., 2022b;a). In this metric, player i receives a score of
DNVI (Bakhtin et al., 2021): Similar to DORA but with policy and value networks initialized from human behavior cloning (BC) pretraining.
C²
E C3 where C₂ represents
the number of supply centers (SCs) controlled by player The average score for an identical agent is 1/7 x= 14.3% The second metric is based the four possible game outcomes: win, most SCs, survived, and defeated. A power wins by controlling 18 or more of the 34 SCs, ending the game. A power is defeated if it loses all its SCs. If no power wins, the game ends in a draw, with the power controlling the most SCs labeled as having the most SCs. When the game ends,
Cicero (Bakhtin et al., 2022a): The SOTA model with 2.7 billion parameters, trained using large-scale human demonstrations and the equilibrium search technique CoShar-piKL.
Prompt: A pure prompt-based based LLM agent developed with techniques such as social reasoning. reflection and subgoal generation, as described Richelieu (Guan
6

---

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
DipNet
DipLLM (Ours)
28-7
29.0
DipLLM -w/o FT
DipLLM
Equilibrium Policy
0.72
26.0
0.6
228
19.1
20.6
0.47
0.42
16.1
16.9
0.4
0.37
15.
14.3
SoS
0.2
9.9
0.13
0.15
0.09
0.11
0
100
Action
Action
Other Action
Number of Games
A BER KIE, MUN
A BER KIE, A MUN- RUK, KIE
SI1 SIL,F DEN
Figure 6. Performance comparison of fine-tuned DipLLM and Dip- Net (Paquette et al., 2019a), both trained using different numbers of games. Each model is evaluated by playing against six original DipNet agents without fine-tuning
Figure 7. Comparison of action distributions between DipI.LM, DipLLM (w/o fine- tunint, FT), and the equilibrium policy gener- ated by piKL-Hedge.
et al., 2024).
combined with equilibrium search, we conducted a prelim- inary experiment under limited computational resources. The results included in Appendix E.2. In addition to its strong performance, DipL.LM demonstrates significantly improved inference efficiency. It requires only 10-20% of Cicero's 's inference time per turn, making it a more practical choice in real-time decision decision-making making scenarios.
SFT: An LLM agent trained with SFT on the same data, without using autoregressive factorization, generating all unit actions in single step.
Evaluation in Baseline Population. To accurately eval- uate the equilibrium policy of our agent, we construct a population comprising DipL.LM and the diverse baseline opponents. Games are then conducted in a 1v6 competition format, with agents randomly selected from this population. Figure 4 illustrates the SoS scores from pairwise agent com- petitions. The results reveal that both the prompt-based and SFT agents underperform. To maintain the competitiveness of the opponent population, we exclude these two models and provide a more detailed evaluation of the performance across the remaining models. Table presents the perfor- mance results for the agents in this population. DipLLM outperforms all other baselines across every metric, with 2.2% advantage in the SoS score and a 1.8% higher win rate compared to the second- model. This demonstrates its strong decision-making capabilities. Additionally, while DipLLM's survival rate is almost identical to Cicero's ss its significantly higher win rate and reduced fallure rate suggest more aggressive and effective decision- making approach.
Comparison of Fine-tuning DipLLM to Domain Model DipNet. We compare the performance of DipLLM, large pre- trainedddd language model, with DipNet, domain sppeciff model, after fine-tuning on the same dataset divided into subsets of varying sizes. DipNet is fine-tuned on the col- lected raw data following the approach in (Bakhtin et al., 2021), while DipLLM is fine-tuned using our proposed loss function on text data. Figure 6 highlights the consistent performance improvements of both models as the dataset size increased. Initially, DipNet, task-specific model pre- trained on domain-relevan data, outperforms DipLLM due to its specialized architecture and prior domain knowledge. In contrast, DipLLM,, which lacks domain-specific pre- training, starts with lower performance. However, DipLLM demonstrates remarkable data efficiency during fine-tuning. With only 100 games of fine-tuning data, DipLLM not only matches but surpasses DipNet's 's performance. As the dataset size grows to 500 games, DipLLM achieves a significant lead, outperforming DipNet by 6.7%. These results high- light DipL.LM's ability to leverage its general reasoning and decision- makinn capabilities, achieving superior perfor- mance with relatively small fine-tuning data. This compari- son underscores the effectiveness of our approach and the advantages of integrating large pre-trained language models into domain tasks.
Playing against Cicero including Equilibrium Search Module. To further evaluate DipLLM's decision-making capabilities, we directly pit it against Cicero, which utilizes an equilibrium search module with varying rollout counts to enhance its strategic depth. Although DipLLM is also compatible with the equilibrium search module, this analy- sis focuses on its end-to-end reasoning performance without relying on additional search steps. Figure 5 shows that, even as the number of rollouts increases and strengthens Cicero's 's cquilibrium search, DipLLM continues to outperform it, maintaining a win rate above the average (14.3%). This demonstrates DipLLM's stable and robust decision makiin performance. To further explore DipL.LM's potential when
A Ablation Study. This ablation study investigates the critical roles of autoregressive factorization (AF) and fine-tuning in optimizing DipLLM's performance. The Without AF agent uses an L.M to select actions for all units in a single step
7

---

DipLLM: Finc-Tuning LLM for Strategic Decision-makinn in Diplomacy
Spring 1909
Fall 1909
Spring 1910
Fall 1910
England (DipLLM): A BUR PARR A BRE, F IRI MAO, F (RI MAO, FLNN
England (DipLLM):
England (DipLLM);
England (DipLLM)): BUR MAR A PAR F HOO NTH MAO SPA/SC POR, F SPAVSC- POR
BEL BUR A PIC S BEL BUR,
A BUR MAR, APC PAR.
ENG BRE, KIE DEN
ENG MMG, HOL NTH NTH SPA/SC
SPA/SC.
France (Cicero):
France (Cicero):
France (Cicero):
France (Cicero):
BRE PIC. MAO - ENG, A GAS S MAR BUR, A PAR - PIC, MAR BUR
F BRE BUR PIC. BUR PIC, G BRE, PAR S BRE MAD
MAD POR.
GAS SPA, A PAR BRE, POR S GAS- SPA, GAS
GAS SSAS, PAR BUR,
Figure An example of DipLLM--ontrolledd England strategically defeating Cicero-controlled France.
Fine- tune
SoS Score
Win
Survived Defeated
equilibrium strategy in the opening phase primarily involves two actions: moving the MUN army either SIL (Action 1) RUH (Action 2). Before fine-tuning, the agent assigns low probability to these critical actions, favoring suboptimal alternatives. After fine-tuning, the agent's action distribu- tion shifts to more closely align with the equilibrium policy, with significantly higher probabilities for Actions and 2. This shift demonstrates that fine-tuning effectively guides the agent toward equilibrium--ike strategies.
AF
X
X
0.2%
0.0%
4.3%
95.7%
X
V
0.8%
0.0%
19.2%
80.8%
V
9.9%
6.7%
40.0%
53.3%
/
V
29.4%
25.2%
45.9 %
29.0% %
Table Ablation study on the effects of autoregressive factoriza- tion and fine-tuning when playing against six DipNet agents.
A Case Study: Feint to Attack, Strike Where Least Expected. DipLLM demonstrates exceptional strategic decision- decision-making, using misdirection to exploit opponent vulnerabilities. As shown Figure 8, in Spring 1909, Eng- land (DipLLM) faced a tense standoff with France (Cicero) the western front while under pressure from Russia and Germany. To break the deadlock, DipLLM staged diver- sion with armies in BUR and PIC to tie down France, while fleets in IRI and ENG launched a surprise attack on the poorly defended MAO region. This feint distracted French forces in BRE and GAS, forcing them retreat POR. England capitalized by capturing SPA by Fall 1910, bypass- ing French forces in POR and effectively encircling France. A few turns later, England conquered all French territories, securing decisive victory. This case highlights DipLLM's effective use of deception and multi-front strategy to turn a vulnerable position into a comprehensive triumph.
(one-step decision- massiin. The Only Fine-tune method fine-tunes the one-step decision-making LLM using joint ac- tion datasets. As shown in Table 2, fine-tuning alone yields minimal gains, the one-step decision-maker framework still struggles with the vast action space. Incorporating AF mitigates this challenge by decomposing the combinato- rial action space into smaller sub-action spaces, resulting in improved performance. However, since the AF-based agent not fine-tuned for an equilibrium strategy, its per- formance remains suboptimal. When AF is combined with fine-tuning within the reduced sub-action spaces, perfor- mance improves significantly. This is because fine-tuning within smaller, well-defined sub-action spaces enables the model to adapt more effectively to the task. These results highlight the critical role of both AF and fine-tuning in ad- dressing complex action spaces and optimizing LLM -baeed agents for strategic games.
6. Conclusions
To further assess whether fine-tune fine-tuning aligns the agent with the equilibrium policy, we analyze the changes in the opening-turn action distributions for England, as controlled by DipLLM. Specifically. we compare the agent's 's action distributions with the equilibrium policy derived from the peKl.-Hedge algorithm, As shown in Figure 7, England's
In this work, propose DipLLM, fine-tuned LLM- based agent that learns equilibrium policies for Diplomacy. DipLLM addresses the complexity of decision-making by leveraging autoregressive factorization to decompose tasks
8

---

DipLLM: Fine-Tuning LLM for Strategic mecision-making in Diplomacy
into smaller, sequential steps. Building on this foundation, we define theoretically grounded learning objective ap- proximate the Nash equilibrium and fine-tune DipLLM to align with this objective. Our agent surpasses SOTA Ci- cero while using only 1.5% of Cicero's training data. These results highlight the potential of LLM-based agents for ad- dressing complex decision decision-making challenges and pave the way for their broader applications in the future.
Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C.,
Fried, D., Goff, A., Gray, J., Hu, H., Jacob, A., Komeili, M., Konath, K., Kwon, M., Lerer, A., Lewis, M., Miller, A., Mitts, S., Renduchintala, A., Roller, S., Rowe, D., Shi, W., Spisak, J., Wei, A., Wu, D., Zhang, H., and Zijlstra, M. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, Nov 2022a.
DipLLM still relies on ternal data generated by other models through equilibrium search, generating data via self-play with the agent offers a more scalable solution for broader application scenarios. We provide preliminary evidence that combining LLM- agents with real-time equilibrium search techniques can sig- nificantly enhance decision-make making performance. However, due to computational constraints, were unable to conduct more extensive investigation of this approach.
Limitations. While
Bakhtin, A., Wu, D., Lerer, A., Gray, J., Jacob, A., Farina, G., Miller, A., and Brown, N. Mastering the game of no-press diplomacy via human-regularizee reinforcement learning and planning. In The Thirteenth International Conference on Learning Representations, 2022b.
Brown, N. and Sandholm, T. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374): 41--4444,4 2018.
Chai, J., Li, S., Fu, Y., Zhao, D., and Zhu, Y. Empower- ing Ilm agents with zero-shot optimal decision- making through q-learning. In The Thirteenth International Con- ference on Learning Representations, 2025.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 62293541 and Grant 62136008, in part by Beijing Natural Science Foun- dation under Grant 4232056, and in part by Beijing Nova Program under Grant 20240484514, in part by the Strategic Priority Research Program of Chinese Academy of Sciences under Grant No. XDA27030100. We would also like to thank Zijie Zhao for his insightful discussions and valuable suggestions during the early stage of this work.
Chebotar, Y.. Vuong, Q., Hausman, K., Xia, F. Lu, Y., Irpan, A., Kumar, A., Yu, T., Herzog, A., Pertsch, K., et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q- funccions.. In Conference on Robot Learning, pp. 3909-3928, PMLR, 2023.
Chen, Y., Cui, W., Chen, Y., Tan, M., Zhang, X., Liu, J., Li, H., Zhao, D., and Wang, H. Robogpt: an long-term decision -aainng embodied agent for instruc- tion following tasks. IEEE Transactions on Cognitive and Developmental Systems, 2025a.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, Y., Tian, S., Liu, S., Zhou, Y., Li, H., and Zhao, D. Conrft: reinforced fine-tuning method for vla mod- els via consistency policy. In Proceedings of Robotics: Science and Systems (RSS), 2025b.
References
Cheng. G., Zhang, C., Cai, W., Zhao, L., Sun, C., and
Bian, J. Empowering large language models on robotic manipulation with affordance prompting. arXiv preprint arXiv:2404. / 1027, 2024.
Anthony, T., Eccles, T., Tacchetti, A., Kramár, J., Gemp, I., Hudson, T., Porcel, N., Lanctot, M., Pérolat, J., Everett, R., al. Learning to play no-press diplomacy with best response policy iteration. Advances in Neural Information Processing Systems, 33: 13:17787-18003 2020.
Farnoosh, A., Azari, B., and Ostadabbas, S. Deep switching auto-regressive factorization: Application to time series forecasting, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7394-7403, 2021.
Bai, H., Zhou, Y., Pan, J., Cemri, M., Suhr, A., Levine, S., and Kumar, A. Digirl: Training in-the-wild device- control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37: 12461-12495, 2024.
Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J.. and Yannakakis, G. N. Large language mod- els and games: A survey and roadmap. arXiv preprint arXiv:2402. 18659, 2024.
Bakhtin, A., Wu, D., Lerer, A., and Brown, N. No-press diplomacy from scratch. Advances in Neural Information Processing Systems,
Gray, J., Lerer, A., Bakhtin, A., and Brown, N. Human- level performance in no-press diplomacy via equilibrium search. arXiv preprint arXiv:2010.02923, 2020.
2021.

---

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
Guan, Z., Kong, X., Zhong, F,, and Wang. Y. Richelieu: Self- evolving llm-based agents for ai diplomacy. Advances in Neural Information Processing Systems, 37:123471- 123497,
Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36: 53728-53741, 2023.
Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprini arXiv:2402.0792. 2024.
Hu, E. J., Shen, Y. Wallis, P., Allen-Zhet Zhu, Z., Li, Y., Wang, S., Wang. I.., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL https:// openreview net/ weaerrummiidneeeeeeeeee
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms.
Jacob, A. P., Wu, D. J., Farina, G., Lerer, A., Hu, H., Bakhtin, A., Andreas, J., and Brown, N. Modeling strong and human-like gameplay with kl-regularized search. In International Conference on Machine Learning, pp. 9695- 9728. PMLR, 2022.
arXiv preprint arXiv:1707.06344 2017.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Jin,& X., Wang. Z., Du, Y., Fang, M., Zhang, H., and Wang, J. Learning to discuss strategically: A case study on one night ultimate werewolf. Advances in Neural Information Processing Systems, 37:77060-77097, 2024.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershel V., Lanctot, M., et al. Mastering the game of with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Li, Y., Wen, H., Wang, W., Li, X., Yuan, Y., Liu, G., Liu, J., Xu, W., Wang. X., Sun, Y., et al. Personal Ilm agents: Insights and survey about the capability, efficiency and security, arXiv preprint arXiv:2401.05459 2024.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Lu, R., Zhu, Y., and Zhao, D. Divergence-regullriized dis- counted aggregation: Equilibrium finding in multiplayer partially observable stochastic games. In The Thirteenth International Conference Learning Representations, 2025.
Wongkamjan, W., Gu, F., Wang. Y., Hermjakob, U., May, J., Stewart, M., Kummerfeld, J. K., Peskoff, D., and Boyd-Graber, J. L.. More victories, less coopera- tion: Assessing cicero's 's diplomacy play. arXiv preprint urXiv:2406.044 04643, 2024.
Moravčík, M., Schmid, M., Burch, N., Lisy, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowl- ing, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337)50085533, 2017.
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.0811 2023.
Pan, J.. Zhang, Y., Tomlin, N., Zhou, Y., Levine, S., and Suhr, A. Autonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.064 06474, 2024.
Xu, Z., Yu, C., Fang, F., Wang, Y., and Wu, Y. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310. 18940, 2023.
Paquette, P., Lu, Y., BOCCO, S. S., Smith, M., O.-G., S., Kummerfeld, J. K., Pineau, J., Singh, S., and Courville, A. C. No-press diplomacy: Modeling multi-agent game- play. In Advances in Neural Information Processing Systems, volume 32, 2019a.
Xu, Z., Yu, C., Fang, F., Wang. Y., and Wu, Y. Language agents with reinforcement learning for strategic play in the werewolf game. In Proceedings of the 41st Interna- tional Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 55434- 55464. PMLR, 21-27 Jul 2024.
Paquette, P., Lu, Y., Bocco, S. S., Smith, M., O-G, S., Kum- merfeld, J. K., Pineau, J., Singh, S., and Courville, A. C. No-press diplomacy: Modeling multi-agent gameplay. Advances in Neural Information Processing Systems, 32, 2019b.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03622 2022.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. mon, S., and Finn, C. Direct preference optimization:
10

---

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
Zhai, S., Bai, H., Lin, Z., Pan, J., Tong, P., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., al. Fine-tuning large vision-language models decision-making agents via reinforcement learning. Advances in neural information processing systems, 37:1 10935-1 10971, 2024.
Zhu, Y. and Zhao, D. Online minimax q network learning for two-player zero-sum markov games. IEEE Transactions on Neural Networks and Learning Systems, 33(3):1228-
1241, 2022. doi: 10.1
Zhu, Y., Huang, S., Zuo, B., Zhao, D., and Sun, C. Multi- task multi-agent reinforcement learning with task-entity transformers and value decomposition training. IEEE Transactions on Automation Science and Engineering, 22:9164-99 2025. doi: 10. 10500409.

---

DipLLM: Fine-Tuning LLM for Strategic mecision-making in Diplomacy
A. Description of Diplomacy
Diplomacy is a strategic board game simulating early 20th-century European geopolitics, where seven powers (Austria, England, France, Germany, Italy, Russia, and Turkey) compete control 34 supply centers distributed across 75 provinces, including land and maritime territories. Gameplay progresses through annual cycles starting in 1901, each comprising five phases: Spring Movement, Spring Retreat, Fall Movement, Fall Retreat, and Winter Adjustment.
Movement Phases. During movement phases, players issue tactical commands to military units. A hold order maintains a unit's defensive position (default if no order is given). A move order directs a unit to attack adjacent provinces- armies tra- verse land or coastal regions, while fleets navigate coastal or aquatic territories. Support orders enhance allied attack/defense capabilities but require the supporting unit to legally access the target province (e.g., Marseille supporting Paris's advance to Burgundy); such support fails if the supporter is attacked. Convoy orders enable armies to cross water via coordinated fleet movements, requiring uninterrupted convoy paths and synchronized orders.
Displaced units from contested provinces must retreat disband. Valid retreats target adjacent unoccupied provinces not involved in prior conflicts. Disbanding occurs automatically if retreat destinations are unavailable, conflicting, or if orders are omitted.
Retreat Phases.
Adjustment Phases, Annual adjustments align unit counts with controlled supply centers. Powers exceeding their supply center count disband surplus units, while those with excess centers build new units in unoccupied original territories (e.g.. Germany builds in Berlin if controlled). Players may voluntarily waive build rights.
Tacit Communication in No-Press Diplomacy. Despite explicit communication restrictions, players convey intentions implicitly through tactical orders. Offensive unit positioning signals aggression, support orders imply alliances, and convoy proposals suggest cooperation. Even invalid orders-such as Russia supporting an impossible English attack from Paris to London-can strategically influence opponents by signaling priorities. This emergent negotiation layer enriches gameplay through action-based inference rather than verbal coordination.
The game concludes when single power secures majority control of supply centers, demanding strategic foresight, adaptive coordination, and precise interpretation of opponents' tactical signals.
B. Full Prompts for DipLLM
We utilize the Llama 3 8B model as the backbone of our framework. The required training data, represented in the form 1:d-1 ag Q+), is structured and stored as part of the prompt.
(s, a
B.1. System Prompt
The system prompt used in our method is listed below.
"role": "system",
"content": "You are an expert in the no-press Diplomacy game environment. As one of seven powers, your task is to use your army and fleet to control the supply centers on the board. You are playing (Your Power] and observing [Game Time and Phase], [Board State], and [Last Moves] below. the [Board State], each power will sequentially display the locations of its army and fleet. Remember, unless specified otherwise, will omit the default attributes for areas, which include the coast, neither supply center nor home center, troops dislodged, and not occupied by anyone."
B.2. User Prompt
The user prompt, which includes the components and at !:d-1 outlined below as utilized our methodology.
12

---

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
"role": "user",
"content":
### textual board state
[Game Time and Phase]:
1905 Spring: Diplomacy
[Board State]:
Your Power Unit:
Turkey's s army: Bulgaria (including Eastern and South Coast, Turkey's supply center)
Turkey's fleet: Aegean Sea (water), Black Sea (water), lonian Sea (water)
Turkey's center without units: Ankara, Constantinople. Smyrna
Other Power Unit:
Austria's army: Budapest (land, Austria's 's supply center), Greece (Austria's supply center), Serbia (land, Austria's supply center), Tyrol (land), Venice (Austria's supply center)
Austria's s fleet: Trieste (Austria's supply center)
Austria's i center without units: Vienna
England's army: Denmark (England's supply center)
England's fleet: Norway (England's 's supply center)
England's center without units: Sweden
France's army: Burgundy (land), Gascony, Marseilles (France's supply center), Picardy, York
France's fleet: English Channel (water), Heligoland Bight (water), London (France's supply center)
France's center without units: Brest, Edinburgh, Liverpool, Paris, Portugal, Spain
Germany's s army: Belgium (Germany's supply center), Holland (Germany's supply center), Ruhr (land)
Germany's s fleet: Kiel (Germany's supply center), North Sea (water)
Germany's center without units: Berlin, Munich
Italy's ' army: Piedmont, Rome (Italy's supply center)
Italy's 's fleet: Tunisia (Italy's supply center)
Italy's 's center without units: Naples
Russia's army: Moscow (land, Russia's supply center), Rumania (Russia's supply center), Warsaw (land, Russia's 'ss supply center)
Russia's fleet: Finland, Sevastopol (Russia's supply center)
Russia's center without units: St. Petersburg
Areas Without Unit:
unoceupied supply center: None
occupied supply center: Ankara (Turkey's), s), Berlin (Germany's) s), Brest (France's), Constantinople (Turkey's) Edinburgh (France's), s), Liverpool (France's), s). Munich (land, Germany's 's), Naples (Italy's) s), Paris (land, France's 's), Portugal (France's s), Smyrna (Turkey's), 's) Spain (including North and South Coast, France's), St. Petersburg (including North and South Coast, Russia's), s)) Sweden (England's), Vienna (land, Austria's)
not supply center: Adriatic Sea (water), Albania, Apulia, Armenia, Baltic Sea (water), Barents Sea (water), Bohemia (land), Bothnia (water), Clyde, Eastern Mediterranean (water), Galicia (land), Irish Sea (water), Livonia, Lyon (water), Mid Atlantic Ocean (water), North Aftica, North Atlantic Ocean (water), Norwegian Sea (water), Prussia, Silesia (land), Skagerrak (water), Syria, Tuscany, Tyrrhenian Sea (water), Ukraine (land), Wales, Western Mediterranean (water)
13

---

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
Last Movel:
Your Power Order:
Turkey: army in Bulgaria supports fleet in Ionian Sea move Greece, fleet in Aegean Sea moves Smyrna, fleet in Black Sea supports army in Bulgaria, fleet in lonian Sea moves to Greece
Other Power Order:
Austria: army in Galicia moves to Budapest, army in Greece supports army in Bulgaria, army in Serbia supports army in Greece, army in Trieste moves to Venice, army in Vienna moves to Tyrol
England: army in Denmark moves to Sweden, fleet Norway moves to St. Petersburg's North Coast
France: army Burgundy holds, anmy in Gascony supports army Burgundy, army in Marseilles supports army in Burgundy, army in Paris moves Picardy, army in York supports fleet in English Channel move to London, fleet in Brest moves to English Channel, fleet in English Channel moves London, fleet in North Sea moves to Heligoland Bight
Germany: army in Belgium supports army in Munich move to Burgundy, army in Holland supports army in Belgium, army in Munich moves to Burgundy, army in Ruhr supports army in Belgium, fleet in Kiel moves to Denmark, fleet London holds
Italy: army in Piedmont moves to Marseilles, army in Venice moves to Rome, fleet in Eastern Mediterranean moves Smymna, fleet in Tyrrhenian Sea moves to Tunisia
Russia: army in Moscow moves St. Petersburg, army in Rumania supports army in Bulgaria, army in Ukraine moves to Warsaw, fleet in Finland moves to Sweden, fleet in Sevastopol supports army in Rumania
### task prompt a: q::d1
In this round, the orders you have previously generated are [army in Bulgaria supports fleet in Ionian Sea move to Greece, fleet in Aegean Sea supports fleet in Ionian Sea move to Greece, fleet in Black Sea supports army in Bulgaria]. The candidate orders for fleet lonian Sea are [moves to Adriatic Sea, moves to Greece, moves to Naples, moves to Tunisia, supports army in Bulgaria move to Greece, supports fleet Aegean Sea move to Greece]. The best order from candidate orders is that fleet in Ionian Sea"
B.3. Other Prompts
The remaining prompts, which incorporate the ground truth data at and Q are provided below.
### Ground truth action a
"role": "assistant",
"content": "moves to Greece",
### Unit Q-value Q "role": "value", "content": "1.28",
14

---

DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
C. Theoretical Analysis and Proofs
First, we recall definitions of factored Q-value and corrsponding policy.
(s, at .aa) a log Σ exp (Q,(s, a3" , a llggg (:((a"" ; ((t1:D(ss) })}
exp (Qi ai 1:d-1 ao4)
* 1) -
Eot exp {Or (s.0g 1:d-1 a0))
Theorem (Objective Equivalence). The joint policy derived using the learning objective in autoregressive factorization form, IIa d.* "i (Q, s,a, "i is equivalent to the original policy distribution π.
factororsii
Proof. We establish equivalence through direct algebraic manipulation. Recalling the original policy definition in piKL. Hedge:
(a; 1:D (s) exp (s))
[s)) (s,
exp {33 (o) (s)) + (, as,
Enyo exp { logg(7 ( (a!:D) 8)) + a
The factorization equivalence follows from:
D
D
exp (Q, a, 1i--1,1000 {a²)
II d=1
d. mi
(Qt
s.a 1:d ) = II
exp (s, a 1id-1 .a9)
D exp log Σ 2 do exp (Q, a lid a ( + 3log (a BlogTi(a !id a d+1:D {)}}
exp log E. LID exp (Q, (s, l:d-1 a d d+1:D) + $ log (a (q. lid-1 a a. d+I:D $)}
D Σ o dID exp (Q, 1:d a. (+1:D) 3 log T (a 1id a.
d+1:D $)}
II
EneD exp (Q, - a DD 3 log T. (a (a) aid $)}
d=1
exp (0, a L.D) B log (a. 8)}
Σ EnD exp (Q, a 1:D) + 3 To (a) {))
= n (a)
Theorem 2 (Optimality of objective in 2p0s games). In two-player zero-sum games, when both players update their policies using the learning objective in autoregressive factorization over iterations, their average policies converge to (..0:=a-.2....)pppoiiiiee. Nash equilibrium as + cc. Here, 8; is defined as max, log (1/Ts (a (a):)).
We extend the foundational result of Jacob al. (2022) with the following advancements:
Corollary 1 (Jacob et al., 2022). For piKL-Hedge in two-player zero-sum games with regularized utilities U:(TT) Ui (T, a (T(T,, a))) 3,D Ti), when both players update their policies using over T iterations, their average policies converge to Nash equilibrium:
max (21 (UT Ui U1( 72)) TE((AA
Building on this, we analyze the autoregressive factorization N, = I2 म. (-|s,, a d.dd1)). Let (T = denote any limit point of average policies. Theorem 1 ensures policy equivalence at each iteration, yielding:
15

---

DipLLM: Fine-Tuning LLM for Strategic mecision-making in Diplomacy
[U1(T]] D. 222) - ( T22}}
max *((EA(
"
{ui D.a, _,,2 x1 DKLLL(T) D.* T1) uz +2 + x1 DKL (#P || [1)}}
max miEA(Ai)
[T| D.+ _22 in ( x - DKL (TP. T1)
max EA((A))
{ui
{ D.* - u1 (T + - DKL
II a * (1 (1(1)
max Ed(AA)
max x;E((Ar)
{ U1
(TI
Di+ 2
1/ (#t =
D II
D II d=1
-81
Σ
पं
(aflat 1:) log
मं (aflaf=--1)
a EAP
ld=1
D
-81 Σ
d.a (a log (I/Ti(a):)))
11
aID EAP
1d=1
a a (aila)---)) log (1/ (1/Ti(E):)))
2 max *1 (AAi)
Di+ =D) - u1 2) - 51
Σ
af EAP
Ld=1
(1I(TT D.+ .12)
U1 ( 2
R1 at.0~111 ~ x, (aflat""-) log (I/TI(a))))
max "f" -(A(A)
max Pi (AAA)A
{UI(TT
D.+ =1)
,72
- U1 |
}
A1 max log (I/T1 (aliP) a dP"~AP
1V
max (EA(A)))
(WI(T1 [T] .22
=D - 0n
D #2
The first inequality holds because the KL divergence is nonnegative by definition. The second incquality follows from the fact that the negative entropy function is nonpositive over the probability simplex. Finally, the third inequality is direct consequence of the definition of 81.
An analogous argument applies to Player 2, yielding the corresponding result.
{ 22 D.)) - uz # A262
max (AA(A))
Thus, the exploitability of &p is bounded by d1 and similarly, the exploitability of π is bounded by X202. This directly establishes the desired result.
Lower Bound of Q-value. Assuming we have sequentially computed the Q-values for ground truth action { a ,..., a, we can derive lower bound using the log-sum-exp inequality follows:
,..,,...,.
(1 1:d-1,+ , a, day a,
log
exp (s, L:d. adt1DD + B log (( 1:d.* ,
adt1:D (s)
d+1D
d+I:Dit) + Blog T (a (a ,a (+1:Di*|s) max Q a ltd,e , ,a;
ldd.
a 4+1.0
(s, a, 1:D.a) + 3 log (a, (s))
We first apply the log-sum-exp inequality to derive a tight lower bound for the decomposed Q-value Q For the ground truth joint action of whiich is obtained via search to maximize the objective Q; + B log T; this bound is achieved exactly. This implies that for any joint action I.D the quantity Q,(s, a.!P) + 3 log T I:D s) serves as a valid and tight lower bound for (s a. a!" Leveraging this bound eliminates the need to compute the full log-sum-exp over the sub-action
space, thereby significantly reducing computational complexity.
16

---

DipLLM: Finc-Tuning LLM for Strategic mecision-making in Diplomacy
C.I. Loss Function Derivation.
The training objective minimizes KL divergence between the defined learning objective and LLM's policy:
arg min B (s.at ()ND INDD DKL (x (-|s, a l:d- 1 (-1s,,( 1
exp (0) 1:d-1 [.ad)
To TT(-|s, a. (a(11s,a!uu abt
arg min
E [ss.a
tid 1nND DNND
DKL En exp 10 (s a agg)
exp 10 (s, a. , ,
Tumi (at1s, (a a
arg min
E (s,a 1o-
log
afEA Eap exp 10
To (a s a. 1
exp { (Q/ a. -- ,a
arg max
E (s.a hd L
ofEAA Eat exp (Qr a a
log TΦ (a+ls at L:dt
= arg max
E (e,00,1d DoD
Σ
log (a (1--1) exp (0 (s, af 1: ,a,)
We
LoNEA!
(e,ogid- (aa llggo Ts (dlls,afi-1) exp (s, - ,a]]]]
arg max
D. Implementation details
D.I. pedle-Hedge in Data Collection
During collecting data, We compute action-specific Q-values using the piKL-Hedge algorithm. The hyperparameters. primarily adopted from the original piKL-Hedge implementation (Jacob et al., 2022), are summarized Table 3.
Hyperparameter
Value
Search iterations
256
Number of candidate actions
50
Trade-off factor 3
0.1
Max candidate actions per unit
Nash explore (e)
0.1
Nash explore, S1901M
0.1
Nash explore, F190IM
0.1
Table 3. Hyperparameters used for piKL-Hedge during data collection.
D.2. Fine-tuning Details
We provide the hyperparameters used for training DipLLM in Table 4. Our model is built on the LLaMA 3 8B architecture as the backbone. During training, employ the Low-Rank Adaptation (LoRA) method (Hu et al., 2022) to update the parameters of the entire LLM.
E. Additional Experiments
E. Performance of DipLLM Against Leading Large Language Models
We selected OpenAI-o3-moa mini and DeepSeek-R. RI as opponents for further comparison with DipLLM. Our results demonstrate that even against models with strong reasoning capabilities, DipL.MM exhibits remarkable superiority.
17

---

DipLLM: Fine-Tuning LLM for Strategic mecksion-making in Diplomacy
Hyperparameter
Value
Optimizer
AdamW
LoRA
32
Dropout Prob
0.05
Batch Size
4
Learning Rate Schedule
Linear
learning_rate
2e-4
Epoch
5
Adaptation
16
Max Seq. Len.
2048
Table 4.
The hyperparameters for fine-tuning DipLLM.
Agent
SoS Scores
Win Rate
Most SC1
Survived
Defeated
DipLLM (Ours)
60.7 1..5%
69.7% +0.7%
68.2 10.6%
27.1 % to.s
3.2 3.2% 10.1%
OpenAl-o3-mini
16.3%11.39
13.9%
11.7%:2..%
51.9%11.5%
34.2%1122%
Deepseek-RI
14.6%
10.8%++0.1%
8.5%
49.3%+40.%
39.9%++0.7%
GPT-4o
3.7%+4.11%
1.8%:+1.2%
2.5%++.119
40.2%:4.3%
57.9%+40.6%
Table 5. Performance of different agents in a population including DipL.LM and top large language models.
E.2. Enhancing DipLLM with Online Search
To further evaluate DipLLM's potential, we conducted additional experiments during the rebuttal period by integrating lightweight search mechanism in 1v6 settings against DipNet. The results show clear performance gain, indicating that DipLLM can benefit from search when resources permit. These findings highlight the model's 's strong decision-making ability and suggest that incorporating online reasoning could further enhance its capabilities.
DipLLM rollouts
SoS Scorest
Win Rate
Most SC1
Survived
Defeated
16. 16.7%1..5
13. %.1.22%
17.5%.12%
47. 1.10.
38.3%1124
18. (8.1% 12.2%
14.7%..0.%
19.3%.102%
47.8% 40.4%
37.5%.102%
n 10
20.2%
15.6 +0.2%
21.6 1..2%
47.4%++0.4
37.0 10.1%
Table 6. Results of DipL.LM with different numbers of equilibrium search rollouts, playing against Cicero.
E.3. Preference Learning in Diplomacy
We explored the use of preference learning by labeling data based on outcome-based reward magnitudes and applying Direct Preference Optimization (DPO) (Rafailov et al., 2023). However, this approach yielded unsatisfactory results. A key difficulty lies in the ambiguous nature of preferences multi multi-agent strategic eerriggs--speeaallly when dealing with joint actions or long-term planning. This ambiguity leads to noisy annotations and poor learning signals, ultimately limiting optimization performance. We also adopt a reward-based learning framework and compare it standard policy optimization methods such as Proximal Policy Optimization (PPO) (Schulman et al., 2017). Experimental results show that our method outperforms PPO, demonstrating the effectiveness of leveraging structured game rewards over noisy preference annotations.
Method
SoS Scorest
Win Rate
Most SC1
Survived
Defeated.
Ours
29.3% % +.20%
25.2%+t0.2%
29.3% +-0.%%
45.8% +0.3%
29.0 % +0.1%
PPO
23.4%.10.2%
20.2%.10.5
23.4%..0.2
10.4%
34.4%.1022 38.2%+0.3%
DPO
1.2%++.11
0.7%++..1%
1.3%
61. 1%.1%%47.
Table 7. Performance of L.I agents fine-tuned using different methods against six DipNet agents.
18


Reviews:
[{"review_id": "LabzUZ41zr", "reviewer": "Reviewer_mCms", "summary": "This paper introduces DipLLM, a fine-tuned Large Language Model (LLM) aimed at learning equilibrium policies for the no-press variant of the game of Diplomacy. The authors propose an autoregressive factorization framework to break down multi-unit action selection into smaller, sequential decisions, thereby mitigating the combinatorial explosion in action space. They then define a learning objective akin to the final policy in piKL-Hedge, showing its theoretical equivalence and approximate optimality for two-player zero-sum settings. Leveraging a small subset of Diplomacy gameplay data (1.5% of Cicero’s dataset), DipLLM reportedly achieves superior performance to the state-of-the-art Cicero model, all while requiring significantly fewer computational and data resources.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Detailed tables are available at https://sites.google.com/view/dipllm.\n\n**Q1. How crucial is base LLM knowledge for Diplomacy? Would a smaller, specialized model with the same data still underperform?**\n\nA1. Our experiments show that **general knowledge in base LLMs is crucial** for success in Diplomacy, and specialized smaller models consistently underperform.\n\nFirst, comparative experiments with Llama models (Appendix E.1, Table 5) show stronger bases (Llama-3-8B) outperform weaker ones (Llama-2-7B) when fine-tuned, highlighting the value of broader knowledge.\n\nLlama|Score\n-|-\n**8B-distill**|**31.8±0.2**\n8B|29.3±0.1\n7B|27.9±0.1\n\nSecond, to further validate this, we conducted additional experiments with Qwen models of various sizes, including standard and knowledge-distilled variants. Table R1 shows that (1) **larger models outperform smaller ones** and (2) among similar-sized models, **knowledge-distilled versions (e.g., DeepSeek) perform better**. This confirms that both model scale and knowledge quality significantly impact performance. The table below shows partial results.\n\nQwen2.5|Score\n-|-\n**1.5B-distill**|**27.7±0.1**\n3B|27.0±0.2\n1.5B|23.1±0.1\n0.5B|20.3±0.1\n\nFinally, our comparison with DipNet (§ 5.2, Figure 6) shows that as data increases, both improve, but LLMs leverage general knowledge for superior data efficiency. At 500 games, DipLLM outperforms DipNet by 6.7%, while **fine-tuning smaller specific models yields only marginal gains**.\n\n---\n\n**Q2. Why didn’t DipLLM use an online search like Cicero, and how does this impact performance?**\n\nA2. As noted in the **Limitations** section, equilibrium search demands **substantial computational resources** (192–448 GPUs) and incurs high inference costs, making its integration with LLMs highly expensive. Despite this, DipLLM outperforms Cicero, even when Cicero utilizes a limited number of search iterations (§5.2, Figure 5).\n\nTo further explore DipLLM’s potential, we integrated equilibrium search during the rebuttal period. Given computational constraints, we conducted as many preliminary experiments as possible. Our evaluation (Table R2) shows that **incorporating search improves decision quality (+3.5% win rate, rollouts=10)**, demonstrating DipLLM’s capacity to leverage enhanced reasoning while preserving sample efficiency.\n\nRollouts|Score\n-|-\n0|16.7±0.1\n5|18.1±0.2\n**10**|**20.2±0.1**\n---\n**Q3. Why doesn't DipLLM use pure self-play? Could iterative self-play improve it?**\n\nA2. **Pure self-play requires the base model to already possess strong reasoning abilities and substantial computational resources to generate data [1]**. Due to the complexity of Diplomacy, unfine-tuned LLMs perform poorly, resulting in low-quality data. To overcome this, we leverage DipNet to generate higher-quality training data, making the process more efficient.\n\nThe suggestion to **improve DipLLM through repeated self-play is valuable**. To evaluate this, we use fine-tuned DipLLM incorporating equilibrium search to generate 100 games for further fine-tuning. As shown in Table R3, performance **improved by 2.3% after just one iteration**, indicating that multiple rounds could be beneficial. However, computational constraints limit scalability. Future work may explore parallelized self-play or distillation to enhance efficiency.\n\nAgent|Score\n-|-\nDipLLM|29.3±0.2\n**DipLLM+self-play**|**31.6±0.1**\n---\n**Q4. Explain details on the transition from DipNet data to the final autoregressive factorization format.**\n\nA4. Our data collection process consists of the following steps:\n1. **Generating Raw Data**:\nDipNet interacts with the Diplomacy environment as player $i$, producing raw data: game states $s$, joint actions $a_i^{1:D}$, and action probability values $\\boldsymbol{\\tau}_i(a_i^{1:D}|s)$.\n1. **Computing Joint Q-Values**: The raw data is processed using the piKL-Hedge algorithm to compute Q-values $\\boldsymbol{Q}(s,a_i^{1:D})$ for joint actions.\n2. **Action Decomposition**: Joint actions are decomposed into unit-level actions, with Q-values $Q_i^d(·,a_i^d)$ and $\\tau_i^d(a_i^d|·)$ assigned to each unit action via Equation 3.\n3. **Textual Formatting**: Game states and decomposed actions are converted to text. Each unit action is stored as ground truth $a_i^d$ sequentially with preceding actions recorded as context $c_i^{1:d-1}$.\n4. **Final Data Storage**: Data is stored in the transition  $(s,c_i^{1:d-1},a_i^d,Q_i^d,\\tau_i^d)$.\n\nFor more details on the stored data, please refer to Appendix B: Full Prompts for DipLLM.\n\n`Summary Response`\n\nThanks for your valuable comments, which helped us strengthen our experimental evaluation. Our results show that **a stronger backbone (Table R1), online search (Table R2), and self-play (Table R3) all further enhance our model's performance**. Please let us know if we have sufficiently addressed your concerns—we would be happy to engage in further discussions to improve our manuscript.\n\n[1] Learning to Reason with LLMs. OpenAI, 2024", "claims_and_evidence": "DipLLM can approximate a Nash-like equilibrium policy via autoregressive factorization and fine-tuning.", "methods_and_evaluation_criteria": "Their main evaluation strategy uses 1v6 tournaments against strong baselines, which has been used in Diplomacy research to assess individual agent strength in a multi-agent environment.", "theoretical_claims": "The joint policy formed by multiplying the factorized distributions can match the original distribution from piKL-Hedge.", "experimental_designs_or_analyses": "he experiments compare DipLLM to well-known baselines, including Cicero.", "relation_to_broader_scientific_literature": "Overall, the paper appears consistent with, and adds to, an emerging literature combining large foundation models with multi-agent game theory.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "Strengths:\nThe explanation of autoregressive factorization is well-presented. Including both theoretical and empirical justification is commendable.\n\nWeaknesses:\nWhile the data usage is small, it is still anchored in offline or external equilibrium-search-based generation (from DipNet + piKL-Hedge). A purely self-play iteration might be tested to illustrate robust self-improvement or autonomy.\n\nAlthough the authors show that Cicero’s performance improves with search rollouts, they do not attempt to incorporate an online search procedure for DipLLM, which might reinforce or surpass results further.", "other_comments_or_suggestions": "M1ore details on the transition from DipNet data to the final autoregressive factorization format would be helpful.", "questions_for_authors": "1.  How critical is the general knowledge from the base LLM (e.g., on language or reasoning tasks) to the success in Diplomacy? Would a specialized but smaller model, with the same data, still fall short?\n\n2.  Could DipLLM be improved further through repeated self-play to generate data and refine its policy iteratively (rather than relying on another anchor model)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "PdgTINGdIF", "reviewer": "Reviewer_ELUa", "summary": "DipLLM is a fine-tuned LLM designed to play the complex multiplayer game Diplomacy. DipLLM leverages an autoregressive factorization framework to simplify multi-unit action assignments into unit-level decisions. By fine-tuning with only 1.5% of the data needed by the state-of-the-art Cicero model, DipLLM achieves superior performance, demonstrating the potential of LLMs in complex strategic decision-making in multiplayer games.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Detailed tables are available at https://sites.google.com/view/dipllm.\n\n**Q1. Why define rewards only for joint actions but not for single actions?**\n\nA1. **This setup follows prior work** on Diplomacy [1], where a player's action is the decisions of all units, with rewards based on the resulting state. As it is difficult to **quantify the contribution of each unit action**, we maintain the setup of defining rewards at the joint action, allowing the model to learn proper credit assignment on its own.\n\n---\n**Q2. Why not use preference learning to optimize the LLM model?**\n\nA2. Preference learning requires high-quality data, but **manual annotation is costly** [2]. Diplomacy’s complexity further complicates data collection. Our attempt to **label preferences by reward magnitude and train with DPO (Table R4) failed**, likely due to unclear preference definitions for factored actions.\n\nConversely, tasks with well-defined reward functions, like Go and Diplomacy, are better **suited for reward optimization**. Diplomacy’s reward function, defined by game scores, enables effective optimization. To highlight our method's advantages, we compared it to the **reward-based algorithm PPO**. Table R4 shows our approach outperforms.\n\nMethod|Score\n-|-\n**Ours**|**29.3±0.2**\nPPO|23.4±0.2\nDPO|1.2±0.1\n---\n**Q3: Why not assign different weights to different unit actions?**\n\nA3. We considered weighting but prioritized aligning the joint policy with the Nash equilibrium over individual unit action distributions. Theorem 1 (L180-L183) shows that if Equation 3 holds, the joint decomposed policy matches the original approximate Nash equilibrium strategy. Thus, **weighting may alter unit action distributions but not the overall joint strategy.**\n\n---\n**Q4. Explain the details of the data collection and evaluation processes. Any risk of data leakage?**\n\nA4. Due to space limits, we refer you to our response to Reviewer mCms, Q4, for data collection details. To evaluate model performance, we use **Meta's offline Diplomacy environment** and an opponent pool. In each game, two agents are randomly sampled in a 1v6 setup—one controls a single power, while identical copies of the opposing agent control the other six powers. There is **no risk of data leakage**, as evaluation is based on adversarial gameplay rather than a fixed test set.\n\n---\n**Q5. Explain the meanings for the scores in Figure 4.**\n\nA5. Figure 4's scores show each **agent's (y-axis) sum-of-squares score** when competing against six copies of another agent (x-axis).\n\n---\n**Q6. Why is Richelieu only compared in Figure 4, not Table 1?**\n\nA6. Since Richelieu's code was unavailable despite an open-sourced repository, Figure 4 (L299) shows the **Prompt (Richelieu) agent performed significantly worse**. Including it in Table 1 would **inflate other agents' scores**, so we excluded it for fairness. To address your concerns, Table R5 includes results with Richelieu's opponent pool.\n\n---\n**Q7. Does DipLLM outperform top LLMs (e.g., OpenAI-o3, DeepSeek-R1)?**\n\nA7. With O3 unavailable, we tested DipLLM against OpenAI-o3-mini and DeepSeek-R1. Table R6 shows **DipLLM's superiority over strong reasoning models**, which struggle with the vast action space.\n\nAgent|Score\n-|-\n**DipLLM**|**60.7±0.5**\nOpenAI-o3-mini|16.3±0.1\nDeepseek-R1|14.6±0.1\nGPT4o|3.7±0.1\n\n---\n**Q8. Clarify contributions to distinguish DipLLM from previous work.**\n\nA8. We are the **first to fine-tune LLMs for Diplomacy**. Previous methods either trained small, specific models on large-scale data or used prompt-driven LLMs. In contrast, we fine-tune LLMs with relatively **small data to approximate Nash equilibrium strategies**, achieving superior performance. To achieve this, we introduced an autoregressive factorization framework and a theoretically grounded fine-tuning approach within this framework.\n\n---\n**Q9. Could SFT lead to overfitting on the joint action's final objective?**\n\nA9. No, our fine-tuning does not cause overfitting to the joint action's final objective. Unlike standard SFT, our objective is to **minimize KL divergence to an approximate Nash equilibrium**, ensuring alignment with the equilibrium distribution rather than mere action imitation. Our ablation study (L416-430) confirms this, showing in Figure 7 that the model **learns the target distribution** instead of overfitting those actions.\n\n---\n**Q10. The improvement in metrics is not compelling, especially in Survived.**\n\nA10. Survived is analogous to a **draw** in chess or Go and not strictly a **\"higher is better\"** metric. When win rate rises while defeat rate stays constant, survival naturally decreases. **DipLLM, using only 1.5% of Cicero’s training data, achieved a higher win rate and lower defeat rate with a similar survival rate**, demonstrating stronger strategic performance.\n\n[1] Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning. ICLR 2023\n\n[2] A Survey of Direct Preference Optimization. ArXiv 2025", "claims_and_evidence": "1、Equations (1) and (2) are in identical forms. The primary innovation by the authors is the Autoregressive Factorization, which decomposes a reward objective into multiple action objectives. The paper's contribution appears to be incremental. The authors should clarify their specific contributions to distinguish their work from existing methods in the learning objective\n2、In Figure 3, raw data is collected by interacting with the environment via DipNet. Q-values are generated through piKLHedge search, and the data is stored in prompt form.  DipLLM seems to enhance the model by generating offline data and using rewards. Why not use preference optimization, which might be more suitable than SFT for this purpose? The authors should consider discussing the rationale behind their chosen approach.", "methods_and_evaluation_criteria": "1、The single action does not have a specified reward; Qi represents the reward for the joint action. Is this setup reasonable? Could SFT lead to overfitting on the joint action's final objective?\n2、In the Data Collection part, for any joint action, the unit-level values are set equal to the original joint value. This setup seems unreasonable; should different weights be assigned?\n3、What are the details of the data collection and evaluation processes? Is there a risk of data leakage?", "theoretical_claims": "I have no comments.", "experimental_designs_or_analyses": "1.\tTable 1 does not compare with the latest method, Richelieu; the comparison with Richelieu only appears in Figure 4.\n2.\tFigure 4 is confusing; the reasons for the scores on the horizontal and vertical axes are unclear.\n3.\tThe improvement in metrics is not compelling. For example, DipLLM achieved 50.3%±0.7% in Survived, while Cicero achieved 50.1%±0.5%.\n4.\tCompared to the best large language models (e.g., reasoning models like OpenAI-o3, DeepSeek-R1), does DipLLM have better performance?", "relation_to_broader_scientific_literature": "I have no comments.", "essential_references_not_discussed": "I have no comments.", "other_strengths_and_weaknesses": "I have no comments.", "other_comments_or_suggestions": "I have no comments.", "questions_for_authors": "1、DipLLM seems to enhance the model by generating offline data and using rewards. Why not use preference optimization, which might be more suitable than SFT for this purpose? The authors should consider discussing the rationale behind their chosen approach.\n2、In the Data Collection part, for any joint action, the unit-level values are set equal to the original joint value. This setup seems unreasonable; should different weights be assigned?\n3、What are the details of the data collection and evaluation processes? Is there a risk of data leakage?\n4、Table 1 does not compare with the latest method, Richelieu; the comparison with Richelieu only appears in Figure 4.\n5、Compared to the best large language models (e.g., reasoning models like OpenAI-o3, DeepSeek-R1), does DipLLM have better performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0gZ4MxMZk1", "reviewer": "Reviewer_FG1V", "summary": "This paper introduces DipLLM, a fine-tuned Large Language Model (LLM) designed for strategic decision-making in the game of Diplomacy. The authors argue that traditional equilibrium search methods require substantial computational resources, whereas fine-tuning an LLM can yield superior performance with significantly less data. The proposed DipLLM employs an autoregressive factorization framework to break down complex multi-unit action assignments into sequential unit-level decisions. The learning objective is designed to approximate the Nash equilibrium, and fine-tuning is performed using a Diplomacy-specific dataset. Empirical results demonstrate that DipLLM outperforms Cicero—a state-of-the-art Diplomacy agent—while requiring only 1.5% of Cicero’s training data. The authors also provide theoretical analysis to establish the equivalence and optimality of their approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Detailed tables are available at https://sites.google.com/view/dipllm.\n\n**Q1. The approach still depends on externally generated datasets.**\n\nA1. Our approach relies on externally generated data to enable efficient data collection and significantly accelerate LLM training. While self-play is a viable alternative that does not require external data, **training LLMs from scratch through self-play is impractical, as it requires the base model to already possess strong reasoning abilities and substantial computational resources to generate data [1]**. However, unfine-tuned LLMs perform poorly in Diplomacy, leading to low-quality data when interacting with the environment. To overcome this, we leverage DipNet to generate higher-quality training data, making the process more efficient.\n\nTo evaluate whether DipLLM can further refine its strategy through self-play, we conducted an experiment in which the fine-tuned DipLLM generated data via self-play and underwent a second round of fine-tuning. As shown in Table R3, this led to further performance improvements, highlighting the potential of self-play. **With advances in LLM inference efficiency and capabilities, iterative self-play could become a viable alternative, reducing reliance on externally generated datasets.**\n\nAgent|Score\n-|-\nDipLLM|29.3±0.2\n**DipLLM+self-play**|**31.6±0.1**\n\n---\n**Q2. Why is the method only evaluated on no-press Diplomacy.**\n\nA2. Diplomacy is widely recognized as a complex multi-agent benchmark, following Chess, Go, and Poker. It presents more difficult  challenges due to its **immense combinatorial action space**—with up to $10^{64}$ possible choices per turn—stemming from its mechanics, where each player controls up to 34 units, each with 26 possible actions.\n\nThis complexity, compounded by **intricate player interactions**, makes Diplomacy a particularly demanding testbed for AI decision-making. Just as prior work on Poker (e.g., **Libratus [2]**) and Go (e.g., **AlphaGo [3]**) focused exclusively on their respective domains, existing Diplomacy research—including **DipNet [4], Brbot [5], SearchBot [6] and DORA [7] —has only evaluated on no-press Diplomacy.** Our evaluation follows this standard to make sure that comparisons with well-known standards are fair and useful.\n\n\n---\n**Q3. Why compare data size between online RL methods and DipLLM (offline)?**\n\nA3. Our primary motivation for comparing data size is to highlight the **cost efficiency** of our method. Cicero, an off-policy RL method, follows a structured data collection pipeline:\n\n1. Generating candidate action sets using its RL policy,\n\n2. Refining actions through equilibrium search,\n\n3. Interacting with the environment and storing the generated trajectories in a **large replay buffer**\n\nOur method follows a similar process, utilizing DipNet and equilibrium search to generate and refine candidate actions, which are then stored as an **offline dataset**. Given these similarities, comparing data size remains meaningful.\n\nAs you pointed out, online RL has low sample efficiency and requires substantial computational resources, which motivates our choice of offline training for LLMs. Regarding computational cost, **Cicero requires 444 GPUs** running for an entire week to generate data. In contrast, **our approach achieves superior performance with just 8 GPUs**, demonstrating a clear advantage in both efficiency and cost.\n\nMoreover, Cicero and DNVI are not purely online RL methods but instead combine supervised learning (SL) with off-policy RL. **Both methods initially pretrain on tens of thousands of human expert games using SL before RL training.** This further highlights the efficiency of our method, as we achieve superior results with significantly lower data and computational costs.\n\n---\n`Summary Response`\n\nThank you for your valuable comments, which have helped us refine our experimental evaluation and gain deeper insights for future research. We are honored to have **your recognition of our paper’s structure, theoretical analysis, and empirical performance**. Please let us know if we have adequately addressed your concerns—we would be happy to engage in further discussions to improve our manuscript.\n\n[1] Learning to Reason with LLMs. OpenAI, 2024\n\n[2] Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 2018.\n\n[3] Mastering the game of Go with deep neural networks and tree search. Nature, 2016.\n\n[4] No-press diplomacy: Modeling multi-agent gameplay. NeurIPS, 2019.\n\n[5] Learning to Play No-Press Diplomacy with Best Response Policy Iteration. NeurIPS, 2020.\n\n[6] Human-Level Performance in No-Press Diplomacy via Equilibrium Search. ICLR, 2021.\n\n[7] No-Press Diplomacy from Scratch. NeurIPS, 2021.", "claims_and_evidence": "The author claims to use only 1.5% of the data required by the state-of-the-art Cicero model. However, comparing the size of data between an online reinforcement learning (RL) method and an offline supervised learning method is inherently flawed and lacks meaningful insight. Online RL methods typically require more data due to their exploration and iterative learning nature, whereas supervised methods rely on pre-collected, labeled datasets.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper is well-situated in the context of:\n\n- Multi-agent reinforcement learning (e.g., equilibrium search methods like Cicero).\n- LLM-based decision-making.\n- Strategic AI in complex games (e.g., Go, Poker, Diplomacy).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "#### Strengths\n- Strong empirical performance: Outperforms Cicero while using only 1.5% of its training data.\n- Theoretical grounding: Provides formal proofs for the equilibrium properties of the method.\n- Well-structured experimental evaluation: Includes ablation studies and comparisons with baseline methods.\n#### Weaknesses\n- Data reliance: The approach still depends on externally generated datasets.\n- Limited generalization: The method is only evaluated on no-press Diplomacy.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "No questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iiC3feyYJS", "reviewer": "Reviewer_vcKR", "summary": "This work proposes to fine-tune LLM with a small amount of data to achieve strong performance in Diplomacy. They propose to factorize the combinatorial joint action space into manageable subspaces in an autoregressive manner, and derive a corresponding learning objective for the factorized actions to fine-tune the LLM. The fine-tuned LLM outperforms various agents including Cicero using a small amount of fine-tune data.\n\n## update after rebuttal\nThe authors' response has addressed my concern. I'll keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Explain the definition of the original joint Q-value.**\n\nA1. In the context of Diplomacy, we define the joint Q-value as the **expected cumulative reward** a player receives after executing a given joint action from the current state [1]. When applying **iterative equilibrium search methods** such as piKL-Hedge, this expected cumulative reward can be computed using the following equation:\n\n$$\\boldsymbol{Q}\\_i(s,a_i)=\\mathbb{E}\\_{\\boldsymbol{a}\\_{-i}\\sim\\boldsymbol{\\pi}\\_{-i}(\\cdot|s)}[u_i(s,a_i,\\boldsymbol{a}\\_{-i})]$$\nwhere $\\pi_{-i}$ ​represents the strategy followed by other players for their joint actions, and $u_{i}$​ denotes the utility function after completing the search under the current state and all players' actions.\n\n---\n**Q2. Explain the theoretical justification for factorization.**\n\nA2. We appreciate your attention to the theoretical justification of our factorization approach. Unlike HAPPO, which first defines decomposed A-values and then establishes equations they satisfy, **our factorization equation serves as a flexible credit assignment condition rather than a strict definition**. It ensures that as long as the total value of $D \\cdot Q$ is distributed across different unit actions, the factorization remains valid. Any method that adheres to this assignment rule does not affect our theoretical guarantees or experimental results.\n\nWhile multiple approaches can satisfy this condition, due to character limits, we provide **a simple example inspired by Q-Transformer** [2]. \n\n$$Q_i\\left(s,c_i^{1:d-1},a_i^d\\right)\\triangleq\\begin{cases}\n\\mathbb{E}\\_{a_i^{d+1}\\sim\\pi\\_i^{d+1}}\\left[Q\\_i\\left(s,c_i^{1:d},a\\_i^{d+1}\\right)\\right],&\\text{if }d\\in\\\\{1,\\ldots,D-2\\\\}\\\\\\\\\n\\mathbb{E}\\_{a_i^{d+1}\\sim\\pi_i^{d+1}}\\left[\\boldsymbol{Q}\\_i\\left(s,[a_i^{1:d},a\\_i^{d+1}]\\right)\\right],&\\text{if }d=D-1\\\\\\\\\nD\\cdot\\boldsymbol{Q}\\_i\\left(s,a\\_i^{1:D}\\right)-\\sum_{d=1}^{D-1}Q\\_i^d\\left(s,c\\_i^{1:d-1},a\\_i^d\\right),&\\text{if }d=D\n\\end{cases}$$\n\nThe intuition behind this formulation is that except for the last two actions (d=D, D-1), the **Q-value is the expected value under the decomposed policy for the next dimension’s actio**n. The penultimate dimension (d=D-1) is assigned the **expected joint Q-value over the remaining action**. Finally, the terminal dimension’s value is determined by the joint Q-value of the complete action sequence. Based on this definition, we now provide a formal proof to justify this factorization.\n\n*Proof*. For any joint action $a_i^{1:D}=\\\\{a_i^1,a_i^2,\\ldots,a_i^D\\\\}$ and its decomposed action components, the following identity holds:\n\n$$\\begin{aligned}\n&\\sum\\_{d=1}^DQ^d_i\\left(s,c_i^{1:d-1},a_i^d\\right)\\\\\\\\\n&=\\sum\\_{d=1}^{D-1}Q^d_i\\left(s,c\\_i^{1:d-1},a\\_i^d\\right)+Q^D_i\\left(s,c\\_i^{1:D-1},a\\_i^D\\right)\\\\\\\\\n&=\\sum\\_{d=1}^{D-1}Q^d_i\\left(s,c\\_i^{1:d-1},a\\_i^d\\right)+\\left[D\\cdot\\boldsymbol{Q}\\_i\\left(s,a\\_i^{1:D}\\right)-\\sum\\_{d=1}^{D-1}Q_i^d\\left(s,c\\_i^{1:d-1},a_i^d\\right)\\right]\\\\\\\\\n&=D\\cdot\\boldsymbol{Q}\\_i\\left(s,a\\_i^{1:D}\\right)\n\\end{aligned}$$\n\nWe also provide a **rigorous fundamental definition of decomposed Q-values** and formally prove their soundness and correctness **using Bayes' theorem and the log-sum-exp inequality**. Due to space constraints, we are unable to present the full details here, but we would be happy to include them in the second round of the rebuttal if you are interested.\n\n---\n**Q3. Explain the equation for $\\tau_i$ in Equation 3 (L217).**\n\nA3. Our formulation distributes the joint anchor policy $\\boldsymbol{\\tau}_i$​ across the factored unit actions. The key property of this distribution is that each unit’s $\\tau_i$​ probability remains aligned with the joint $\\boldsymbol{\\tau}_i$, ensuring consistency. This relationship can be formally expressed as:\n$$\\tau_i\\left(a_i^d|s,c_i^{1:d-1}\\right)\\triangleq\\begin{cases}\n\\tau_i\\left(a_i^{d+1}|s,c_i^{1:d}\\right),&\\text{if }d\\in\\\\{1,\\ldots,D-1\\\\}\\\\\\\\\n\\boldsymbol{\\tau}_i\\left(a_i^{1:D}|s\\right),&\\text{if }d=D \n\\end{cases}$$\n\nThis formulation leads to the result that $\\prod_{d=1}^D\\tau_i\\left(a_i^d|s,c_i^{1:d-1}\\right)\\neq\\boldsymbol{\\tau}_i(a_i^{1:D}|s)$. However, this discrepancy has no impact on our approach because our primary concern is not whether the factored $\\tau_i$ precisely reconstructs $\\boldsymbol{\\tau}_i$, but rather whether the original and factored policies $\\boldsymbol{\\pi}_i$ remain equivalent.\n\n---\n`Summary Response`\n\nWe sincerely appreciate your attention to the theoretical justification for factorization. Your insights help strengthen our theoretical analysis and provide valuable directions for future research. Please let us know if our response has adequately addressed your concerns. We would be delighted to engage in further discussions to refine and improve our manuscript.\n\n[1] Mastering the game of no-press diplomacy via human-regularized reinforcement learning and planning. ICLR, 2023.\n\n[2] Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. CoRL, 2023.", "claims_and_evidence": "Most of the claims are clear and convincing. My main concern is explained in detail in the theoretical claim part.", "methods_and_evaluation_criteria": "This submission is well-motivated by the combinatorial action space of Diplomacy and the inherent limitation of prompt-based LLM agents. The proposed autoregressive factorization method for fine-tuning is intuitive and effective.", "theoretical_claims": "My main concern is about the theoretical justification for the factorization of the Q-function in Eq. (3). \n\n> Claim 1 (L208-209): $\\mathbf{Q}_i$ represents the **reward** for the joint action.\n\nThe factorization of Eq. (2) is built on piKL-Hedge from Eq. (1). In the piKL-Hedge paper [1] and in standard RL literature [2], $Q_i(s, a)$ is the expected future reward for player $i$ from playing action $a$ in state $s$, not the reward for the joint action, which is $r_i(s, a)$. The same problems exist in Eq. (1) (L150) where the $Q$ is used without definition, and in Eq. (3) (L165) where $Q_i^d$ is defined to \"represent the **reward** of action $a_i^d$\".\n\nMoreover, for multi-player games, the Q function not only depends on the state and action of the current player, but also depends on the actions or policies of other players [3]. The non-standard and unclear definition of $Q$ makes it hard to verify the correctness of factorization in Eq. (3).\n\n[1] Jacob, Athul Paul, et al. \"Modeling strong and human-like gameplay with KL-regularized search.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. Vol. 1. No. 1. Cambridge: MIT press, 1998.\n\n[3] Littman, Michael L. \"Markov games as a framework for multi-agent reinforcement learning.\" Machine learning proceedings 1994. Morgan Kaufmann, 1994. 157-163.\n\n> Claim 2 (L215): $\\frac{1}{D}\\sum_{d=1}^D Q_i^d(s, c_i^{1:d-1},a_i^d) = \\mathbf{Q}_i(s, a_i^{1:D})$\n\nThis factorization is the key equation used in subsequent theoretic results but it is not justified. If the Q here is the expected future reward as defined in piKL-Hedge, it is not trivial to see why this factorization holds. Some works in MARL like [4, 5] also discuss decomposition in autoregressive or sequential policy, and their decompositions are usually in the form of expectation over policies. I would suggest the authors theoretically justify why this factorization holds since it is the core equation in their method.\n\n[4] Kuba, Jakub Grudzien, et al. \"Trust region policy optimisation in multi-agent reinforcement learning.\" arXiv preprint arXiv:2109.11251 (2021).\n\n[5] Fu, Wei, et al. \"Revisiting some common practices in cooperative multi-agent reinforcement learning.\" arXiv preprint arXiv:2206.07505 (2022).\n\n> Claim 3 (L217): $\\tau_i^d(a_i^d|s,c_i^{1:d-1}) = \\mathbf{\\tau}(a_i^{1:D}|s)$\n\nI think there might be a missing $\\Pi_{d=1}^D$ on the left-hand side of this equation. The probability of the joint action should be the product of the probability of each sub-action. I would suggest the authors check other potential typos in their manuscript.", "experimental_designs_or_analyses": "The experiment is extensive and valid. The authors compare their method with various methods including SOTA agents like Cicero. They also provide detailed ablations and case studies to analyze their method.", "relation_to_broader_scientific_literature": "This paper proposes to fine-tune LLM in specific games for strategic decision-making. This approach can be generally applied to many other domains. It also provides a new competitive agent for Diplomacy, which is a common game used for studying language and strategic play.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "N.A.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "This work is well-motivated. The proposed method is intuitive and effective. And the empirical experiment results are strong and valid. \n\nMy main concern is about the theoretic justification for the factorization in Eq. (3), which is discussed in detail in the \"Theoretical Claims\" part of my review. The soundness of this equation is of great importance because it is the foundation of subsequent theoretic results. The answer to this question would influence my final assessment of this work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose fine-tuning an LLM to play no-press Diplomacy and demonstrate superior performance to prior SOTA. Their technique uses a piKL-Hedge inspired learning objective that they show how to efficiently factorize over Diplomacy’s large combinatorial action space. The reviewers all found the approach clearly explained. While DipLLM outperforms prior approaches, it does so by building on top of pre-trained models (LLMs, DipNet) that have previously consumed large amounts of data making the strict training-data comparison a bit unfair. Lastly, I will note that the authors made a significant effort to address all reviewers’ concerns in the rebuttal process.
Decision: Accept (poster)