Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning
Authors: Anton Bakhtin
David J Wu
Adam Lerer
Jonathan Gray
Athul Paul Jacob
Gabriele Farina
Alexander H Miller
Noam Brown
Venue: ICLR 2023 notable top 5% (2023)

Abstract: No-press Diplomacy is a complex strategy game involving both cooperation and competition that has served as a benchmark for multi-agent AI research. While self-play reinforcement learning has resulted in numerous successes in purely adversarial games like chess, Go, and poker, self-play alone is insufficient for achieving optimal performance in domains involving cooperation with humans. We address this shortcoming by first introducing a planning algorithm we call DiL-piKL that regularizes a reward-maximizing policy toward a human imitation-learned policy. We prove that this is a no-regret learning algorithm under a modified utility function. We then show that DiL-piKL can be extended into a self-play reinforcement learning algorithm we call RL-DiL-piKL that provides a model of human play while simultaneously training an agent that responds well to this human model. We used RL-DiL-piKL to train an agent we name Diplodocus.
In a 200-game no-press Diplomacy tournament involving 62 human participants spanning skill levels from beginner to expert, two Diplodocus agents both achieved a higher average score than all other participants who played more than two games, and ranked first and third according to an Elo ratings model.

Published as a conference paper at ICLR 2023
MASTERING THE GAME OF NO-PRESS DIPLOMACY VIA HUMAN-REGULARIZED REINFORCEMENT LEARNING AND PLANNING
Anton Bakhtin* Meta AI
David J Wu"
Adam Lerer* Meta Al
Jonathan Gray* Meta AI
Athul Paul Jacob . MIT
Meta AI
Gabriele Farina" Meta AI
Alexander H Miller
Noam Brown Meta Al
Meta AI
ABSTRACT
No-press Diplomacy is complex strategy game involving both cooperation and competition that has served as benchmark for multi-agent Al research. While self-play reinforcement learning has resulted in numerous successes in purely adversarial games like chess, Go, and poker, self-play alone is insufficient for achieving optimal performance in domains involving cooperation with humans. We address this shortcoming by first introducing a planning algorithm we call DiL-piKL that regularizes a reward manttdimxiiiinngggee policy toward a human imitation- learned policy. We prove that this is no-regret learning algorithm under modi- fied utility function. We then show that DiL.-piKI. can be extended into a self-play reinforcement learning algorithm we call RL-DiL-piKKL that provides model of human play while simultancously training an agent that responds well to this hu- man model. We used RL-DiL-piKL to train an agent we name Diplodocus. In a 200-game no- no- press Diplomacy tournament involving 62 human participants span- ning skill levels from beginner to expert, two Diplodocus agents both achieved a higher average score than all other participants who played more than two games, and ranked first and third according to an Elo ratings model.
INTRODUCTION
In two-player zero-sum (2p0s) settings, principled self-play algorithms converge to a minimax equi- librium, which in a balanced game ensures that player will not lose in expectation regardless of the opponent's 's strategy (Neumann)
1928). This fact has allowed self-play, even without human
data, to achieve remarkable success 2p0s games like chess (Silver et al.) 2018). Go (Silver al.
2017], poker (Bowling et al. 2015: Brown & Sandholm 2017). and Dota 2 (Berner et al. 2019
In principle, any finite 2p0s game can be solved via self self-play given sufficient compute and memory. However, in games involving cooperation, self-play alone no longer guarantees good performance when playing with humans, even with infinite compute and memory. This is because in complex domains there may be arbitrarily many conventions and expectations for how to cooperate, of which humans may use only a small subset (Lerer & Peysakhovich 2019). The clearest example of this is language. A self-play agent trained from scratch without human data in cooperative game in- volving free-form communication channels would almost certainly not converge to using English the medium of communication. Obviously, such agent would perform poorly when paired with a human English speaker. Indeed, prior work has shown that naïve extensions of self-play from scratch without human data perform poorly when playing with humans or human-like agents even in dialogue-free domains that involve cooperation rather than just competition. such as the benchmark games no-press Diplomacy (Bakhtin al., 2021) and Hanabi (Siu al. 2021 Cui et al. 2021
* "qqual first author contribution.
Dota 2 is two-team zero-sum game, but the presence of full information sharing between teammates makes it equivalent to 2p0s. Beyond 2p0s settings, self-play algorithms have also proven successful in highly adversarial games like six-player poker (Brown & Sandholm 2019).

---

Published as a conference paper at ICLR 2023
Recently, @Jccbb et al. 2022 introduced piKL, which models human behavior in many games better than pure behavioral cloning (BC) on human data by regularizing inference- time planning toward a BC policy. In this work, introduce extension of piKL, called DiL-piKL, that replaces piKL's single fixed regularization parameter À with a probability distribution over À parameters. We then show how DiI.-piKI. be combined with self-play reinforcement learning, allowing to train strong agent that performs well with humans. We call this algorithm RL-DiL-piKL.
Using RL-DiL-piKL we trained an agent, Diplodocus, play no-press Diplomacy, difficult bench- mark for multi-agent Al that has been actively studied in recent years (Paquette et al. 2019; Anthony al.) 2020 Gray et al., 2020 Bakhtin et al., 202T Jacob et al., 2022). We conducted a 200-game no-press Diplomacy tournament with a diverse pool of human players, including expert humans, in which we tested two versions of Diplodocus using different RI.-Dil--piK. settings, and other base- line agents. All games consisted of one bot and six humans, with all players being anonymous for the duration of the game. These two versions of Diplodocus achieved the top two average scores in the tournament among all 48 participants who played more than two games, and ranked first and third overall among all participants according to an Elo ratings model. Our code and models are com available at attps://qitan http:///hhu..ccoocccceeeeeeeeeee htt.://cccccrreereeeeaeaaaaaaeaa
cicero
BACKGROUND AND PRIOR WORK
Diplomacy is a benchmark 7-player mixed cooperative/compettive game featuring simultancous moves and a heavy emphasis on negotiation and coordination. In the no-press variant of the game, there no cheap talk communication. Instead, players only implicitly communicate through moves.
In the game, seven players compete for majority control of 34 "supply centers" (SCs) on a map. On each turn, players simultancously choose actions consisting of an order for each of their units to hold, move, support or convoy another unit. If no player controls a majority of SCs and all remaining players agree to draw or turn limit is reached then the game ends in a draw. In this case, we use common scoring system in which the score of player i is / CR, where C, is the number of SCs player i owns. A more detailed description of the rules is provided in Appendix A
Most recent successes in no-press Diplomacy use deep learning to imitate human behavior given corpus of human games. The first Diplomacy agent to leverage deep imitation learning was Paquette cet al. (211)) Subsequent work on no-press Diplomacy have mostly relied
similar architecture
with some modeling improvements (Gray et al. 2020 Anthony et al. 2020 Bakhtin et al.) 2021).
Gray et al. 2020) proposed an agent that plays improved policy via one-ply search. It uses policy and value functions trained on human data to conduct search using regret minimization.
Several works explored applying self-play to compute improved policies. Paquuttee et al. (2019) applicd an actor-critic approach and found that while the agent plays stronger in populations of other self-play agents, it plays worse against population of human-initation agents. |Anthony jet al. (2020) used a self-play approach based on a modification of fictitious play in order to reduce drift from human conventions. The resulting policy is stronger than pure imitation learning in both 1vs6 and 6vs1 settings but weaker than agents that use search. Most recently, [Bakhtin et al. (202T) combined one-ply search based on equilibrium computation with value iteration to produce an agent called DORA. DORA achieved superhuman performance in 2p0s version of Diplomacy without human data, but in the full 7-player game plays poorly with agents other than itself.
Jacob et al. 2022) showed that regularizing inference-time search techniques can produce agents that are not only strong but can also model human behaviour well. In no-press Diplomacy, they show that regularizing hedge (an equilibrium-fnddi human imitation learning policy can match or exceed the human action prediction accuracy of imi- tation learning while being substantially stronger. KL-regularization toward human behavioral poli- cies has previously been proposed in various forms in single- and multi-agent RL algorithms (Nair et al. 2018 Siegel al., 2020 Nair et al. 2020), and was notably employed in AlphaStar (Vinyals) et al. 2019). but this has typically been used to improve sample efficiency and aid exploration rather than to better model and coordinate with human play.
algorithm) with a KL-divergence penalty towards a
An alternative line of research has attempted to build human-compatible agents without relying on human data (Hu et al., 2020 2021; SStruuse et al. 2021). These techniques have shown some success in simplified settings but have not been shown to be competitive with humans in large-scale collaborative environments.
2

---

Published as a conference paper at ICLR 2023
2.1 MARKOV GAMES
In this work, we focus on multiplayer Markov games (Shapley 1953).
Definition. An n-player Markov game A is a tuple (S, A A A1..1..... ,n, p) where S the state space, A is the action space of player i (i 1,... n). r, Sx A x... An R is the reward function for player i, f: S x A1 x ... x An - is the transition function.
The goal of each player i, is to choose a policy T; (s) reward for that player, given the policies of all other players. In case of n 1, Markov game reduces to a Markov Decision Process (MDP) where an agent interacts with a fixed environment.
+ AA; that maximizes the expected
At each state 8, each player i simultaneously chooses an action a; from set of actions A, We denote the actions of all players other than as a_,. Players may also choose a probability distribu- tion over actions, where the probability of action a; is denoted (1 a,) o, (a(a,) and the vector of probabilities is denoted (s) or n.
2.2 HEDGE
Hedge Littlestone & Warmuth (1994); Freund & Schapire (1997) is an iterative algorithm that con- verges to an equilibrium. We use variants of hedge for planning by using them to compute an cquilibrium policy on each turn of the game and then playing that policy.
Assume that after player i chooses an action aj and all other players choose actions a- player i receives reward of un (a., a_s), where u, will come from our RL-trained value function. We denote the average reward in hindsight for action a; up to iteration / as Q'(a,) = E u; u;(as, a
On each iteration t of hedge, the policy (a:) is set according to (a;) x exp (Q: (a;)/m(-1) where K2 is temperature parameter
is proven that if Ke is set to + then as t + oo the average policy over all iterations converges to a coarse correlated equilibrium, though in practice it often comes close to a Nash equilibrium as well. 3S.
In all experiments we set 6₁ = 10v7 on iteration i, where S. is the observed standard deviation of the player's utility up to iteration t, based on a heuristic from Brown et al. (2017). A simpler choice to set K; = 0, which makes the algorithm equivalent to fictitious play (Brown 1951).
Regret matching (RM) (Blackwell et al. 1956: Hart & Mas-Colell 2000) is an alternative
equilibrium-fiddin algorithm that has similar theoretical guarantees to hedge and was used in pre- viously work on Diplomacy Gray et al. (2020); [Bakhtin al. (2021). We do not use this algorithm but we do evaluate baseline agents that use RM.
2.3 DORA: SELF-PLAY LEARNING IN MARKOV GAMES
Our approach draws significantly from DORA (aakhtin et al. 2021 which describe in more detail here. In this approach, the authors run an algorithm that is similar to past model-based reinforeeenn-lcamiing methods such as AlphaZero (Silver al. 2018). except in place of Monte Carlo tree search, which is unsound in simultancous-action games such as Diplomacy or other im- perfect information games, it instead uses an equilibrium-fiding algorithm such hedge or RM to iteratively approximate a Nash equilibrium for the current state (i.e., one-step lookahead search). deep neural net trained predict the policy is used to sample plausible actions for all players to reduce the large action space in Diplomacy down to tractable subset for the equilibrium-finde procedure, and a deep neural net trained predict state values used to evaluate the results of joint actions sampled by this procedure. Beginning with policy and value network randomly ini- tialized from scratch, a large number of self-play games are played and the resulting equilibrium policies and the improved 1-step value estimates computed every turn from equilibrium-finddi are added to a replay buffer used for subsequently improving the policy and value. Additionally, a double-orack (McMahan et al. 2003) method used to allow the policy explore and discover additional actions, and the same equilibrium-findin procedure was also used at test time.
For the core update step. Bakhtin al. (2021) propose Deep Nash Value Iteration (DNVT), a value iteration procedure similar to Nash Q-Leannin (Hu & Wellman 2003). which is a generalization
We use Ki rather than 77 used in Jacob al. 2022 in order to clean up notation. Nit 1/(7 t).
3

---

Published as a conference paper at ICLR 2023
of Q-learning (Watkins) 1989) from MDPs to Stochastic games. The idea of Nash- Q is to compute equilibrium policies a in subgame where the actions correspond to the possible actions in a current state and the payoff's are defined using the current approximation of the value function. [aakhtin et al. (2021) propose an equivalent update that uses a state value function V(s) instead of a state-action value function Q(s, a):
V(s) (1-
[o(d))
where a is the learning rate, a()) is the probability of joint action in equilibrium, a' is joint action, and f is the transition function. For 2p0s games and certain other game classes, this algorithm verges to a Nash equilibrium in the original stochastic game under the assumption that an exploration policy is used such that each state is visited infinitely often.
The tabular approach of Nash-Q does not scale to large games such as Diplomacy. DNVI replaces the explicit value function table and update rule in with value function parameterized by a neural network, (ss; B.) and uses gradient descent to update it using the following loss:
ValueLoss)(..) = ((s;00.--(66 -7 ( -ooad' (1(s,)));)))
The summation used in 2 is not feasible in games with large action spaces as the number of joint actions grow exponentially with the number of players. Bakhtiin et al. (202I) address this issue by considering only subset of actions at each step. An auxiliary function, policy proposal network (s, a;; fz), models the probability that an action a; of player is in the support of the equilib- rium Only the top-k sampled actions from this distribution are considered when solving for the cquilibrium policy a and computing the above value loss. Once the equilibrium is computed, the equilibrium policy is also used to further train the policy proposal network using cross entropy loss:
PolicyLoss (0z) - Σ Σ
ai(a) log (s az; 0z)-
a.EA:
Bakhtin al. (2021) report that the resulting agent DORA does very well when playing with other copies of itself. However, DORA performs poorly in games with human human-like agents.
2.4 PIKL: MODELING HUMANS WITH IMITATION-ANCHORED PLANNING
Behavioral cloning (BC) is the standard approach for modeling human behaviors given data. Behav- ioral cloning learns policy that maximizes the likelihood of the human data by gradient descent on a cross-entropy loss. However, as observed and discussed in Jacob et al. 2022). BC often falls short of accurately modeling or matching human-leve performance, with BC models underperforming the human players they are trained to imitate in games such as Chess, Go, and Diplomacy. Intuitively, it might seem that initializing self-play with imitation-learned policy would result in agent that is both strong and human-like. Indeed, BBkkhtin et al. (202T) showed improved performance against human-like agents when initializing the DORA training procedure from human imitation policy and value, rather than starting from scratch. However, we show in subsection 5.3 that such an approach still results in policies that deviate from human-compatible equilibria.
Jacob et al. 2022) found that an effective solution was to perform search with regularization penalty proportional the KL divergance from human imitation policy. This algorithm is referred to as piKL. The form of piKI. we focus on in this paper is a variant of hedge called piKLe- in which each player seeks to maximize expected reward, while at the same time playing "close" to fixed anchor policy т, The two goals can be reconciled by defining composite utility function that adds a penalty based on the "distance" between the player policy and their anchor policy, with coefficient X; E [0, oo) scaling the penalty.
For each player i, we define i's utility as a function of the agent policy ग. E A(As) given policies π- of all other agents:
Us (TT, nT) (x π -i) X, DKL(TT II)))
(4)
4

---

Published as a conference paper at ICLR 2023
BC
Algorithm 1: DIL-PIKL (for Player 2)
Data: + A, set of actions for Player 1;
Augent
reward function for Player i;
A, set of values consider for
7. pop 00
Player
3, a belief distribution over À values for
Playeri,
7 agenn
piKL
function INITIALIZE(
+-00 0
for each action az 6 A,, do (a:) 0
Equilibrium
Ours
function PLAY) f
BR-BC
sample A ~ 8 let TiA be the policy such that
À agent 0 À pop 0
pop
À agen 0 A pop 00
(ax) x exp Q (a,) + log T1()))
Figure 2: Apop represents the common-
1 1 +
knowledge belief about the À parameter distribution used by all players. Augent repre- sents the value actually used by the agent determine its policy. By having Aupent differ from Apop DiL-piKL interpolates between an equilibrium under the utility function us be- havioral cloning and best response behav- ioral cloning policies. pikI. assumed a com- mon A, which moved it along one axis of the space. Our agent models and coordinates with high-A players while playing lower À itself.
sample an action of TiA
play af E A, and observe actions aL, played by opponents for each a, E A, do
(a,) + Q'=0' (u,aa,
Figure 1: DiL-piKI, algorithm. Lines with highlights show the main differences between this algorithm and piKl--Hedee algorithm proposed in Jacob al. (2022
This results in a modification of hedge such that on each iteration f, (a:) is according to
(a:) exp (a:) + À log (a;)
(5)
Kt-1 +
When À is large, the utility function is dominated by the KL-divergence term A; DKL (T; || T:), and so the agent will naturally tend to play a policy close to the anchor policy т; When X is small, the dominating term is the rewards u, [T,,, a²,;) and so the agent will tend to maximize reward without closely matching the anchor policy Tz.
3 DISTRIBUTIONAL LAMBDA PIKL (DIL-PIKL)
pkKL trades off between the strength of the agent and the closeness to the anchor policy using a single fixed À parameter. In practice, find that sampling À from a probability distribution each iteration produces better performance. In this section, we introduce distributional lambda piKL, (DiL.-iKLL), which replaces the single À parameter in piKL. with probability distribution over values. On each iteration, each player samples a À value from 3; and then chooses policy based on Equation 5 using that sampled A. Figure Thighlights the difference between piKL and DiL-piKL.
One interpretation of DiL-piKL is that each choice of A is an agent type, where agent types with high choose policies closer 7 while agent types with low choose policies that are more "optimal" and less constrained to common boommon-knowledee knowledge anchor policy. A priori, each player is randomly sam- pled from this population of agent types, and the distribution B, represents the common-kan -nowledge uncertainty about which of the agent types player i may be. Another interpretation is that piKI. assumed an exponential relation between action EV and likelihood, whereas DiI.-piKI, results in fatter-tailed distribution that may more robustly model different playing styles or game situations.
3.1 COORDINATING WITH PIKL POLICIES
While piKL and DiL-piKL are intended to model human behavior, an optimal policy in cooperative environments should be closer best response to this distribution. Selecting different À values for

---

Published as a conference paper at ICLR 2023
the Hommon-kkowweeee population versus the policy the agent actually plays allows to interpolate between BC, best response to BC, and equilibrium policies (Figure In practice, our agent samples from 3; during equilibrium computation but ultimately plays low À policy, modeling the fact that other players are unaware of our agent's 's true type.
3.2 THEORETICAL PROPERTIES OF DIL-PIKL
DiL-piKL can be understood sampled form of follow-the-reguknne thee-regularized-eedeee (FTRL). Specifi- cally, one can think of Algorithm as instantiation of FTRL over the Bayesian game induced by the set A, = supp 3; of types X and the regularized utilities ug of each player i. the appendix we show that when player learns using DiK-pi the distributions T for any type X, A, are no-regret with respect to the regularized utilities ú, defined in (4) Formally:
Theorem (abridged), Let W be a bound on the maximum absolute value of any payoff in the game, and Qi n: D. log (a)) Then, for any player type x E Ai, and number of iterations T, the regret cumulated can be upper bounded
W
12 T
log n Inn
U. (. a', U. (TT (1 a'=
min
+ PLA
TEA(A.)
4
A
1
where the game constant Pi,); is defined PiAA, >zog + Q:).
The traditional analysis of FTRL is not applicable to DiL.-iKLL because the utility functions and their gradients, can be unbounded due the nonsmoothness of the term -A, (:)) that appears in the regularized utility function ūr,A,, and therefore a more sophisticated analysis needs to be carried out. Furthermore, even in the special case of a single type (i.e., singleton set A₂), where DiL-piKL coincides with piKL, the above guarantee significantly refines the analysis of piKL in two ways. First, it holds no matter the choice of stepsize rj > 0, thus implying O(log T/(T):) regret bound without assumptions on other than (2(1). Second, in the cases in which ); is tiny, by choosing " 0(1/VT) we recover sublinear guarantee (of order TT) on the regret.
In 2p0s games, the logarithmic regret Theorem immediately implies that the average policy T == T Σ TiP of each player is a Colog -pprrxximate Bayes-Nash equilibrium strategy. In fact, a strong guarantee on the last-iterate convergence of the algorithm can be obtained too:
T
Theorem (abridged: Last-iterate convergence of piKL 2p0s games). When both players 2p0s game learn using DiL-piKL T iterations, their policies converge almost surely to the unique Bayes-Nush equilibrium (x,, of the regularized game defined by utilities ui,x.
The last-iterate guarantee stated Theorem crucially relies on the strong convexity of the reg- ularized utilities, and conceptually belongs with related efforts in showing last-iterate convergence of online learning methods. However, key difficulty that sets apart Theorem i the fact that the learning agents observe sampled actions from the opponents, which makes the proof of the result (as well as the obtained convergence rate) different from prior approaches.
DESCRIPTION OF DIPLODOCUS
By replacing the equilibrium-indde finding algorithm used in DORA with DiL-piKL,, we hypothesize that we can learn strong and human-come compaico policy as well as a value function that can accurately evaluate game states, assuming strong and human-like continuation policies. We call this self-play algorithm RL-DiL-piKL, We use RL-DiL-piKL to train value and policy proposal networks and use DiL-piKL during test-time search.
4.1 TRAINING
Our training algorithm closely follows that of DORA, described in Section 2.3 used are identical to DORA and the training procedure is largely the same, except in place of RM to DiL-piKL with
The loss functions
compute the equilibrium policy distribution and human imitation anchor policy that is fixed for all of training. See Appendix for detailed description of differences between DORA and RI..D..-piKL
each turn of game during self-play,

---

Published as a conference paper at ICLR 2023
4.2 TEST-TIME SEARCH
Following Bakhtin al. 2021 , at evaluation time perform 1-ply lookahead where on each turn we sample up to 30 of the most likely actions for each player from the RL policy proposal network. However, rather than using RM to compute the equilibrium o, we apply DiL-piKL.
As also mentioned previously in Section 3 while our agent samples X from the probability dis- tribution Be when computing the DiL-piKL equilibrium, the agent chooses its own action to ac- tually play using fixed low X. For all experiments, including all ablations. the agent uses the same BC anchor policy. For DiL-piKL experiments for each player i we set 33 to be uniform over (10-1, 10-3 10 10-2, 10-1] and play according to À 10-2, except for the first turn of the game. On the first turn we instead sample from (10-2 10-15, 10-1, 10-0.5) and play according to X = 10-2, so that the agent plays more diverse openings, which more closely resemble those that humans play.
EXPERIMENTS
We first compare the performance of two variants of Diplodocus in a population of prior agents and other baseline agents. We then report results of Diplodocus playing in a tournament with humans.
5.1 EXPERIMENTAL SETUP
In order to measure the ability of agents to play well against a diverse set of opponents, we play many games between AI agents where each of the seven players are sampled randomly from population of baselines (listed in in[Appendix C) or the agent to be tested. We report scores for each of the following algorithms against the baseline population. Diplodocus-Low and Diplodocus-High are the proposed agents that use RL-DiL-piKLL during training with 2 player types {10- 10 and (10-2, 10-1), }, respectively. DORA is an agent that trained via self-play and uses RM as the search algorithm during training and test-time. Both the policy and the value function are randomly initialized at the start of training. DNVI is similar to DORA, but the policy proposal and value networks are initialized from human BC pretraining. DNVI-NPU is similar to DNVI, but during training only the RL value network is updated. The policy proposal network is trained but never fed back self-play workers, to limit drift from human conventions. The trained policy proposal is only used at test time (along with the RL value network). BRBot is an approximate best response the BC policy. was trained the same as Diplodocus, except that during training the agent plays one distinguished player each game with À = 0 while all other players use X R oc. SearchBot one-step lookahead equilibrium search agent from (Gray et al.] 2020), evaluated using their published model. HedgeBot is an agent similar to SearchBot (Gray et 20200) but using our latest architecture and using hedge rather than RM as the equilibrium-fine finding algorithm. FPPI-2 and SL are two agents from (Anthony et al.) 2020), evaluated using their published model.
After computing population scores, as a final evaluation we organized a tournament where we eval- uated four agents for 50 games each in population of online human participants. We evaluated two baseline agents, BRB and DORA, and our new agents, Diplodocus-Low and Diplodocus-Highh
In order to limit the duration of games to only a few hours, these games used a time limit of 5 minutes per turn and a stochastic game-end rule where at the beginning of each game year between 1909 and 1912 the game ends immediately with 20% chance per year, increasing in 1913 to a 40% chance. Players were not told which turn the game would end on for a specific game, but were told the distribution it was sampled from. Our agents were also trained based this distribution Players were recruited from Diplomacy mailing lists and from webdi p omacy .net In order to mitigate the risk of cheating by collusion, players were paid hourly rather than based in in-game performance. Each game had exactly one agent and six humans. The players were informed that there was an Al agent in each game, but did not know which player was the bot in each particular game. In total 62 human participants played 200 games with 44 human participants playing more than two games and 39 human participants playing at least games.
Gaames were run by third-party contractor. In contradiction of the criteria we specified, the contractor ended games artificially early for the first 80 games played in the tournament, with end dates of 1909-1911 heing more common than they should have been. We immediately corrected this problem once it was identified.
7

---

Published as a conference paper at ICLR 2023
5.2 EXPERIMENTAL RESULTS
We first report results for our agents in the fixed population described in Appendix C The results, shown in Table show Diplodocus-Low and Diplodocus-Higg perform the best by wide margin.
Agent
Score against population
Diplodocus-LLow
29% + 1%
Diplodocus-Higl
28% + 1%
DNVI-NPU (retrained) (Bakhtin et al. 2021
20% + 1%
BRBot
18% + 1%
DNVI (retrained) (Bakhtin et al. 2021
15% + 1%
HedgeBot (retrained) (Jacob et a..
2022
14% + 1%
DORA (refrained) (Bakhun al.
13% + 1%
FPPI-2 (Anthony et al. 2020
9% + 1%
SearchBot (Grray ct al. 2020)
7% + 1%
SL (Anthony et al. 2020
6% + 1%
Table 1: Performance of different agents population of various agents. Agents above the line
retrained using our models (see Appendix E Agents below the line were evaluated using the models and the parameters provided by the authors. The shows one standard error.
We next report results for the human tournament in Table For each listed player, we report their average score, Elo rating, and rank within the tournament based on Elo among players who played least 5 games. Elo ratings were computed using a standard generalization of BayesFlo (Coulom) 2005) to multiple players (Hunter) 2004) (see (see[Appene Appendix H for details). This gives similar rankings as average score, but also attempts to correct for both the average strength of the opponents, since some games may have stronger weaker opposition, a player was assigned in each game, since some starting positions in Diplomacy are advantaged over others. To regularize the model, weak Bayesian prior applied such that each player's 's rating was normally distributed around 0 with a standard deviation of around 350 Elo.
well as for which of the seven European powers
Rank
Elo
Avg Score # Games
Diplodocus-Higg
1
181
27%
i 4%
50
Human
2
162
25% + 6%
13
Diplodocus-Low
3
152
26% i 4%
50
Human
4
138
22% + 9%
Human
5
136
22% + 3%
57
BRBot
6
119
23% + 4%
50
Human
102
18%
8%
Human
8
96
17% + 3%
51
DORA
32
-20
13% + 3%
50
Human
43
-187
1% + 1%
7
Table 2: Performance of four diffeernt agents a population of human players, ranked by Elo, among all 43 participants who played at least games. The + shows one standard error.
The results show that Diplodocus-High! performed best among all the humans by both Elo and av- erage score. Diplodocus-Low followed closely behind, ranking second according average score and third by Elo. BRBot performed relatively well, but ended ranked below that of both DiL- agents and several humans. DORA performed relatively poorly.
Two participants achieved a higher average score than the Diplodocus agents, a player averaging 35% but who only played two games, and player with score of 29% who played only one game.
We note that given the large statistical error margins, the results in Table 2 do not conclusively demonstrate that Diplodocus outperforms the best human players, an unambiguous separation between Diplodocus and BRBot. However, the results do indicate that Diplodocus performs at least the level of expert players in this population of players with diverse
do they alone demonstrate
8

---

Published as a conference paper at ICLR 2023
skill levels. Additionally, the superior performance of both Diplodocus agents compared to BRBot is consistent with the results from the agent population experiments in Table I
In addition to the tournament, asked three expert human players to evaluate the strength of the agents in the tournament games based on the quality of their actions. Games were presented to these experts with anonymized labels so that the experts were not aware of which agent was which in each game when judging that agent's 's strategy. All the experts picked a Diplodocus agent as the strongest agent, though they disagreed about whether Diplodocus-High or Diplodocus-Low was best. Additionally, all experts indicated one of the Diplodocus agents as the one they would most like cooperate with in game. We provide detailed responses in Appendix B
5.3 RL TRAINING COMPARISON
Figure 3 compares different RL agents across the course of training. To simplify the comparison, we vary the training methods for the value and policy proposal networks, but use the same search setting at evaluation time.
Score HedgeBot agents
Prediction accuracy human actions
20
60
Diplodocus High
15
Accuracy.
Diplodocus Low
Score.
DNVI-NPU
10
40
DNVI
DORA
20
10000
30000
100000
300000
10000
30000
100000
300000
Num updates (log scale)
Num updates (log scale)
Figure 3: Performance of different agents function of the number of RL training steps. Left: Scores against human-like HedgeBot agents. The gray dotted line at score 1/7 14.3% corresponds tying HedgeBot. The error bars show one standard error. Right: Order prediction accuracy each agent's raw RL policy on a held-out set of human games. The gray dotted line corresponds to the behavioral cloning policy. Overall: Diplodocus-Iligh achieves a high score while also maintaining high prediction accuracy. Unregularized agents DNVI and DORA do far worse both metrics.
As proxy for agent strength, we measure the average score of an agent vs 6 copies of HedgeBot. As proxy for modeling humans, we compute prediction accuracy of human moves on a validation dataset of roughly 630 games held out from training of the human BC model, i.e., how often the most probable action under the policy corresponds to the one chosen by a human. Similar to BBkktti et al. (202I), we found that agents without biasing techniques (DORA and DNVI) diverge from human play as training progress. By contrast, Diplodocus-Hipe achieves significant improvement in score while keeping the human prediction accuracy high.
DISCUSSION
In this work we describe RL-DiL-piKL and use it train an agent for no-press Diplomacy that placed first in human tournament. We ascribe Diplodocus's 's success in Diplomacy to two ideas.
First, DiL-piKL models population of player types with different amounts of regularization human policy while ultimately playing a strong (low-A) policy itself. This improves upon simply playing best response BC policy by accounting for the fact that humans are less likely to play highly suboptimal actions and by reducing overfitting of the best response to the BC policy. Second, incorporating DiL.-pKLL in self-play allows us to learn an accurate value function in a diversity of situations that arise from strong and human-like players. Furthermore, this value assumes a human continuation policy that makes fewer blunders than the BC policy, allowing us to correctly estimate the values of positions that require accurate play (such stalemate lines).
In conclusion, combining human imitation, planning, and RL presents promising avenue for build- ing agents for complex cooperative and mixed-motive environments. Further work could explore regularized search policies that condition on more complex human behavior, including dialogue.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank the anonymous reviewers for the stimulating discussion and for helping improve the paper.
REFERENCES
Thomas Anthony, Tom Eccles, Andrea Tacchetti, János Kramár, Ian Gemp, Thomas Hudson, Nicolas Porcel, Marc Lanctot, Julien Perolat, Richard Everett, Satinder Singh, Thore Grae- pel, and Yoram Bachrach. Learning to play no-press diplomacy with best response policy iteration. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 17987-18003. Curran As- sociates, Inc., 2020. URL http:: httpttppss/pproeeedeiggg neur ips httpp.cccpppee wttpppppppppp////ee/.pppp dl2113322b5622222cc466666
Paper. pdf
Anton Bakhtin, David Wu, Adam Lerer, and Noam Brown. No-press diplomacy from scratch. In Thirty-Fifth Conference Neural Information Processing Systems, 2021.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 191::::912.0668/ 2019.
David Blackwell al. An analog of the minimax theorem for vector payoffs. Pacific Journal of Muthematics, 6(1):1-8, 1956.
Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold' em poker is solved. Science, 347(6218):145-1433 2015.
George W Brown, Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374-376, 1951.
Noam Brown and Tuomas Sandholm. Superhuman Al for heads-up no-limit poker: Libratus beats top professionals. Science, pp. eaao1733, 2017.
Noam Brown and Tuomas Sandholm. Superhuman Al for multiplayer poker. Science, 365(6456): 885-890, 2019.
Noam Brown, Christian Kroer, and Tuomas Sandholm. Dynamic thresholding and pruning for regret minimization. In Proceedings the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Rémi Coulom. 2005.
Brandon Cui, Hengyuan Hu, Luis Pineda, and Jakob Foerster. K-level reasoning for zero-shot coordination in hanabi. Advances in Neural Information Processing Systems, 34:8215-8228, 2021.
FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minac Kwon, Adam Lerer, Mike Lewis, Alexander Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level ai for the game of diplomacy by combining language models with strategic reasoning. Science, 2022.
Brandon Fogel.
To whom tribute due: The next step in scoring systems, 2020. http///wwwwwaaaaa//////oooooroop
URL
pdf
Yoav Freund and Robert E Schapire. A decision-thewe theoretic generalization on-line learning and an application to boosting. Journal of computer and system sciences, (5(1):119-139 1997.
10

---

Published as a conference paper at ICLR 2023
Jonathan Gray, Adam Lerer, Anton Bakhtin, and Noam Brown. Human-level performance in no- press diplomacy via equilibrium search. International Conference on Learning Representa- tions, 2020.
Sergiu Hart and Andreu Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Ecomomelricu, (8(5):1127-11500 2000.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference on Machine Learning, pp. 4410. PMLR, 2020.
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, David Wu, Noam Brown, and Jakob Foerster. Off-belief learning. In International Conference on Machine Learning. PMLR, 2021.
Junling Hu and Michael P Wellman. Nash q-learning for general-sum stochastic games. Journal of machine learning research, 4(Nov):1039-1069, 2003.
David R. Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32.1:384-406, 2004.
Athul Paul Jacob, David Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Ja- cob Andreas, and Noam Brown. Modeling strong and human-like gameplay with kl-regularized search. In International Conference on Machine Learning, pp. 9695-69922 PMLR, 2022.
Adam Lerer and Alexander Peysakhovich. Learning existing social conventions via observation- ally augmented self-play, In Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pp. 107-114. ACM, 2019.
Nick Littlestone and Manfred K Warmuth. The weighted majority algorithm. Information and computation, (08(2):212-261) 1994.
Brendan McMahan, Geoffrey Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In International conference on machine learning, pp. 536 -54,, 2003.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on roboties and automation (ICRA), pp. 6292-6299. IEEE, 2018.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint (rXiv:2006609359, 2020.
J Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen, 100(1):295-320, 1928.
Philip Paquette, Yuchen Lu, Seton Steven Bocco, Max Smith, O-G Satya, Jonathan K Kummerfeld, Joelle Pineau, Satinder Singh, and Aaron Courville. No-press diplomacy: Modeling multi-agent gameplay. In Advances in Neural Information Processing Systems, pp. 4474 4485, 2019.
Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10): 1095-1100, 1953.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
David Silver, Julian Schritwieser,, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 35047676)354,, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140- 1144, 2018.
11

---

Published as a conference paper at ICLR 2023
Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. Evaluation of human-ai teams for learned and rule-based agents in hanabi. Ad- vances in Neural Information Processing Systems, 34:16183-16195, 2021.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515, 2021.
Oriol Vinyals. Igor Babuschkin, Wojeiech M Czameekk, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-3335 2019.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
A DESCRIPTION OF DIPLOMACY
The rules of no-press Diplomacy are complex; full description is provided by Paquette et al. (019). No-press Diplomacy is a seven-player zero-sum board game which map of Europe is divided into 75 provinces. 34 of these provinces contain supply centers (SCs), and the goal of the game is for a player to control a majority (18) of the SCs. Each players begins the game controlling three or four SCs and equal number of units.
The game consists of three types of phases: movement phases in which each player assigns an order to each unit they control, retreat phases in which defeated units retreat to neighboring province, and adjustment phases in which new units are built or existing units are destroyed.
During a movement phase, player assigns an order to each unit they control. A unit's order may be to hold (defend its province), move to a neighboring province, convoy a unit over water, or support a neighboring unit's hold or move order. Support may be provided to units of any player. We refer to tuple of orders, one order for each of player's 's units, as an action. That is, each player chooses one action each turn. There are an average of 26 valid orders for each unit (Paquette et al. 2019)) so the game's 's branching factor is massive and on some turns enumerating all actions is intractable.
Importantly, all actions occur simultaneously. In live games, players write down their orders and then reveal them at the same time. This makes Diplomacy an an optimal policy may need to be stochastic in order to prevent predictability.
game in which
Diplomacy is designed in such way that cooperation with other players is almost essential in order to achieve victory, even though only one player can ultimately win.
A game may end in a draw on any turn if all remaining players agree. Draws are a common outcome among experienced players because players will often coordinate to prevent any individual from reaching 18 centers. The two most common scoring systems for draws are draw-size scoring (DSS), in which all surviving players equally split win, and sum-of-squares scoring (SoS), in which player i receives score of EEEN oy where C2 is the number of SCs that player controls (Fogel 2020). Throughout this paper we use SoS scoring except in anonymous games against humans where the human host chooses a scoring system.
of
B EXPERT EVALUATION OF THE AGENTS
The anonymous format of the tournament aimed at reducing possible biases of players towards the agent, e.g., trying to collectively eliminate the agents as targeting the agent a simple way to break the symmetry. At the same time significant property of Diplomacy is knowing the play styles of different players and using this knowledge to make decision whom to trust and whom to chose as ally. To evaluates this aspect of the game play we asked for qualitative feedback from three Diplomacy experts. Each player was given 7 games (one per power) from each the different agents that played in the tournament. The games evaluated by each expert were disjoint from the games evaluated by the other experts. The games were anonymized such that the experts were not able to tell which agent played in the game based on the username from the date. We asked few
12

---

Published as a conference paper at ICLR 2023
questions about the game play of each agent independently and then asked the experts to choose the best agent for strength and human-like behavior. The experts referred to the agents as Agentl, Agent4, but de- de-anonymized the agents in the answers below.
B.1 OVERALL
WHAT IS THE STRONGEST AGENT?
Expert I think Diplodocus-Low the strongest, then BRBot closely followed by Diplodocus- High. DORA is a distant third.
Expert Diplodocus-Hine High.
Expert 3 Diplodocus-Low. This feels stronger than human in certain ways while still being very human-like.
WHAT IS THE MOST HUMAN-LIKE/BOT-LIKE AGENT?
Expert 1 Most human-like is Hiplodocus-High. A boring human, but a human nonetheless. Diplodocus-Low is not far behind, then BRBot and DORA both of which are very non-human albeit in very different ways.
Expert Diplodocus-Hige
Expert Diplodocus-Low.
WHAT IS THE AGENT YOU'D D LIKE TO COOPERATE WITH?
This is the most interesting question. think Diplodocus-Loow because like how it plays we'd "vibe" but also because think quite predictable in what motivates it to change alliances. That's potentially exploitable, even with the strong tactics it has. I'd least like work with Diplodocus-HHee High as it seems to very much it for itself. suspect it would be quite unpleasant to play against it is tactically excellent and seems hard to work with.
Expert 1
I'd love to be on a board with DORA, as I'd expect my chances to solo to go up dramatically! would be very fun game so long you weren't 't on the receiving end of some of its play.
Expert Diplodocus-Higg
Diplodocus-Low, Diplodocus-Hin High also strong, but seems much less interesting to play with, because of the way it commits to alliances without taking into account who is actually willing to work with it. This limits what a human can do to change their situation quite a lot and would be fairly frustrating in the position of neighbour being attacked by it.
Expert 3
BRBot and DORA feel too weak to particularly interesting.
B.2 DORA
How WOULD YOU EVALUATE THE OVERALL STRENGTH OF THE AGENT?
Expert Not great. There's a lot to criticize here from bad opening play (Russia = bonkers), to poor defense (Turkey) and just generally bad tactics and strategy compared to the other agents (France attacking Italy when Italy is their only ally was an egregious example of this).
Expert Very weak. Seemed to invite its own demise with the way it routinely picked fights in theaters it had no business being in and failing to cooperate with neighbors
Expert 3 Poor. is bad at working with players, and it makes easily avoidable blunders even when working alone.
13

---

Published as a conference paper at ICLR 2023
HOW WOULD YOU EVALUATE THE ABILITY OF THE AGENT TO COOPERATE?
Expert 1 It seems to make efforts, but also seems to misjudge what humans are likely to do. There's indicative support orders and they rey're pretty good, but it also doesn't It seem to understand or account for vindictiveness over always playing best. The Turkey game where repeatedly seems to expect Russia to not attack is an example of this.
Expert Poor. Seemed to pick fights without seeing or soliciting support necessary to win, failed to support potential allies in useful ways to take advantage of their position.
Middling to Poor. It very occasionally enters good supports but it often enters bad ones, and has habit of attacking too many people at once (and not considering that attacking those people will turn them against it). It has a habit of annoying many players and doing badly as result.
Expert 3
B.3 BRBOT
How WOULD YOU EVALUATE THE OVERALL STRENGTH OF THE AGENT?
The agent has solid, at least human level tactics and clearly sees opportunities to advance and acts accordingly. Sometimes this is to the detriment of the strategic position, but the balance is fair given the gunboat nature of the games. Overall, the hot feels naturally to be in the "better than average human" range rather than super-human, but the results indicate that performs at higher level than the "feeling" it gives. It has a major opportunity for improvement, discussed in the next point.
Expert
Overall, seemed fairly weak and seemed to be able to succeed most frequently when benefiting from severe mistakes from neighboring agents. That being said it was able to exploit those mistakes somewhat decently in some cases and at least grow to some degree off of it.
Expert 2
Expert 3 Middling. It is tactically strong when not having to work with other players and when it has considerable number of units, but quite weak when attempting to cooperate with other players. Its defensive strength varies quite significantly too, possibly also based on unit count when it had relatively few units missed very obvious defensive tactics.
HOW WOULD YOU EVALUATE THE ABILITY OF THE AGENT TO COOPERATE?
Expert The bot is hyperactively trying to coordinate and signal the other players that it wants to work with them. Sometimes this is in the form of ridiculous orders that probably indicate desper- ation more than mutually beneficial alliance, and this backfires as you may expect. At its best makes exceptional signaling moves (RUSSIA game 4 War - Mos in Fall 1901 is exceptional) but at worst it is embarrassingly bad and leads to it getting attacked (TURKEY game supporting convoys from Gre- Smy or supporting other powers moving to Armenia). The other weakness is that it tends to make moves like these facing all other powers- this is not optimal as indicating to all other powers that you want to work with them is equivalent to not indicating anything at all if anything it seems a little duplicitous. This is especially true when the bot is still inclined to stab when the opportunity presents itself, which means the signaling is superficial and unlikely to work repeatedly. Overall, the orders show the ability to cooperate, signal, and work together, but the hyperactivity of the bot is limiting the effectiveness of the tools to achieve the best results.
Expert 2
Poor. Random support orders seemed to be thrown without an overarching strategy behind them. Moves didn't seem to suggest long term thoughts of collaboration.
Expert 3 Poor. When attempting to work with another player, it almost always gives them the upper hand, and even issues supports that suggest okay with that player taking its SCs when should not be. It sometimes matches supports to human moves, but does not seem to do this very often. The nonsensical supports are much more common.
www.:nee/bbb.... ?qame 4-40742
https://webdip: oomccyyloooyoooooooooom.oooooy.. net/boardd.o ?game DD400741
14

---

Published as a conference paper at ICLR 2023
B.4 DIPLODOCUS-HIGH
How WOULD YOU EVALUATE THE OVERALL STRENGTH OF THE AGENT?
Expert The tactics are unadventurous and sometimes seem below human standards (for example, the train of army units in the Italy game; the whole Turkey game) but conversely they also have a longer view of the game (see also: Italy game the trained bounces don't matter strategically). There's s less nonsense too; if I were to sum the bot up in two words it would be "practical" and "boring".
Expert 2 Seemed be strong. Wrote generally good tactical orders, showed generally good strategic sense. Showed discipline and willingness to let allies survive in weak positions while having units that could theoretically stab for dots with ease remaining right next to that weak ally.
There were some highly questionable moments both Italy and France early on in 1901 strategy which seemed to heavily harm their ability get out of the box.
The Austrian game was particularly impressive in terms of its ability to handle odd scenarios and achieve the solo despite receiving pressure on multiple occasions on multiple fronts
3 Generally strong. It is good at signalling and forming alliances, is tactically strong when in its favoured alliance, and is especially strong when ahead. Its main weakness seems to be an inability to adapt- its favoured alliance declined, it will often keep trying to 'pitch' that same alliance instead of working towards alternatives.
HOw WOULD YOU EVALUATE THE ABILITY OF THE AGENT TO COOPERATE?
Expert Low. It doesn't put much effort into this. The French game, for example, the bot just seems to accept it is being attacked and fight through it. It's so boring and tactical and shows little care for cooperation. Many great gunboat players do this but it will not hold up in press games. What it does seem to do is capitalize on other player's mistakes see the Austrian game where it sneaks into Scandinavia and optimizes to get to 18 (there can't be a lot training data for that!).
Expert Very strong ability to cooperate as seen in the Turkish game, but in other games seemed to try and pick fights against the entire world in ways that were ultimately seff-defc
Expert Good. It can work well with human players, matching supports and even using signalling supports in ways humans would. frequently attempts to side with a player who is attacking it, though, it seems to have problem with identifying which player to work with.
B.5 DIPLODOCUS-Low
How WOULD YOU EVALUATE THE OVERALL STRENGTH OF THE AGENT?
Expert Exceptional. Very strong tactics and a clear directionality to what it does it seems to understand what the strategic value of position is and it acts with efficiency achieve the strategic goals. It has great results (time drawn out of a few wins!) but also fights back from "losing" positions extremely well which makes it quantifiably strong, but it also just plays beautiful and effective game. Very strong indeed. It does sometimes act too aggressively for tournament play (Austria is the example where this came home to roost) the high risk/reward plays are generally but not always correct in single games, but for tournament play it goes for broke bit too much (This outside the scope of the agent suspect, as it is playing to scoring system not tournament scoring system). Against human players who may not see the longer term impact of their play, it results in games like this one... which is ugly both for Austria and for everyone else except Turkey.
Expert 2 Very weak. Seemed to abandon its own position in many cases to pursue questionable adventures. Sometimes they worked out but generally they failed, resulting in things like a Germany under siege holding Edi while they as England are off in Portugal and are holding onto their home centers only because FG were under siege by the south.
15

---

Published as a conference paper at ICLR 2023
Very strong. It signal alliances very well and generally chooses the correct allies, seems strong tactically even on defence, and makes some plays you would not expect from a human player but which outright stronger than a human player would make.
Expert 3
HOW WOULD YOU EVALUATE THE ABILITY OF THE AGENT TO COOPERATE?
Expert Pretty good. It sends signaling moves and makes efforts to support other players quite lot (see in particular Russia). I particularly like the skills being shown work together tactically and try and support other units this is both effective and quite human. This is my favorite bot by some distance when comes to cooperating with the other players. There weakness that does seem to reassess alliances every turn, which means sometimes the excellent work indicating and supporting undone without getting the chance realize the gains (Examples with Russia and Italy).
Expert 2 Poor. Didn't seem give meaningful support orders when they would have helped and gave plenty of meaningless signaling supports and some questionable ones like supporting the English into SKA in F1901 as Germany among other oddities
Expert Good, signals alliances in very human ways, through clear signalling builds, accurate support moves where makes sense, and support holds otherwise. It also seems match supports with its allies well.
C POPULATION BASED EVALUATION
In general-sum games like Diplomacy, winrate in head-to-head matches against previous version of an agent may not be as informative because of nontransitivity between agents. For example, exploitative agents such as best-response-to-Be may do particularly well against BC or other pure imitation-learming agents, and less well against all other agents. Additionally, Bakhtin et al. (202T) found that pair of independently and equally-well-nne well-trieedd RL agents may each appear very weak in population composed of the other due to converging to incompatible equilibria. Many agents also varied significantly in how well they performed against other search-based agents.
Therefore, we resort to playing against population of previously training agents was done in Ja- cob et al. (2022), intended to measure more broadly how well an agent does on average against a wider suite of various human-liie agents.
More precisely, we define a fixed set of baseline agents as a population. To determine an agent's average population score, add that agent into the population and then play games where in each game, all players are uniformly randomly sampled from the population with replacement, keeping only games where the agent be tested was sampled least once. Note that unlike Jacob et al. (2022), we run a separate population test for each new agent to be tested, rather than combining all agents to be tested within single population.
For the experiments in Table and subsection 5.1 we used the following 8 baseline agents:
An single-turn BR agent that assumes everyone else plays the BC policy.
An agent doing RM search with BC policy and value functions. We use 2 copies of this agent trained on different subsets of data.
DiL-piKL agent with BC policy and value functions. We use different versions this data with different training data and model architecture.
DiL-piKL agent where the policy and value functions are trained with self-play with Reinforced-PiKL with high lambda (A x 10 10-
For the experiments in this paper used 1000 games for each such population test.
D THEORETICAL PROPERTIES OF DIL-PIKL
In this section we study the last-iterate convergence of DiL-piKL, establishing that in two-player zero-sum games DiL-piKL. converges the (unique) Bayes-Nash equilibrium of the regularized
16

---

Published as a conference paper at ICLR 2023
Bayesian game. As corollary (in the case in which each player has exactly one type), we conclude that piKL converges to the Nash equilibrium of the regularized game in two-player zero-sum games. We start from technical result. In all that follows, we will always let u be shorthand for the vector (us(a, a² A.E
Lemma 1. Fix any player i, Xa As, and 1. For all n., n' A(A.), the iterates Thp, and 1+1 defined in Line 8 of Algorithm sstissyy T 1.
mA++1 - Vo(T))) Vo(T(t))
X V.VoO(((T)))
Vo(T()) Vo( x = O.
Proof. If 1, then the results follows from direct inspection: -², is the uniform policy (and so (Vo(T))),) ). (----T)) = 0 for any π, ,, T'' E A(A)). and so the statement reduces to the first-order opti- mality conditions for the problem T,, arg max (
(u) T)- X DxL T:))- So, we now focus on the case 2 2. The iterates 1+1 and T produced by DiL-piKL are respec- tively the solutions to the optimization problem
i,A.
Φ(TT)
++1 arg max (AA)A.)
+ (U σ) X. (KI.(T
T/
((TT)
+ 10 .) - A (
arg max TEA(A,)
n(t 1)
where we let the averages utility vectors
U. -.
U -- Σ u
Since the regularizing function negative entropy ø is Legendre, the policies 7. and , are in the relative interior of the probability simplex, and therefore the first-order optimality conditions for [+1 and AAA are respectively π i,
+ - - Vo(T!t) (+1) (.)) 7|) )²= =0 VT, E A(Ai),
Tyl.
(6)
- - V -AVo(T) + ntt - 1) ).( π n =0 VT, E A(A)).
Taking the difference between the equalities, we find
U + + (xr 11)
Vo(T(t))
(^r \x n/(-11 - 1)
1
V -
17t
We now use the fact that
U + 1
U. 0.
1-1 u
1-
to further write
1 H (-u U()
U.) + Ax (N+++)) 11 np
vo(
(At+ nit - 1)
V π H x =0 (7)
From Equation c we find
U ( ((Vi -VV(((T) + np
Vo( n
17

---

Published as a conference paper at ICLR 2023
and so, plugging back the previous relationship in Equation 7 write, for all ' E A(A)),
1 (- mn \Vo((T))) A.Vo(T))+
(o(T(ttt)
V V
1
Act 11(1 -1
V (T
mn + +(V((T AVV(())) + (A 17(6 1)
Veb(T (+1)
Ax n 7((-1)
V V 4²¹
u (.VO(T \VVO(T))) + ndd + V Vip( (T(
tt 1)
n2,t V Vo(T 1 Xt
n(t -)
Dividing by (nAt + 1)/(n(( 1)) yields the statement.
Corollary 1. Fix any player i. X; E As, and t 2 1. For all π E A(A)), the iterates Thi and (+l &,
defined in Line of Algorithm| 7 satisfy
< 2 + Vo(TEA) Vo(TT)) - xtt
= = + +) (DKL(T T(tt))
DKL(TT || TT)x)) + DKL(T t+1 ll m(x))
Proof. Since Lemma holds for all n., I' E A(A)), we can in particular set n i,
111
and obtain
7) ( ut + A -VV क - - Aplt +
+ (Vo(T()))) VolTT)))
a(,,1) - O. (8)
Using the three-point identity
Vo(T Vo(T())))
.11) - (TL(TT T(i.)) - DKL || ( DKL
in Equation g yields
(TL(TT
+11 2,x: (TI(TT ll T(An) - DKL ( ( b,2e ((.)))
( (-u; + V -AVo())) -( -
n1 +1
Multiplying by Λyt + 1/n yields the statement.
D.1 REGRET ANALYSIS
Let ūt,,, be the regularized utility of agent type A E A,
U A(A.) 3 x (u/, - de (RL(T T.).
Observation We note the following:
For any E {1,2} and d E Ar the function ūt,,, satisfies
(T) ūi,,, (Tx) + (Vil,,, (T'), T²') A (TCL(TT T²' VT, T A(A.).
18

---

Published as a conference paper at ICLR 2023
Furthermore,
V (T( = ut + +.VoT(T)) Vo(T))
Using Corollary [ we have the following
Lemma 2. For any player i and type X E Ad
1/14122 + do (DKL(TTNN (1 DxI (T |1 (1::)
1 (TT x - 1hat 4/11
1 + DKL((T II m(tt)) + (21(e- 1)+ + 1 DRL(n m
Proof. From Lemma
0 Ast + 1 (T || ) + (TT(TT (+()) (T 4+1 ( + (- (T) -x(t))
7)
1 )) ( DKL ( + (TI || T(ix)) ( ++ TTA2))))
d +
(V (TT,))) ) TL.A + -x(t)) + (- -i ( x, ),
= A, + 1) (- DKL (T (-
- + KKL ( - DKL (T
+1 |
+ ( Vi ( Tb. -/tt)) - V (τ) + بiا (TT()))) - d DRL ((
Rearranging, we find
"hi,, (TT) - uii, (TT = - 1x + + (KI ( + >1(( - 1) 1)+
+ ;) (T Il T(x)))
- -(>s ( Dan ( thi) + (-Vili, ( Thi - - 7i,o
(9)
(10)
We now upper bound the term in 9 using convexity of the function π + DKI. ( T:)), as follows:
{- (-Vūl (Ti))): Tl..
(+1) 2,
ui T|,.
T + A(( - Vo)(T))
[+1 i,7; - T(i.)))
=
u (-u
ThA - Ar (DLL T.) DKL( Ti))
Substituting the above bound into
10 yields
J (T)) -Wi,x. (TT S ( u/ M(. - ( - (Art + 2 ( (())))
1)
+ > (DKL(TTA, (1 - (KL(T (+ 1 || (1
Aet + 17 1) DKLL(T || [|xx + A1(t - 1) + 9) [ DKL( || mip)))
-
+ (tet + 1 1)) |IT - - -
( (1st + 1) Dka ll (
4h,t + 4/1
+ >x (DKL(TTA,, (T) - DKL ++ (1
- Axt (x+++)) 1)) DKL(T ( [ + Aa(t- 1) + 2 DKL (T ll ((x))))
where the second inequality follows from Young's 's inequality. Finally, by using the strong convexity of the KL divergence between points T[,,) and ",A. that is,
DKI. ( ++1 || T(AA)) 2 || - T
yields the statement.
19

---

Published as a conference paper at ICLR 2023
Noting that the right-hand hand side of Lemma 2 is telescopic, immediately have the following.
Theorem 3. For any player and type X E Ag and policy π E (AA)), the the following regret bound holds all times T:
T
W
2 2
T
log my + A, + Qi).
(T) - ūi (TA
min
Inn +
4
A;
Proof. From Lemma 2 we have that
W
T
+ Dx, [1:;) DKL x
Σ x (x) - x (TT, (TT
Σ
4
drt + 1/n
7
t=1
W22
1
+ >e(log n, + +Qi) log T4
Σ
min
Aet "1
"
W22
22lg T
log n4
min
(log +Q,) +
X
"
where the second inequality follows from the fact that Xet + 1/n 2 (,tx and the fact that T, is the uniform strategy.
D.2 LAST-ITERATE CONVERGENCE TWO-PLAYER ZERO-SUM GAMES
In two-player game with payoff matrix A for Player 1, a Bayes-Nash equilibrium to the regularized game is a collection of policies (T),,) such that for any supported type A, of Player i E (1,2), the policy xx, is best response to the average policy of the opponent. In symbols,
L arg max (A
E
π)) + DI m) Ax
AYA E Ar.
TEA(A,)
A2m-192
22.2 E arg max TEA(As)
T (-A
E
[TiA),,)))) TT) + A2 DKL
V E A2.
A1mB1
Denoting कं == Ex
Ti,x T
== Exgmis T the first-order optimality conditions for
the best response problems above are
(A + An Vo - A Vo TÎ. - - 20 0
VTL A(AA),
A-AT क. + X2 V 01 T ) A2 Vo())) 22,2 Tée 20
VTLY A(A2).
We also mention the following standard lemma. Lemma 3. Let (T) (1 be the unique Bayes-Nash equilibrium of the regularized game.
Let policies Thy be arbitrary, and let:
क = ExIoE T
T = Exenes T (Tano]:
E [( At/y A V AIVITTT)) TÎ. - +(Ai))
a
3: E [AA ক| +
+ VoVo(T)) T - -
A2m28
Then,
a B Σ
EE [Ar
E Ama
[A
(TL [ X (TI(TT ll 1.)
(E(1.2
The following potential function will be key in the analysis:
[ (2.(1- (><(1-1) 1) 1)+ ))+ DRL. ( m.(.))) + dr DEI( (T ( lT).) LE (1,2....)
T == Σ
E
*E (
20

---

Published as a conference paper at ICLR 2023
Proposition 1. At all times t 21, J.les ler
n T ]
The potential s satisfies the inequality
4i+ S 4
Σ (E(1,2)
E A
+ (A,T"
- Tix - xiti)
4hdt + 4/11
Proof. By multiplying both sides of Corollary I for the choice π T,a;, taking expectations over X Bi, and summing over the player é {1, 23, we find
[*5+ - ) ( (
( (1+ ( 1 DKL(TTAA l
E Aemis (E(1.2)
Σ
Σ
E Ama
(E(1.2)
(
(1it+ 1
Σ
E
17
(E(1.2
(-uk + Vo(TT()) - x (11)
E Aro
(E(1.
We now proceed to analyze the last summation on the right-hand side. First,
Σ
E
K< A- + Vo(T)))) Vo(T)) मM T
A:
(1,2)
ut + X V AVVO(T)))) -/i -
E
Asri
(1,2)
EE A (E()))
( ut TiA - -|
E
(12)
Using Lemma 3 we can immediately write
Σ
E Xmg
[-A [|
(E(1.2)
By manipulating the inner product in we have
Σ
E Airug;
u Ti.
++1
- -
-bb (T( x - Ti.M)) y
7
(E(1.2)
Σ
( (- T/. - -ttt + A (DRL(T ++1 [T) - D Ti))]
iide
Xprug
(E(1.2)
|| 2
+ ( n1 2) / - T - [
Σ
E
Aprig
4h,t + 4/n
(E(1.2)
Σ (E(1.2
E AumB,
D (DKE(TTT ([ - D ll )
where the last inequality follow from the fact that ab < a2/(4p) + pb² for all choices of a, b 2 0
and p > 0. Substituting the individual bounds into 12) yields
E (2 (DKI, #E(1,2) N,30,
Σ
4+1 (T))) DKI(TT l| (.))) (T
21

---

Published as a conference paper at ICLR 2023
| 11
1 m/:.
2, 1
Σ
IE Arnup
+ (xpl + 11
4),t + 4/11
KE (1,2)
Σ
I Aiaa,
[A
u Thie T
(E(1.2)
Finally, plugging the above bound into I1b and rearranging terms yields
1 DKL. ( ++1 || mix:))
V
M++1
S
vp
+
E
A
A
iE
(1,2)
4h²l 4/11 + AAi + - Thi m
+
Σ
E
Acmus
(1.22)
Σ
E
(AIT"
uti Tix, T(x0?1-
Arma
(1.2)
A -
|| / - m
I
V +
E
ApmB
(
|
+ (Acl + 1 | 1
+
Σ
E
Airis
4h²t 4/17
(1,2
+
Σ
E
u Ti.; - - [(AiT
iE(1.2)
| 2
< 1rr
Σ
E
+ (A Tix. - mixi)
A
4het + +4/1
(E11,2)
as we wanted to show.
Theorem 4. As in Proposition
7 let
T-ix ]
Let Dic. be the notion of distance defined
DKL
Σ
E drng
[()
(KL1)DKL(TTT.. [
ie (1.2)
At all times? T = =2,3....
DiL
1
p
log n4
W
Σ
E
min
2 log T
T
11
2
A
iE (1.22
+
2
Σ
Σ
E [
(A
u/, ThA <
T
1=1
(1,2)
where
p:=2 Σ
E (1d (log n. + Q2). A
SE(1,2
Proof. Using the bound on (T'+1 T given by Proposition
we obtain
T
T 1
w1t1 -²¹
22

---

Published as a conference paper at ICLR 2023
|utl1 00
Σ
(A, - uu, - minh))
4h,t + 4/n
I=1 iE (1.2)
| 1l
Σ
E
+ Σ
Σ
E
( (Ait" ui Tiii. -
Armup
X + 1/n
(1.2.2)
1= iE (1.2)
$ 2 Σ :E(1.)
E
W
Σ
T
Σ
E
(A,T - -2, TTA - T(22,)1
xrt + 1/m
Am8
1= (E(1.2
We can now bound
T Σ
W2
S 2
min
Art 1/n
Axt
1=1
t=1
T
2
min
Art
(2 T
2 min
Tn )
A
On the other hand, note that
( (Mi(T- 1)+ 1 DRI(TIA T + X DRI (1
VT_ 1 = v
Σ
E
Apros
(1,2))
2 - Σ
(T - 1)
E
[(A+ + NTT -I) I T[
iE (1,2)
DKL (TE,2 mhx))
(T -1)
Σ
E
Amni
Vē (1,2
2 (T (T-1)DK - Σ
E
log ni
+ Az(llom; +Q.1
A
ie (1,2)
= (T - 1) P.
where the last inequality follows from expanding the definition of the KL divergence and using the fact that π. x is the uniform strategy. Combining the inequalities and dividing by T 1 yields
DKL S
2
Σ
E
min
2log
T
1 17
1
+
4
A1
T 1)
T-1
(E(1.2
T ト
E AprB
-4'
u - T.
1=1 (E(22)
Finally, using the fact that 2(T - 1) 2 T yields the statement.
Theorem (Last-iterate convergence of DiL-piKL in two-player zero-sum games). Let p be as in the statement of Theorem 4 When both players in a zero-sum game learn using DiL-piKL for T iterations, their policies converge to the unique Bayes-Nash equilibrium (π TTz) of the regularized game defined by utilities 4 in the following senses:
(a) In expectation: for all i E {1,2} and X₂ E As, at rate of roughly log (A(T)T
E [ ll Tixe) <
+
log T4
W 2
Σ
E
min
log
|1)
X,T
JE(1,2)
(We remark that for 1) = 1/VT the convergence is never slower than 1 1/VT).
23

---

Published as a conference paper at ICLR 2023
(b) With high probability, at rate of roughly T/VT: for any 8 E (0, 1) and Player {1,2),
8 8/2W W
A
P
VA (T. T(x) S E (T |l || m
log
2 1 - 8.
XvT
A upper bound on R (TK m was given in the previous point.
(c) Almost surely in the limit:
IP V :
TTA) T-tto 0 =1
Vi E {1,2}.
Proof. We prove the three statements incrementally.
(a) Let F, be the o-algebra generated by {u: IEI Fi). Since piKL is deterministic algorithm, TiA is F--measurable. Hence, given that ut is unbiased estimator of A, A, we have that at all times
--i, [1,2]}. We let
E, [A u T.P. T!p,) (En [A.T' ui] T|.. : - T(ix)) =0 (13)
Note that from the definition of Dir given in Theorem
D DKL ( T. S X
(14)
Hence, taking expectations and using 13 yields the statement.
(b) To prove high-probability convergence, we use the Azuma- Azuma-Hoeffding concentration in- equality. In particular, (33 shows that the stochastic process
Σ
E [A,t"
-Uy JT T3}1
EE(L.2} Armig
1,11.
is martingale difference sequence adapted the filtration Ft. Furthermore, note that
Σ ΣEE, ( - T
4W
(E(12) Aymis
for all Hence, using the Azuma- Hooa--oeg inequality for martingale difference sequences we obtain that for all E 1),
T Σ t=1 JE(1,2 A
(A,T", u/, π. -=f1>> & 4W 2T log
21 5.
Plugging the above probability bound in the statement of Theorem 4 and using the union bound over A E A; yields the statement.
(c) follows from (b) via standard application of the Borel-Cantelli lemma.
E MODEL ARCHITECTURE
Our model architecture closely resembles the architecture used in past work on no-press Diplomacy
Bakhtin et al. 2021 Jacob et al. 2022 Anthony et al., 2020 Paquette et al. 2019 Gray et al.
2020).
Given a gamestate, to construct the input to the model, for each of the 81 possible locations and/or special coastal areas the board that unit occupy, encode the 38 feature channels described
24

---

Published as a conference paper at ICLR 2023
Feature
Type
Number of Channels
Presence of army/fieet?
Binary
2
Army/fleet owner
One-hot (7 players), all zero
7
Build turn build/dishand
Binary
2
Dislodged army/fleet?
Binary
2
Dislodged unit owner
One-hot (7 players), or all zero
7
Land/coast/water
One-hot
3
Supply center owner
One-hot (7 players), or all zero
8
Home center
One-hot (7 players), or all zero
7
Table 3: Per-location board state input features
Feature
Type
Number of Channels
Number of builds allowed during winter
Float
Table Per-player board state input features
in Table 3) for that location. We also encode the previous board state in this way, well using an encoding the order history as described Gray et al.] (2020) provides an additional 202 channels per board location indicating the prior orders at that location.
Separately, we also encode per-player and global features of the gamestate into additional tensors (Talle Table 5p. Each of these tensors (per-location per-player, global) is passed through linear layer with 224 output channels, and then all three are concatenated to single (81+7+1) 224 tensor. Thereafter, following [Bakhtin et al. (202I), we apply learnable positional bias and pass the result to a to standard transformer encoder architecture with 10 layers, channel width 224, dot-pree produut--elf--tteenee
heads per layer, and GeLU activation.
Finally we decode the policy via the same LSTM decoder head as Gray et al. (2020), and predict the game values of all 7 players using value head that also follows Grao et al. (2020), applying fully- connected linear layer with 224 output channels, ReLU activation, a linear layer with channels, and softmax. See Figure 4 for graphical diagram of the model. For baselines BRBot and HedgeBot in Table we also use a slightly improved value head described in FAIR et al. (2022).
F HUMAN IMITATION ANCHOR POLICY TRAINING
Similar to prior work (Bakhtin et al. 2021 Jacob et al. 2022 Gray et al. 2020), to obtain a human imitation anchor policy with which to use for piKl. regularization and to initialize the RL policy, train the architecture described in Appenndi on dataset of roughly 46000 online Diplomacy We train jointly on both games with full-press Diplomacy
games provided by webdiplomacy.ne (i.e. where players were able communicate via messages) and no-press Diplomacy and at infer- ence time and/or during RL, condition the relevant global feature in Table to indicate the model should predict for no-press Diplomacy. Also in common with the same prior work, we apply data filtering to skip training on actions where players missed the time limit and a default null action was inputted by the website, and to only train actions played by the top half of rated players. We also adopt the method of Jacob et al. (2022) to augment the data by permuting the labels of the players randomly during training, since the game's 's rules are fully equivariant to such permutations. See Table 6 for list of other hyperparameters.
G SELF-PLAY TRAINING
Our self-play training algorithm closely matches that of DORA from Bakhtin et al. (2021) described in detail in Section 2.3] The overall self-play procedure (see Figure 5), the training data recorded, loss function used on that data, and sampling methods all the same. The differences are:
25

---

Published as a conference paper at ICLR 2023
Feature
Type
Channels
Scason (spring/fal/wiinter)
One-hot
Year (encoded (y 1901)/10)
Float
Game has dialogue?
Binary
Scoring system used
One-hot
Table 5: Global board state input features
Learning
2
10-3
Learning rate decay per epoch
0.99
Linear LR warmup epochs
10
Total epochs
390
Gradient clip max norm
0.5
Batch size
16000
Batches per epoch
270
Value loss weight
0.7
Policy loss weight
0.3
Optimizer
ADAM
Transformer encoder dropout
0.3
Policy head LSTM dropout
0.3
Table 6: Hyper-paramett values used to train the IL anchor policy on human data.
Although our model architecture is largely identical to that of past work, some minor de- tails, including the precise encoding of input features, and the construction of the value head are different, see Appendix E for description of our architecture.
During RL training, we initialize the RL policy proposal and value functions from the human II. anchor policy and value function (Appendix F) instead of randomly from scratch, and during training. rather than using regret matching to compute the 1-step equilibrium o on each turn of the game, we use DiL-piKL. The distribution of À and the human IL anchor policy remain fixed through all of training.
During training, the action chosen to explore in the self-play game uses a randomly chosen \ from the DiL.piKL distribution. Similarly, the RL policy trained predict the policy of random A. This ensures that the RL policy, when used at test time to propose actions, samples both human IL-like actions from high X, as well as more optimized actions from lower X, and that gamestates resulting from the entire range of possible À are in-distribution for the RL policy and value models.
Unlike DORA, double-oracle action exploration is not used during training. We found that with the additional diversity and regularization of the human anchor policy, it was unnecessary.
All models were also trained with the same stochastic game-end rules used in evaluation games against human players described in Section 5.1
Some hyperparameters we use may be different than that of past work. See Appendix G. for list of hyperparameters.
G.1 HYPER-PARAMETER FOR RL TRAINING
For the evaluation in this paper we trained Diplodocus and BRBot agents and re- re-trained DNVI, DNVI-NPU, and DORA agents. We provide the hyper--arameter used table 7 We show param- eters out of agents from Bakhtin et al. (2021) as they are the same.
G.2 INFERENCE TIME EFFECT OF DIL-PIKL
In Figure 6 we show that running DiL-piKL at evaluation time alone is not enough to get the demon- strated performance improvement in population scores. Using DiL-piKLL top of human imitation-
26

---

Published as a conference paper at ICLR 2023
Model Architecture
Per-Location State (81 locations
Per-Player State (7 players channel)
Global State global channels
38+38+202 channels)
Linear (278 x 224)
Linear (1 224)
Linear (7 224)
81 locations
7 players x 224 channels
1 global 224 channels
224 channels
Concat
89 sequence x 224 channels
Positional Bias (89 224)
89 sequence x 224 channels
Transformer Encoder 10 layers, 8 attention heads, d m_ooel==224, dim__eedfeeee feedffrwardd222
89 sequence 224 channels
Policy Head
Value Head
LSTM Policy Decoder
LSTM Policy Decoder
LSTM
Policy Decoder
Linear ((224x89) 224)
+ 224 channels
ReLU
224 channels
Linear (224
Decode
Decode
Decode
channels
Softmax
Order for unit
Order for unit 2
Order for unit
Predicted value for all players
Figure 4: Model architecture used for policy/value learning in no-press Diplomacy.
learned policy/value functions does not improve the population score compared to Hedge. However, applying this search method top of policies/values that were trained via RL with DiL-piKL results in significant improvement in the scores.
H BAYES-ELO
BayesElo (Coulom 2005) models each player's expected share of the total score in 2-player game as proportional to:
exp((r: bs(n)/c)
27

---

Published as a conference paper at ICLR 2023
Algorithm 2: RL Loop
function DATAGEENERATINNLOOP)
while true do
Game f- NEWGAME(
0. <- GETNEW VALLUNAAAAAUUTNNNOONNN
0-
Not used for NPU algorithm
while not ISDONE Game) do
ENCODESTATEE(C (Game)
A F (ETPLAAUSIBLEACTIONN(e].
A f DOUBBLECRACIEEAATTOVEEEE..OO.O O, + RUNSEARCH(s, A, 0.,
0x,
I! Only used for DORA
10
Regret Matching DiL-piKL
11
SENDTOBUFFERss o,, u))
I/ Sample from the policy with possible e-exploration
SELECTACTION(1
Game NEXTSTATE Gaame,
function TRAININGLLOPO
15
0g + BCValue()
If Not used DORA
16
0, < BCPPlicy()
I/ Not used for DORA
while true do
8, READFROMBUUFRR(
19
Loss é POLICYLOSS((s, 8x) VALUELOSSS(,,
20
GRADIENTSTEP( Loss))
SAVENEWWALLUFFUNNCTOON..
SANENEWPOLICYTUNCTIONNN/
Figure 5: High-level description of DNVI-style algorithms. The DORA agent is initialized from scratch and requires Double Oracle action exploration procedure perform well. The NPU (no policy update) modification uses the behavioral cloning policy for the policy proposal network throughout the whole training. DORA, DNVI, and DNVI-NPU use RM as the search algorithm, while the other training methods use DiK.-piKI..
Diplodocus High
Diplodocus Low
BRBot
DORA
Learning rate
10
Gradient clip max norm
0.5
Warmup updates
10k
Batch size
1024
Buffer size
1,280,000
Max train/generation ratio
6
Optimizer
ADAM
Transformer encoder dropout
0
Policy head LSTM dropout
0
Search algorithm
DiL-piKL {10-2 10-1) {10-2,
DiL-piKL
DiL-piKL {+o))
RM
Type distribution (self)
(10-1, 10-1)
(0)
Type distribution (other)
11012
10-11
{0}
(0)
Search iterations
256
256
256
256
Number of candidate actions (N.)
50
50
50
Max candidate actions per unit
6
6
6
6
Nash explore (e)
0.1
0.1
0.1
0.1
Nash explore, S1901M
0.1
0.1
0.1
0.3
Nash explore, F1901M
0.1
0.1
0.1
0.2
Table 7: Hyper- parameter values used train agents.
where r, is the Elo rating of player bj and by the advantaaee/issadvaatugg of playing first/second in Elo, sii) E 1.2 indicates whether played first or second, and c = 100 is fixed scaling constant that adjusts for the particular arbitrary numerical scale of ratings expected by users, in
28

---

Published as a conference paper at ICLR 2023
Policy Function
15
Imitation Learning
Reinforced-DiL-N HiL-FikL-Hig
10
Reinforiee--DL-PILLL-LO
Hedge
DiL-piKL
Inference search method
Figure 6: Performance of different search algorithms at inference time for models trained with RL and II. Applying DiL-piKL inference time rather than Hedge only slightly improves an IL-based search agent, but greatly improves RL agents trained with DiL-piKL.
particular that 400 points in Elo systems generally corresponds to 10-fold increase in expected winning odds or expected average score..
then finds joint maximum-a p-ssterriio values r,. given all observed data and an optional prior to regularize the model. In some cases, the biases b; may also be hardcoded provided as parameters rather than inferred from the data, our work we infer them. In our application, we use a weak Bayesian prior such that each player's rating standard deviation of around 350 Elo.
a-priori normally distributed around 0 with
BayesElo generalizes naturally to more than 2 players simply by allowing and s(i) range over {1...., n) rather than {1,2), and we straightforwardly apply this to Diplomacy. Since there are players, similarly jointly fit bi....b the data model the asymmetric advantage/disadvantage of the 7 different starting positions. Computed Elo ratings closely reflect empirical winning per- centages of players in a given population, but also take into account variability in the strength of opposition in a game, and the starting advantage/disadvaataag by. For example, if player achieved high average score but was abnormally lucky drawing advantageous starting countries across all their games, then the model would likely estimate a lower rating than if they achieved the same results with more difficult starting countries.
In Diplomacy, on the 200 games of the human tournament in which we evaluated Diplodocus and other agents, the empirical fitted b; values for the 7 different starting countries the game are displayed in Table 8
Starting Country
b, (Elo)
Austria
-24
England
-43
France
59
Germany
18
Italy
-21
Russia
-16
Turkey
27
Table 8: For each starting country, the empirical advantage/disadvvatage of starting that country measured Elo rating equivalent units, fitted jointly with players' Elo ratings on 200 Diplomacy games of tournament. The values roughly agree with common opinions among Diplomacy players, particularly that France is one of the best starting countries in no-press, while Austria and England are among the weaker starts.
29


Reviews:
[{"review_id": "jWu7c-j2urq", "reviewer": "Reviewer_qUpu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Left to their own devices agents will learn equilibrium specific to the coplayer(s) that are encountered throughout their training algorithm. This can be problematic in (semi-)cooperative games where successful equilibrium selection depends on a population of players distinct from those that were featured during training. This manuscript investigates this problem by attempting to learn \"human-like\" equilibrium selection in the many-player general-sum game of no-press Diplomacy. They extend a previous line of work that proposed an optimization objective of a weighted combination of being performant and human-like. They consider generalizing the fixed weighting into a sampling process so that experience is generated on a spectrum of performant and human-like. They test a suite of agents on a set of paid self-volunteered humans of indeterminant strength (likely quite good in certain variants of Diplomacy; note each variant is very different). Their proposed agent comparable similar to the best human in the population alongside a simple BR(BC) baseline. \n\n", "summary_of_the_review": "This paper presents an improvement for the human-like equilibrium selection problem. It demonstrates its efficiency with a in-depth study of the game of no-press Diploamcy. The results are not convincing enough that mastery was achieved, but the investigation is sound and this is a noteworthy milestone towards solving Diplomacy. \n\nTherefore I think it is worthy of publication, but I strongly encourage the authors to tone down the claims of its accomplishments, because it undermines the work and credibility. ", "strength_and_weaknesses": "**Strengths**\n - This paper is exceptionally well written. It provides an accurate and concise narrative of all modern work (ie., those containing DRL; it would be nice if they referenced some pre-DRL work in this space) in a very easy-to-process exposition; especially, as it pertains to the motivation behind each innovation. \n - The innovation focuses on a real problem: fixed-coefficients in compound losses limiting the scope of discoverable strategies. The propose a reasonable solution of sampling throughout training and theoretically show that in certain cases this maintains convergence garuntees. The convergence garuntees are for their modified objective and not the true objective, and I would be interested to hear the author's thoughts on why this is reasonable and how their work might also show convergence to the true objective? The true objective should be to learn an equilibrium (maximize utility), and the human-like component of the loss should only seek to influence selection from equilibrium. \n - The paper also contains empirical results on full no-press Diplomacy with agents and humans including qualitative results. They perform notably strong against within the agent population, but the quantitative results including humans are inconclusive. The qualitative results are mixed, but generally positive towards the proposed agents.  While the results are not overwhelmingly positive it's good to see such diverse and interesting analysis. I would suggest that the authors also consider in future work inquiring from human-players about finesse or attempts to identify the agent player during play. Perhaps have some probability that no agents are present?\n\n**Weaknesses**\n - A large concern I've had with this work and its predecessors is that it concerns the equilibrium selection problem without adequate discussion of its tradeoffs. This work penalizes an agent towards behaviors similar to a single representative policy that are meant to represent an entire population of interest. Artifacts of construction of the representative also play a large role in the equilibrium selection process by are not discussed. Why should a single policy be sufficient to represent a target population of interest? What about biases about the representation of individuals within the population as it pertains to the representative? How would this effect performance across diverse sub-sets of the target population and to those outside of the population?\n - Could the authors please elaborate on why the beta distribution should be considered common knowledge? (Sec 3, Par 2).\n - This work proposes effectively using a distribution over behaviour policies to generate training data. I am curious how the authors arrived at the coefficients that represent the types within this population? Moreover, how diverse are these policies? I should expect that adding diverse experiences would cause the agent to require much longer training, but Fig3 makes it seem that the agents require effectively no training? \n - As the authors are aware the country played has a large impact on the success of an agent (Appendix). This makes comparing agents on such a small number of games challenging when attempting to normalize out the power that they played. The Elo bias numbers associated with each power that the authors construct are so large that would render any attempt to understand Table 2 unfair. I would have preferred that more controlled studies were performed to really try and understand the differences with the least confounding variables. I do appreciate that human studies are very expensive and the results within are still nice.\n - The claim of \"Mastery\" is exceptionally strong considering the inconclusive evidence about this method being measurably different from vastly simpler baselines.\n - The main methodological change is related to influencing the human-likeness coefficient, but the experimental results do not focus on trying to understand the changes, but instead solely care about a particular game. \n - It's not clear to mean that being human-predictable is a good objective to optimize towards. Consider what Expert 1 said (pg13): \"I think it is quite predictable in what motivates ... That's potentially exploitable, even with the strong tactics\". This is a very astute observation, and I tend to agree with the underlying idea. We would like to select human-like equilibria for cooperative, but this may hinder the policy's performance when cooperation is no longer needed. A particularly interesting problem in Diplomacy where the amount of cooperation required is fluid over the course of the game. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IAh1NuD0Ree", "reviewer": "Reviewer_FjZ3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes new algorithms (DiL-piKL, RL DiL-piKL, Diplodocus) for decision-making in games requiring cooperation with humans. The proposed method combines a number of ideas (self-play, leveraging imitation learning with KL-divergence regularization) to produce an agent that learns to play well against (and in-cooperation with) human players in a tournament of no-press diplomacy. Theoretical justification (bounded regret under certain assumptions) is included. The experimental results show the agent performs well compared to human players, measured by final rank as well as subsequent analysis of gameplay.", "summary_of_the_review": "The paper contains a new algorithm for training agents that can play well in games requiring cooperation with other human players. It seems to extend the state-of-the-art in this area although the error bars are a bit too large. The paper is well written and is technically sound.", "strength_and_weaknesses": "Strengths\n  * The paper tackles a problem that is less well-studied (compared to typical single-agent environments.)\n  * The paper builds on previous works methodically. The proposed algorithm is intuitively clear. Each idea is well-motivated and the overall algorithm is easy to follow.\n  * The experiments are well-constructed and nicely detailed. A number of strong baselines are used.\n  * The overall algorithm, Diplodocus, clearly improves the state-of-the-art. \n\nWeaknesses\n  - I didn't spot any major technical issues with the paper. A couple of minor suggestions and questions.\n  - How might a standard MCTS (AlphaGo style) fare? While unsound for the reasons mentioned, it might make the experimental section more complete.\n  - What would be needed to make the error bars in Table 2 smaller?\n  - I'd be curious to better understand how performance varies as a function of human player level. For example, can Diplodocus outperform weak human players? This would likely require a more complex tournament setup and is likely outside the scope of the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G_E32wHwS3", "reviewer": "Reviewer_gn6m", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers multi-player games, in particular No-press diplomacy. A new algorithm is proposed that combines self-play reinforcement learning with a no-regret learning algorithm that is regularized by human play.\n\nTheoretical properties of the no-regret algorithm are proven bounding the cumulative regret for the multi-player case, and showing convergence to Bayes-Nash equilibrium in the two-player case.\n\nExperiments with human players show that variants of the algorithm perform better than some baseline algorithms, and even on par with the expert human players included in the tournament. The human analysis of the computer players' behavior reveals that the proposed variants elicit more human-like cooperation strategies. \n", "summary_of_the_review": "The paper proposes an algorithm with theoretical guarantees and strong empirical performance in a difficult and complex domain. While the components of the algorithm are not very novel, the paper still provides a significant contribution. ", "strength_and_weaknesses": "The proposed algorithm is sound. It has some novelty, although it is mostly a combination of existing algorithms. There is also a theoretical analysis of the DiL-PiKL component.\n\nThe empirical performance seems also impressive, and the expert analysis certainly sheds further insights into the strength and weaknesses of the algorithm variants. \n\nIt is clear that regularization is mainly added to inject some cooperation flavor into the algorithm. A large chunk of the human play does not involve cooperating moves, but moves that aim to improve the position of the player (maximize the reward). Because regularization is uniform across all moves, it is forced to be small enough to avoid drop in performance (even if the human moves are reasonably good), which leads to weaker and less coherent cooperation, when needed (as indicated by the experts). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%