Human-Level Performance in No-Press Diplomacy via Equilibrium Search

1 INTRODUCTION . A primary goal for AI research is to develop agents that can act optimally in real-world multi-agent interactions ( i.e. , games ) . In recent years , AI agents have achieved expert-level or even superhuman performance in benchmark games such as backgammon ( Tesauro , 1994 ) , chess ( Campbell et al. , 2002 ) , Go ( Silver et al. , 2016 ; 2017 ; 2018 ) , poker ( Moravčík et al. , 2017 ; Brown & Sandholm , 2017 ; 2019b ) , and real-time strategy games ( Berner et al. , 2019 ; Vinyals et al. , 2019 ) . However , previous large-scale game AI results have focused on either purely competitive or purely cooperative settings . In contrast , real-world games , such as business negotiations , politics , and traffic navigation , involve a far more complex mixture of cooperation and competition . In such settings , the theoretical grounding for the techniques used in previous AI breakthroughs falls apart . In this paper we augment neural policies trained through imitation learning with regret minimization search techniques , and evaluate on the benchmark game of no-press Diplomacy . Diplomacy is a longstanding benchmark for research that features a rich mixture of cooperation and competition . Like previous researchers , we evaluate on the widely played no-press variant of Diplomacy , in which communication can only occur through the actions in the game ( i.e. , no cheap talk is allowed ) . Specifically , we begin with a blueprint policy that approximates human play in a dataset of Diplomacy games . We then improve upon the blueprint during play by approximating an equilibrium for the current phase of the game , assuming all players ( including our agent ) play the blueprint for the remainder of the game . Our agent then plays its part of the computed equilibrium . The equilibrium is computed via regret matching ( RM ) ( Blackwell et al. , 1956 ; Hart & Mas-Colell , 2000 ) . Search via RM has led to remarkable success in poker . However , RM only converges to a Nash equilibrium in two-player zero-sum games and other special cases , and RM was never previously shown to produce strong policies in a mixed cooperative/competitive game as complex as no-press Diplomacy . Nevertheless , we show that our agent exceeds the performance of prior agents and for the first time convincingly achieves human-level performance in no-press Diplomacy . Specifically , we show that our agent soundly defeats previous agents , that our agent is far less exploitable than previous agents , that an expert human can not exploit our agent even in repeated play , and , most ∗Equal Contribution . importantly , that our agent ranks in the top 2 % of human players when playing anonymous games on a popular Diplomacy website . 2 BACKGROUND AND RELATED WORK . Search has previously been used in almost every major game AI breakthrough , including backgammon ( Tesauro , 1994 ) , chess ( Campbell et al. , 2002 ) , Go ( Silver et al. , 2016 ; 2017 ; 2018 ) , poker ( Moravčík et al. , 2017 ; Brown & Sandholm , 2017 ; 2019b ) , and Hanabi ( Lerer et al. , 2020 ) . A major exception is real-time strategy games ( Vinyals et al. , 2019 ; Berner et al. , 2019 ) . Similar to SPARTA as used in Hanabi ( Lerer et al. , 2020 ) , our agent conducts one-ply lookahead search ( i.e. , changes the policy just for the current game turn ) and thereafter assumes all players play according to the blueprint . Similar to the Pluribus poker agent ( Brown & Sandholm , 2019b ) , our search technique uses regret matching to compute an approximate equilibrium . In a manner similar to the sampled best response algorithm of Anthony et al . ( 2020 ) , we sample a limited number of actions from the blueprint policy rather than search over all possible actions , which would be intractable . Learning effective policies in games involving cooperation and competition has been studied extensively in the field of multi-agent reinforcement learning ( MARL ) ( Shoham et al. , 2003 ) . Nash-Q and CE-Q applied Q learning for general sum games by using Q values derived by computing Nash ( or correlated ) equilibrium values at the target states ( Hu & Wellman , 2003 ; Greenwald et al. , 2003 ) . Friend-or-foe Q learning treats other agents as either cooperative or adversarial , where the Nash Q values are well defined Littman ( 2001 ) . The recent focus on “ deep ” MARL has led to learning rules from game theory such as fictitious play and regret minimization being adapted to deep reinforcement learning ( Heinrich & Silver , 2016 ; Brown et al. , 2019 ) , as well as work on game-theoretic challenges of mixed cooperative/competitive settings such as social dilemmas and multiple equilibria in the MARL setting ( Leibo et al. , 2017 ; Lerer & Peysakhovich , 2017 ; 2019 ) . Diplomacy in particular has served for decades as a benchmark for multi-agent AI research ( Kraus & Lehmann , 1988 ; Kraus et al. , 1994 ; Kraus & Lehmann , 1995 ; Johansson & Håård , 2005 ; Ferreira et al. , 2015 ) . Recently , Paquette et al . ( 2019 ) applied imitation learning ( IL ) via deep neural networks on a dataset of more than 150,000 Diplomacy games . This work greatly improved the state of the art for no-press Diplomacy , which was previously a handcrafted agent ( van Hal , 2013 ) . Paquette et al . ( 2019 ) also tested reinforcement learning ( RL ) in no-press Diplomacy via Advantage Actor-Critic ( A2C ) ( Mnih et al. , 2016 ) . Anthony et al . ( 2020 ) introduced sampled best response policy iteration , a self-play technique , which further improved upon the performance of Paquette et al . ( 2019 ) . 2.1 DESCRIPTION OF DIPLOMACY . The rules of no-press Diplomacy are complex ; a full description is provided by Paquette et al . ( 2019 ) . No-press Diplomacy is a seven-player zero-sum board game in which a map of Europe is divided into 75 provinces . 34 of these provinces contain supply centers ( SCs ) , and the goal of the game is for a player to control a majority ( 18 ) of the SCs . Each players begins the game controlling three or four SCs and an equal number of units . The game consists of three types of phases : movement phases in which each player assigns an order to each unit they control , retreat phases in which defeated units retreat to a neighboring province , and adjustment phases in which new units are built or existing units are destroyed . During a movement phase , a player assigns an order to each unit they control . A unit ’ s order may be to hold ( defend its province ) , move to a neighboring province , convoy a unit over water , or support a neighboring unit ’ s hold or move order . Support may be provided to units of any player . We refer to a tuple of orders , one order for each of a player ’ s units , as an action . That is , each player chooses one action each turn . There are an average of 26 valid orders for each unit ( Paquette et al. , 2019 ) , so the game ’ s branching factor is massive and on some turns enumerating all actions is intractable . Importantly , all actions occur simultaneously . In live games , players write down their orders and then reveal them at the same time . This makes Diplomacy an imperfect-information game in which an optimal policy may need to be stochastic in order to prevent predictability . Diplomacy is designed in such a way that cooperation with other players is almost essential in order to achieve victory , even though only one player can ultimately win . A game may end in a draw on any turn if all remaining players agree . Draws are a common outcome among experienced players because players will often coordinate to prevent any individual from reaching 18 centers . The two most common scoring systems for draws are draw-size scoring ( DSS ) , in which all surviving players equally split a win , and sum-of-squares scoring ( SoS ) , in which player i receives a score of C 2 i∑ j∈N C 2 j , where Ci is the number of SCs that player i controls ( Fogel , 2020 ) . Throughout this paper we use SoS scoring except in anonymous games against humans where the human host chooses a scoring system . 2.2 REGRET MATCHING . Regret Matching ( RM ) ( Blackwell et al. , 1956 ; Hart & Mas-Colell , 2000 ) is an iterative algorithm that converges to a Nash equilibrium ( NE ) ( Nash , 1951 ) in two-player zero-sum games and other special cases , and converges to a coarse correlated equilibrium ( CCE ) ( Hannan , 1957 ) in general . We consider a game with N players where each player i chooses an action ai from a set of actions Ai . We denote the joint action as a = ( a1 , a2 , . . . , aN ) , the actions of all players other than i as a−i , and the set of joint actions as A . After all players simultaneously choose an action , player i receives a reward of vi ( a ) ( which can also be represented as vi ( ai , a−i ) ) . Players may also choose a probability distribution over actions , where the probability of action ai is denoted πi ( ai ) and the vector of probabilities is denoted πi . Normally , each iteration of RM has a computational complexity of Πi∈N |Ai| . In a seven-player game , this is typically intractable . We therefore use a sampled form of RM in which each iteration has a computational complexity of ∑ i∈N |Ai| . We now describe this sampled form of RM . Each agent imaintains an external regret value for each action ai ∈ Ai , which we refer to simply as regret . The regret on iteration t is denoted Rti ( ai ) . Initially , all regrets are zero . On each iteration t of RM , πti ( ai ) is set according to πti ( ai ) = max { 0 , Rti ( ai ) } ∑ a′ i ∈Ai max { 0 , Rti ( a′i ) } if ∑ a′i∈Ai max { 0 , Rti ( a′i ) } > 0 1 |Ai| otherwise ( 1 ) Next , each player samples an action a∗i fromAi according to πti and all regrets are updated such that Rt+1i ( ai ) = R t i ( ai ) + vi ( ai , a ∗ −i ) − ∑ a′i∈Ai πti ( a ′ i ) vi ( a ′ i , a ∗ −i ) ( 2 ) This sampled form of RM guarantees that Rti ( ai ) ∈ O ( √ t ) with high probability ( Lanctot et al. , 2009 ) . If Rti ( ai ) grows sublinearly for all players ’ actions , as is the case in RM , then the average policy over all iterations converges to a NE in two-player zero-sum games and in general the empirical distribution of players ’ joint policies converges to a CCE as t→∞ . In order to improve empirical performance , we use linear RM ( Brown & Sandholm , 2019a ) , which weighs updates on iteration t by t.1 We also use optimism ( Syrgkanis et al. , 2015 ) , in which the most recent iteration is counted twice when computing regret . Additionally , the action our agent ultimately plays is sampled from the final iteration ’ s policy , rather than the average policy over all iterations . This reduces the risk of sampling a non-equilibrium action due to insufficient convergence . We explain this modification in more detail in Appendix F .

Summary: In this paper, the authors apply an interesting twist on 1-ply search to the problem of playing no-press Diplomacy. Diplomacy is an especially interesting application, because it is neither zero-sum nor two-player, in contrast to many recent AI success stories.  Before each action, they compute an equilibrium of the next step of the game, assuming that each player will thereafter play according to a "blueprint" strategy learned via imitation learning.  The equilibrium is computed using regret matching.  The resulting agent has very strong performance against both the previous state of the art bots, and against expert human players.