No-Press Diplomacy from Scratch

1 Introduction Classic multi-agent AI research domains such as chess , Go , and poker feature action spaces with at most thousands of actions per state , and are therefore possible to explore exhaustively a few steps deep . In contrast , modern multi-agent AI benchmarks such as StarCraft , Dota , and Diplomacy feature combinatorial action spaces that are incredibly large . In StarCraft and Diplomacy , this challenge has so far been addressed by bootstrapping from human data [ 32 , 24 , 3 , 11 ] . In Dota , this challenge was addressed through careful expertdesigned reward shaping [ 4 ] . In contrast , in this paper we describe an algorithm that trains agents through self play without any human data and can accommodate the large action space of Diplomacy , a longstanding AI benchmark game in which the number of legal actions for a player on a turn is often more than 1020 [ 24 ] . We propose a form of deep RL ( reinforcement learning ) where at each visited state the agent approximates a Nash equilibrium for the stage game with 1-step lookahead search , plays the equilibrium policy , and uses the equilibrium value as the training target . Since the full action space is too large to consider , this equilibrium is computed over a smaller set of candidate 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . actions . These actions are sampled from an action proposal network that is trained in unison with the value model . We show that our deep Nash value iteration algorithm , when initialized with a human-bootstrapped model , defeats multiple prior human-derived agents in no-press Diplomacy by a wide margin . Next , we investigate training an agent completely from scratch with no human data in Diplomacy . In order to improve action exploration when training from scratch , the proposal actions are augmented with exploratory actions chosen through a double-oracle-like process , and if these discovered actions end up as part of the equilibrium then they will be learned by the proposal network . Using this technique , which we call DORA ( Double Oracle Reinforcement learning for Action exploration ) , we develop a superhuman agent for a popular two-player variant of Diplomacy . Our agent is trained purely through self-play with no human data and no reward shaping . We also train a DORA agent for the 7-player game of no-press Diplomacy , the first no-press Diplomacy agent to be trained entirely from scratch with no human data . When 6 DORA agents play with 1 human-data-based agent , all human-data-based agents perform extremely poorly . However , when one DORA agent plays with 6 human-data-based agents , the bot underperforms . These results suggest that self-play in 7-player Diplomacy may converge to equilibria that are incompatible with typical human play . This is in contrast to previous large-scale multi-agent benchmarks like multiplayer poker [ 9 ] and validates Diplomacy as a valuable benchmark for learning policies that are compatible with human behavior . 2 Background and Related Work . 2.1 Description of Diplomacy . In this section we summarize the rules of Diplomacy . For a more detailed description , see [ 24 ] . No-press Diplomacy is a zero-sum board game where 7 players compete for control on a map of Europe . The map contains 75 locations , 34 of which are supply centers ( SCs ) . Each player begins with 3-4 units and 3-4 SCs and a player wins by controlling a majority ( 18 ) of the SCs . If no player controls a majority and all remaining players agree to a draw , then the victory is divided between the uneliminated players . In case of a draw , we use the Sum-of-Squares ( SoS ) scoring system that defines the score of player i as C2i / ∑ i′ C 2 i′ , where Ci is the SC count for player i . Each turn during movement , all players simultaneously choose an action composed of one order for every unit they control . An order may direct a unit to hold stationary , to move to an adjacent location , to support a nearby unit ’ s hold or move , to move across water via convoy , or to act as the convoy for another unit . Because a player may control up to 17 units with an average of 26 valid orders for each unit , the number of possible actions is typically too large to enumerate . Additionally , in Diplomacy , often supports or other critical orders may be indistinguishable from a no-op except in precise combinations with other orders or opponent actions . This makes the action space challenging to learn with black-box RL . Actions are simultaneous , so players typically must play a mixed strategy to avoid being predictable and exploitable . In the popular France vs. Austria ( FvA ) variant of Diplomacy , the number of players is reduced to two , but the victory condition and all game rules , including the number of locations and centers , remain the same . FvA serves as an effective way to isolate the challenge of an astronomically large imperfect-information action space from the challenge of Diplomacy ’ s multi-agent nature . 2.2 Stochastic Games . Stochastic games [ 26 ] are a multi-agent generalization of Markov decision processes . At each state , each player chooses an action and the players ’ joint action determines the transition to the next state . Formally , at each state s ∈ S agents simultaneously choose a joint action a = ( a1 , . . . , aN ) ∈ A1×· · ·×AN , whereN is the number of players . Agents then receive a reward according to a vector function r ( s , a ) ∈ RN and transition to a new state s′ = f ( s , a ) . The game starts at initial state s0 and continues until a terminal state is reached , tracing a trajectory τ = ( ( s0 , a0 ) , ... , ( st−1 , at−1 ) ) where f ( st−1 , at−1 ) is terminal . Each agent i plays according to a policy πi : S → ∆Ai , where ∆Ai is a set of probability distributions over actions inAi . Given a policy profileπ = ( π1 , . . . , πN ) , the probability of a trajectory is p ( τ |π ) = ∏ ( s , a ) ∈τ p ( a|a ∼ π ( s ) ) , and we define the vector of expected values of the game rooted at state s as V ( π , s ) = ∑ τ∈T ( s ) p ( τ |π ) ∑ ( st , at ) ∈τ γ tr ( st , at ) where T ( s ) is the set of terminal trajectories beginning at s. We denote by V ( π ) : = V ( π , s0 ) the value of a game rooted in the initial state . We denote by π−i the policies of all players other than i . A best response ( BR ) for agent i against policy profile π is any policy that achieves maximal expected value . That is , π∗i is a BR to π if Vi ( π ∗ i , π−i ) = maxπ′i Vi ( π ′ i , π−i ) . A policy profile is a Nash equilibrium ( NE ) if all players ’ policies are BRs to each other . Letting Vi be the ith component of V , π is a NE if : ∀i ∀π′i : Vi ( π ) ≥ Vi ( π′i , π−i ) An -NE is a policy profile in which an agent can only improve its expected reward by at most by switching to a different policy . If a policy profile is an -NE , we say its exploitability is . We may also talk about Nash equilibria for one stage of the game rather than the whole game . For any state s , supposing we have values Q ( s , a ) that represent the expected values of playing joint action a in state s assuming some policy profile is played thereafter , we may define a new 1-step game where instead the game ends immediately after this action andQ ( s , a ) is itself the final reward . For such stage games , notationally we denote NE using the symbol σ rather than π . 2.3 Nash Q-Learning . Value iteration methods such as Nash-Q have been developed that provably converge to Nash equilibria in some classes of stochastic game , including 2p0s ( two-player zero-sum ) stochastic games [ 19 , 14 ] . Nash-Q learns joint Q values Q ( s , a ) that aim to converge to the state-action value of ( s , a ) assuming that some NE π is played thereafter . This is done by performing 1-step updates on a current estimated functionQ as in standard Q-learning , but replacing the max operation with a stage game NE computation . Formally , suppose that a is played in s leading to s′ . A NE σ , σi ∈ ∆Ai , is computed for the stage game at s′ whose payoffs for each joint action a′ are specified by the current values of Q ( s′ , a′ ) . The value of this game to each player assuming this NE is then used as the target values ofQ ( s , a ) . The update rule is thus : Q ( s , a ) ← ( 1− α ) Q ( s , a ) + α ( r ( s , a ) + γ ∑ a′ σ ( a′ ) Q ( s′ , a′ ) ) , ( 1 ) where α is the learning rate , γ is the discount factor , and σ ( a ) : = ∏ i σi ( ai ) is the probability of joint action a . As presented later in Section 3.1 , the foundation of our algorithm is an adaptation of Nash QLearning , with several modifications to make it suitable for deep RL and adjusted to use a value function rather than a Q function . 2.4 Double Oracle . Double oracle ( DO ) is an iterative method for finding a NE over a large set of actions in a matrix game [ 22 ] . It leverages the fact that computing a best response is computationally much cheaper than computing a NE . Starting from a small set of candidate actions A0i ⊆ Ai for each player i , on each iteration it computes a NE σt in the matrix game restricted to the candidate actions At . Then , at each iteration t for each player i it finds the best response action at+1i ∈ Ai to the restricted NE σt and creates At+1i : = A t i ⋃ { at+1i } . Formally , a t+1 i is an action in the full action set Ai that satisfies at+1i = arg max a′∈Ai ∑ a∈At Qi ( s , a1 , . . . , ai−1 , a ′ , ai+1 , . . . , aN ) ∏ j 6=i σtj ( aj ) . ( 2 ) DO-inspired methods have been used in a variety of other work for handling large or continuous action spaces [ 15 , 7 , 18 , 1 , 21 ] . Although in the worst case the equilibrium policy may have support over a large fraction of the action space , in practice in many problems a small subset of actions may suffice , and in this case DO-based methods can be computationally efficient due to being able to restrict equilibrium computation to only a tiny fraction of the total number of actions . As we will present in Section 3.2 , we make use of a DO-like procedure both within the RL loop and at inference time , and show it to be an effective way to introduce novel good actions to the policy .

Summary: The paper proposes learning methods for the No-Press variant of the board game Diplomacy.   The method uses an equilibrium search method (similar to [9]) to solve for Nash in a single turn of the game, estimating future returns with a value function one turn in the future. Then a policy is trained to predict the equilibrium found, while the value function is trained to predict the values of the Nash equilibrium, which implements value iteration.  The equilibrium search method is based on prior work [9]. This tackles the large action space by restricting search to only the most likely actions from a policy network. This is extended in this work by using Double Oracle to expand the action set, and using a Diplomacy-specific method to generate proposal actions, which exploits the graph structure of the game. The paper also introduces a transformer-based model for learning to play Diplomacy, which outperforms previous Graph-Network approaches, and advances methods to estimate the exploitability of a Diplomacy agent.  Experimental results show that this results in an agent that can defeat human opponents convincingly in a 2-player variant of Diplomacy.