No-Press Diplomacy from Scratch

1 Introduction Classic multi-agent AI research domains such as chess , Go , and poker feature action spaces with at most thousands of actions per state , and are therefore possible to explore exhaustively a few steps deep . In contrast , modern multi-agent AI benchmarks such as StarCraft , Dota , and Diplomacy feature combinatorial action spaces that are incredibly large . In StarCraft and Diplomacy , this challenge has so far been addressed by bootstrapping from human data [ 32 , 24 , 3 , 11 ] . In Dota , this challenge was addressed through careful expertdesigned reward shaping [ 4 ] . In contrast , in this paper we describe an algorithm that trains agents through self play without any human data and can accommodate the large action space of Diplomacy , a longstanding AI benchmark game in which the number of legal actions for a player on a turn is often more than 1020 [ 24 ] . We propose a form of deep RL ( reinforcement learning ) where at each visited state the agent approximates a Nash equilibrium for the stage game with 1-step lookahead search , plays the equilibrium policy , and uses the equilibrium value as the training target . Since the full action space is too large to consider , this equilibrium is computed over a smaller set of candidate 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . actions . These actions are sampled from an action proposal network that is trained in unison with the value model . We show that our deep Nash value iteration algorithm , when initialized with a human-bootstrapped model , defeats multiple prior human-derived agents in no-press Diplomacy by a wide margin . Next , we investigate training an agent completely from scratch with no human data in Diplomacy . In order to improve action exploration when training from scratch , the proposal actions are augmented with exploratory actions chosen through a double-oracle-like process , and if these discovered actions end up as part of the equilibrium then they will be learned by the proposal network . Using this technique , which we call DORA ( Double Oracle Reinforcement learning for Action exploration ) , we develop a superhuman agent for a popular two-player variant of Diplomacy . Our agent is trained purely through self-play with no human data and no reward shaping . We also train a DORA agent for the 7-player game of no-press Diplomacy , the first no-press Diplomacy agent to be trained entirely from scratch with no human data . When 6 DORA agents play with 1 human-data-based agent , all human-data-based agents perform extremely poorly . However , when one DORA agent plays with 6 human-data-based agents , the bot underperforms . These results suggest that self-play in 7-player Diplomacy may converge to equilibria that are incompatible with typical human play . This is in contrast to previous large-scale multi-agent benchmarks like multiplayer poker [ 9 ] and validates Diplomacy as a valuable benchmark for learning policies that are compatible with human behavior . 2 Background and Related Work . 2.1 Description of Diplomacy . In this section we summarize the rules of Diplomacy . For a more detailed description , see [ 24 ] . No-press Diplomacy is a zero-sum board game where 7 players compete for control on a map of Europe . The map contains 75 locations , 34 of which are supply centers ( SCs ) . Each player begins with 3-4 units and 3-4 SCs and a player wins by controlling a majority ( 18 ) of the SCs . If no player controls a majority and all remaining players agree to a draw , then the victory is divided between the uneliminated players . In case of a draw , we use the Sum-of-Squares ( SoS ) scoring system that defines the score of player i as C2i / ∑ i′ C 2 i′ , where Ci is the SC count for player i . Each turn during movement , all players simultaneously choose an action composed of one order for every unit they control . An order may direct a unit to hold stationary , to move to an adjacent location , to support a nearby unit ’ s hold or move , to move across water via convoy , or to act as the convoy for another unit . Because a player may control up to 17 units with an average of 26 valid orders for each unit , the number of possible actions is typically too large to enumerate . Additionally , in Diplomacy , often supports or other critical orders may be indistinguishable from a no-op except in precise combinations with other orders or opponent actions . This makes the action space challenging to learn with black-box RL . Actions are simultaneous , so players typically must play a mixed strategy to avoid being predictable and exploitable . In the popular France vs. Austria ( FvA ) variant of Diplomacy , the number of players is reduced to two , but the victory condition and all game rules , including the number of locations and centers , remain the same . FvA serves as an effective way to isolate the challenge of an astronomically large imperfect-information action space from the challenge of Diplomacy ’ s multi-agent nature . 2.2 Stochastic Games . Stochastic games [ 26 ] are a multi-agent generalization of Markov decision processes . At each state , each player chooses an action and the players ’ joint action determines the transition to the next state . Formally , at each state s ∈ S agents simultaneously choose a joint action a = ( a1 , . . . , aN ) ∈ A1×· · ·×AN , whereN is the number of players . Agents then receive a reward according to a vector function r ( s , a ) ∈ RN and transition to a new state s′ = f ( s , a ) . The game starts at initial state s0 and continues until a terminal state is reached , tracing a trajectory τ = ( ( s0 , a0 ) , ... , ( st−1 , at−1 ) ) where f ( st−1 , at−1 ) is terminal . Each agent i plays according to a policy πi : S → ∆Ai , where ∆Ai is a set of probability distributions over actions inAi . Given a policy profileπ = ( π1 , . . . , πN ) , the probability of a trajectory is p ( τ |π ) = ∏ ( s , a ) ∈τ p ( a|a ∼ π ( s ) ) , and we define the vector of expected values of the game rooted at state s as V ( π , s ) = ∑ τ∈T ( s ) p ( τ |π ) ∑ ( st , at ) ∈τ γ tr ( st , at ) where T ( s ) is the set of terminal trajectories beginning at s. We denote by V ( π ) : = V ( π , s0 ) the value of a game rooted in the initial state . We denote by π−i the policies of all players other than i . A best response ( BR ) for agent i against policy profile π is any policy that achieves maximal expected value . That is , π∗i is a BR to π if Vi ( π ∗ i , π−i ) = maxπ′i Vi ( π ′ i , π−i ) . A policy profile is a Nash equilibrium ( NE ) if all players ’ policies are BRs to each other . Letting Vi be the ith component of V , π is a NE if : ∀i ∀π′i : Vi ( π ) ≥ Vi ( π′i , π−i ) An -NE is a policy profile in which an agent can only improve its expected reward by at most by switching to a different policy . If a policy profile is an -NE , we say its exploitability is . We may also talk about Nash equilibria for one stage of the game rather than the whole game . For any state s , supposing we have values Q ( s , a ) that represent the expected values of playing joint action a in state s assuming some policy profile is played thereafter , we may define a new 1-step game where instead the game ends immediately after this action andQ ( s , a ) is itself the final reward . For such stage games , notationally we denote NE using the symbol σ rather than π . 2.3 Nash Q-Learning . Value iteration methods such as Nash-Q have been developed that provably converge to Nash equilibria in some classes of stochastic game , including 2p0s ( two-player zero-sum ) stochastic games [ 19 , 14 ] . Nash-Q learns joint Q values Q ( s , a ) that aim to converge to the state-action value of ( s , a ) assuming that some NE π is played thereafter . This is done by performing 1-step updates on a current estimated functionQ as in standard Q-learning , but replacing the max operation with a stage game NE computation . Formally , suppose that a is played in s leading to s′ . A NE σ , σi ∈ ∆Ai , is computed for the stage game at s′ whose payoffs for each joint action a′ are specified by the current values of Q ( s′ , a′ ) . The value of this game to each player assuming this NE is then used as the target values ofQ ( s , a ) . The update rule is thus : Q ( s , a ) ← ( 1− α ) Q ( s , a ) + α ( r ( s , a ) + γ ∑ a′ σ ( a′ ) Q ( s′ , a′ ) ) , ( 1 ) where α is the learning rate , γ is the discount factor , and σ ( a ) : = ∏ i σi ( ai ) is the probability of joint action a . As presented later in Section 3.1 , the foundation of our algorithm is an adaptation of Nash QLearning , with several modifications to make it suitable for deep RL and adjusted to use a value function rather than a Q function . 2.4 Double Oracle . Double oracle ( DO ) is an iterative method for finding a NE over a large set of actions in a matrix game [ 22 ] . It leverages the fact that computing a best response is computationally much cheaper than computing a NE . Starting from a small set of candidate actions A0i ⊆ Ai for each player i , on each iteration it computes a NE σt in the matrix game restricted to the candidate actions At . Then , at each iteration t for each player i it finds the best response action at+1i ∈ Ai to the restricted NE σt and creates At+1i : = A t i ⋃ { at+1i } . Formally , a t+1 i is an action in the full action set Ai that satisfies at+1i = arg max a′∈Ai ∑ a∈At Qi ( s , a1 , . . . , ai−1 , a ′ , ai+1 , . . . , aN ) ∏ j 6=i σtj ( aj ) . ( 2 ) DO-inspired methods have been used in a variety of other work for handling large or continuous action spaces [ 15 , 7 , 18 , 1 , 21 ] . Although in the worst case the equilibrium policy may have support over a large fraction of the action space , in practice in many problems a small subset of actions may suffice , and in this case DO-based methods can be computationally efficient due to being able to restrict equilibrium computation to only a tiny fraction of the total number of actions . As we will present in Section 3.2 , we make use of a DO-like procedure both within the RL loop and at inference time , and show it to be an effective way to introduce novel good actions to the policy .

Summary: This paper describes an algorithm that trains agents through self-play with no human data and no reward shaping, and can accommodate the large action space of Diplomacy. It is an algorithm for action exploration and equilibrium approximation in games with combinatorial action spaces. This algorithm simultaneously performs value iteration reinforcement learning while learning a policy proposal network. Meanwhile, a double oracle step is used to explore additional actions to add to the policy proposals. They train an agent completely from scratch for a 2-player variant of Diplomacy and show that it achieves superhuman performance. They also train an agent for the 7-player no-press Diplomacy entirely from scratch with no human data for the first time and show that this agent differs radically from past agents that required human data. This approach therefore opens up the ability to investigate novel ways of playing the game.