No-Press Diplomacy from Scratch

1 Introduction Classic multi-agent AI research domains such as chess , Go , and poker feature action spaces with at most thousands of actions per state , and are therefore possible to explore exhaustively a few steps deep . In contrast , modern multi-agent AI benchmarks such as StarCraft , Dota , and Diplomacy feature combinatorial action spaces that are incredibly large . In StarCraft and Diplomacy , this challenge has so far been addressed by bootstrapping from human data [ 32 , 24 , 3 , 11 ] . In Dota , this challenge was addressed through careful expertdesigned reward shaping [ 4 ] . In contrast , in this paper we describe an algorithm that trains agents through self play without any human data and can accommodate the large action space of Diplomacy , a longstanding AI benchmark game in which the number of legal actions for a player on a turn is often more than 1020 [ 24 ] . We propose a form of deep RL ( reinforcement learning ) where at each visited state the agent approximates a Nash equilibrium for the stage game with 1-step lookahead search , plays the equilibrium policy , and uses the equilibrium value as the training target . Since the full action space is too large to consider , this equilibrium is computed over a smaller set of candidate 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . actions . These actions are sampled from an action proposal network that is trained in unison with the value model . We show that our deep Nash value iteration algorithm , when initialized with a human-bootstrapped model , defeats multiple prior human-derived agents in no-press Diplomacy by a wide margin . Next , we investigate training an agent completely from scratch with no human data in Diplomacy . In order to improve action exploration when training from scratch , the proposal actions are augmented with exploratory actions chosen through a double-oracle-like process , and if these discovered actions end up as part of the equilibrium then they will be learned by the proposal network . Using this technique , which we call DORA ( Double Oracle Reinforcement learning for Action exploration ) , we develop a superhuman agent for a popular two-player variant of Diplomacy . Our agent is trained purely through self-play with no human data and no reward shaping . We also train a DORA agent for the 7-player game of no-press Diplomacy , the first no-press Diplomacy agent to be trained entirely from scratch with no human data . When 6 DORA agents play with 1 human-data-based agent , all human-data-based agents perform extremely poorly . However , when one DORA agent plays with 6 human-data-based agents , the bot underperforms . These results suggest that self-play in 7-player Diplomacy may converge to equilibria that are incompatible with typical human play . This is in contrast to previous large-scale multi-agent benchmarks like multiplayer poker [ 9 ] and validates Diplomacy as a valuable benchmark for learning policies that are compatible with human behavior . 2 Background and Related Work . 2.1 Description of Diplomacy . In this section we summarize the rules of Diplomacy . For a more detailed description , see [ 24 ] . No-press Diplomacy is a zero-sum board game where 7 players compete for control on a map of Europe . The map contains 75 locations , 34 of which are supply centers ( SCs ) . Each player begins with 3-4 units and 3-4 SCs and a player wins by controlling a majority ( 18 ) of the SCs . If no player controls a majority and all remaining players agree to a draw , then the victory is divided between the uneliminated players . In case of a draw , we use the Sum-of-Squares ( SoS ) scoring system that defines the score of player i as C2i / ∑ i′ C 2 i′ , where Ci is the SC count for player i . Each turn during movement , all players simultaneously choose an action composed of one order for every unit they control . An order may direct a unit to hold stationary , to move to an adjacent location , to support a nearby unit ’ s hold or move , to move across water via convoy , or to act as the convoy for another unit . Because a player may control up to 17 units with an average of 26 valid orders for each unit , the number of possible actions is typically too large to enumerate . Additionally , in Diplomacy , often supports or other critical orders may be indistinguishable from a no-op except in precise combinations with other orders or opponent actions . This makes the action space challenging to learn with black-box RL . Actions are simultaneous , so players typically must play a mixed strategy to avoid being predictable and exploitable . In the popular France vs. Austria ( FvA ) variant of Diplomacy , the number of players is reduced to two , but the victory condition and all game rules , including the number of locations and centers , remain the same . FvA serves as an effective way to isolate the challenge of an astronomically large imperfect-information action space from the challenge of Diplomacy ’ s multi-agent nature . 2.2 Stochastic Games . Stochastic games [ 26 ] are a multi-agent generalization of Markov decision processes . At each state , each player chooses an action and the players ’ joint action determines the transition to the next state . Formally , at each state s ∈ S agents simultaneously choose a joint action a = ( a1 , . . . , aN ) ∈ A1×· · ·×AN , whereN is the number of players . Agents then receive a reward according to a vector function r ( s , a ) ∈ RN and transition to a new state s′ = f ( s , a ) . The game starts at initial state s0 and continues until a terminal state is reached , tracing a trajectory τ = ( ( s0 , a0 ) , ... , ( st−1 , at−1 ) ) where f ( st−1 , at−1 ) is terminal . Each agent i plays according to a policy πi : S → ∆Ai , where ∆Ai is a set of probability distributions over actions inAi . Given a policy profileπ = ( π1 , . . . , πN ) , the probability of a trajectory is p ( τ |π ) = ∏ ( s , a ) ∈τ p ( a|a ∼ π ( s ) ) , and we define the vector of expected values of the game rooted at state s as V ( π , s ) = ∑ τ∈T ( s ) p ( τ |π ) ∑ ( st , at ) ∈τ γ tr ( st , at ) where T ( s ) is the set of terminal trajectories beginning at s. We denote by V ( π ) : = V ( π , s0 ) the value of a game rooted in the initial state . We denote by π−i the policies of all players other than i . A best response ( BR ) for agent i against policy profile π is any policy that achieves maximal expected value . That is , π∗i is a BR to π if Vi ( π ∗ i , π−i ) = maxπ′i Vi ( π ′ i , π−i ) . A policy profile is a Nash equilibrium ( NE ) if all players ’ policies are BRs to each other . Letting Vi be the ith component of V , π is a NE if : ∀i ∀π′i : Vi ( π ) ≥ Vi ( π′i , π−i ) An -NE is a policy profile in which an agent can only improve its expected reward by at most by switching to a different policy . If a policy profile is an -NE , we say its exploitability is . We may also talk about Nash equilibria for one stage of the game rather than the whole game . For any state s , supposing we have values Q ( s , a ) that represent the expected values of playing joint action a in state s assuming some policy profile is played thereafter , we may define a new 1-step game where instead the game ends immediately after this action andQ ( s , a ) is itself the final reward . For such stage games , notationally we denote NE using the symbol σ rather than π . 2.3 Nash Q-Learning . Value iteration methods such as Nash-Q have been developed that provably converge to Nash equilibria in some classes of stochastic game , including 2p0s ( two-player zero-sum ) stochastic games [ 19 , 14 ] . Nash-Q learns joint Q values Q ( s , a ) that aim to converge to the state-action value of ( s , a ) assuming that some NE π is played thereafter . This is done by performing 1-step updates on a current estimated functionQ as in standard Q-learning , but replacing the max operation with a stage game NE computation . Formally , suppose that a is played in s leading to s′ . A NE σ , σi ∈ ∆Ai , is computed for the stage game at s′ whose payoffs for each joint action a′ are specified by the current values of Q ( s′ , a′ ) . The value of this game to each player assuming this NE is then used as the target values ofQ ( s , a ) . The update rule is thus : Q ( s , a ) ← ( 1− α ) Q ( s , a ) + α ( r ( s , a ) + γ ∑ a′ σ ( a′ ) Q ( s′ , a′ ) ) , ( 1 ) where α is the learning rate , γ is the discount factor , and σ ( a ) : = ∏ i σi ( ai ) is the probability of joint action a . As presented later in Section 3.1 , the foundation of our algorithm is an adaptation of Nash QLearning , with several modifications to make it suitable for deep RL and adjusted to use a value function rather than a Q function . 2.4 Double Oracle . Double oracle ( DO ) is an iterative method for finding a NE over a large set of actions in a matrix game [ 22 ] . It leverages the fact that computing a best response is computationally much cheaper than computing a NE . Starting from a small set of candidate actions A0i ⊆ Ai for each player i , on each iteration it computes a NE σt in the matrix game restricted to the candidate actions At . Then , at each iteration t for each player i it finds the best response action at+1i ∈ Ai to the restricted NE σt and creates At+1i : = A t i ⋃ { at+1i } . Formally , a t+1 i is an action in the full action set Ai that satisfies at+1i = arg max a′∈Ai ∑ a∈At Qi ( s , a1 , . . . , ai−1 , a ′ , ai+1 , . . . , aN ) ∏ j 6=i σtj ( aj ) . ( 2 ) DO-inspired methods have been used in a variety of other work for handling large or continuous action spaces [ 15 , 7 , 18 , 1 , 21 ] . Although in the worst case the equilibrium policy may have support over a large fraction of the action space , in practice in many problems a small subset of actions may suffice , and in this case DO-based methods can be computationally efficient due to being able to restrict equilibrium computation to only a tiny fraction of the total number of actions . As we will present in Section 3.2 , we make use of a DO-like procedure both within the RL loop and at inference time , and show it to be an effective way to introduce novel good actions to the policy .

Summary: Diplomacy is a complex 7-player strategy game that poses a challenge for RL due to its simultaneous moves and combinatorially large action space. In recent years significant progress has been made on learning human-level play by starting from imitation learning on a dataset of human play from websites, and then using some policy improvement algorithms. This paper tackles the exploration problem, introducing a policy improvement algorithm that reaches competitive-quality play without requiring a dataset, using Nash Q-learning, a double oracle-like method, regret matching, and geographically local perturbations for action proposals. They also find advantage from switching to a transformer-based architecture.  Their method (DORA) exhibits strong results in 1v1 human play, as well as being the first from-scratch agent able to play on par with DipNet (an imitation baseline) in a 1v6 setting, in a metagame that DORA wasn’t trained in.  Separately, their policy improvement method on top of an imitation network beats previous SOTA (SearchBot).