Deep Synoptic Monte-Carlo Planning in Reconnaissance Blind Chess

1 Introduction . Choosing a Nash equilibrium strategy is rational when the opponent is able to identify and exploit suboptimal behavior [ Bowling and Veloso , 2001 ] . However , not all opponents are so responsive , and computing a Nash equilibrium is intractable for many games . This paper presents deep synoptic Monte Carlo planning ( DSMCP ) , an algorithm for large imperfect information games that seeks a best-response strategy rather than a Nash equilibrium strategy . When opponents use fixed policies , an imperfect information game may be viewed as a partially observable Markov decision process ( POMDP ) with the opponents as part of the environment . DSMCP treats playing against specific opponents as related offline reinforcement learning ( RL ) problems and exploits predictability . Importantly , the structure of having opponents with imperfect information is preserved in order to account for their uncertainty . DSMCP uses sampling to break the “ curse of dimensionality ” [ Pineau et al. , 2006 ] in three ways : sampling possible histories with a particle filter , sampling possible futures with upper confidence bound tree search ( UCT ) [ Kocsis and Szepesvári , 2006 ] , and sampling possible world states within each information state uniformly . It represents information states with a generally-applicable stochastic abstraction technique that produces a “ synopsis ” from sampled world states . This paper assesses DSMCP on reconnaissance blind chess ( RBC ) , a large imperfect information chess variant . 2 Background . Significant progress has been made in recent years in both perfect and imperfect information settings . For example , using deep neural networks to guide UCT has enabled monumental achievements in abstract strategy games as well as computer games [ Silver et al. , 2016 , 2017a , b , Schrittwieser et al. , 2019 , Wu , 2020 , Tomašev et al. , 2020 ] . This work employs deep learning in a similar fashion . Recent advancements in imperfect information games are also remarkable . Several programs have reached superhuman performance in Poker [ Moravčík et al. , 2017 , Brown and Sandholm , 2018 , 2019 , Brown et al. , 2020 ] . In particular , ReBeL [ Brown et al. , 2020 ] combines RL and search by converting 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . imperfect information games into continuous state space perfect information games with public belief states as nodes . This approach is powerful , but it relies on public knowledge and fails to scale to games with hidden actions and substantial private information , such as RBC . Information set search [ Parker et al. , 2006 , 2010 ] is a limited-depth algorithm for imperfect information games that operates on information states according to a minimax rule . This algorithm was designed for and evaluated on Kriegspiel chess , which is comparable to RBC . Partially observable Monte Carlo planning ( POMCP ) [ Silver and Veness , 2010 ] achieves optimal policies for POMDPs by tracking approximate belief states with an unweighted particle filter and planning with a variant of UCT on a search tree of histories . In practice , POMCP can suffer from particle depletion , requiring a domain-specific workaround . This work combines an unweighted particle filter with a novel information state abstraction technique which increases sample quality and supports deep learning . Smooth UCT [ Heinrich and Silver , 2015 ] and information set Monte Carlo tree search ( ISMCTS ) [ Cowling et al. , 2012 ] may be viewed as multi-agent versions of POMCP . These two algorithms for playing extensive-form games build search trees ( for each player ) of information states . These two algorithms and DSMCP all perform playouts from determinized states that are accurate from the current player ’ s perspective , effectively granting the opponent extra information . Still , Smooth UCT approached a Nash equilibrium by incorporating a stochastic bandit algorithm into its tree search . DSMCP uses a similar bandit algorithm that mixes in a learned policy during early node visits . While adapting perfect information algorithms has performed surprisingly well in some imperfect information settings [ Long et al. , 2010 ] , the theoretical guarantees of variants of counterfactual regret minimization ( CFR ) [ Neller and Lanctot , 2013 , Brown et al. , 2018 ] are enticing . Online outcome sampling ( OOS ) [ Lisy et al. , 2015 ] extends Monte Carlo counterfactual regret minimization ( MCCFR ) [ Lanctot et al. , 2009 ] by building its search tree incrementally and targeting playouts to relevant parts of the tree . OOS draws samples from the beginning of the game . MCCFR and OOS are theoretically guaranteed to converge to a Nash equilibrium strategy . Specifically , CFR-based algorithms produce mixed strategies while DSMCP relies on incidental stochasticity . Neural fictitious self-play ( NFSP ) [ Heinrich and Silver , 2016 ] is an RL algorithm for training two neural networks for imperfect information games . Experiments with NFSP employed compact observations embeddings of information states . DSMCP offers a generic technique for embedding information states in large games . Dual sequential Monte Carlo ( DualSMC ) [ Wang et al. , 2019 ] estimates belief states and plans in a continuous domain via sequential Monte Carlo with heuristics . 3 Reconnaissance blind chess . Reconnaissance blind chess ( RBC ) [ Newman et al. , 2016 , Markowitz et al. , 2018 , Gardner et al. , 2020 ] is a chess variant that incorporates uncertainty about the placement of the opposing pieces along with a private sensing mechanism . As shown in Figure 1 , RBC players are often faced with thousands of possible game states , and reducing uncertainty increases the odds of winning . Game rules Pieces move in the same way in RBC as in chess . Players can not directly observe the movement of the opposing pieces . However , at the beginning of each turn , players may view the ground truth of any 3×3 patch of the board . The information gained from the sensing action remains private to that player . Players are also informed of the location of all captures , but not the identity of capturing pieces . When a requested move is illegal , the move is substituted with the closest legal move and the player is notified of the substitution . For example , in Figure 2 ( a ) , if Black attempted to move the rook from h8 to f8 , the rook would capture the queen on g8 and stop there instead . Players are always able to track the placement of their own pieces . Capturing the opposing king wins the game , and players are not notified about check . Passing and moving into check are legal . Official competition This paper introduces the program Penumbra , the winner of the official 2020 RBC rating competition . In total , 34 programs competed to achieve the highest rating by playing public games . Ratings were computed with BayesElo [ Coulom , 2008 ] , and playing at least 100 games was required to be eligible to win . Figure 2 shows ground truth positions from the tournament in which Penumbra voluntarily put its queen in danger . Players were paired randomly , but the opponent ’ s identity was provided at the start of each game which allowed catering strategies for specific opponents . However , opponents were free to change their strategies at any point , so attempting to exploit others could backfire . Nonetheless , Penumbra sought to model and counter predictable opponents rather than focusing on finding a Nash equilibrium . Other RBC programs RBC programs have employed a variety of algorithms [ Gardner et al. , 2020 ] including Q-learning [ Mnih et al. , 2013 ] , counterfactual regret minimization ( CFR ) [ Zinkevich et al. , 2008 ] , online outcome sampling ( OOS ) [ Lisy et al. , 2015 ] , and the Stockfish chess engine [ Romstad et al. , 2020 ] . Another strong RBC program [ Highley et al. , 2020 , Blowitski and Highley , 2021 ] maintains a probability distribution for each piece . Most RBC programs select sense actions and move actions in separate ways while DSMCP unifies all action selection . Savelyev [ 2020 ] also applied UCT to RBC and modeled the root belief state with a distribution over 10,000 tracked positions . Input to a neural network consisted of the most-likely 100 positions , and storing a single training example required 3.5MB on average which was large enough to hinder training . This work represents training examples with compact synopses which are less than 1kB without compression . 4 Terminology . Consider the two-player extensive-form game with agents P = { self , opponent } , actions A , “ ground truth ” world states X , and initial state x0 ∈ X . Each time an action is taken , each agent p ∈ P is given an observation op ∈ O that matches ( ∼ ) the possible world states from p ’ s perspective . For simplicity , assume the game has deterministic actions such that each a ∈ A is a function a : X → X defined on a subset of world states X ⊂ X . Define Ax as the set of actions available from x ∈ X . An information state ( infostate ) s ∈ S for agent p consists of all observations p has received so far.1 Let X s ⊂ X be the set of all world states that are possible from p ’ s perspective from s. In general , Xs contains less information than s since some ( sensing ) actions may not affect the world state . Define a collection of limited-size world state sets L = { L ⊂ Xs : s ∈ S , |L| ≤ ` } , given a constant ` . Let ρ : X → P indicate the agent to act in each world state . Assume that Ax = Ay and ρ ( x ) = ρ ( y ) for all x , y ∈ X s and s ∈ S. Then extend the definitions of actions available A∗ and agent to act ρ over sets of world states and over infostates in the natural way . A policy π ( a|s ) is a distribution over actions given an infostate . A belief state B ( h ) is a distribution over action histories . Creating a belief state from an infostate requires assumptions about the opponent ’ s action policy τ ( a|s ) . Let 1An infostate is equivalent to an information set , which is the set of all possible action histories from p ’ s perspective [ Osborne and Rubinstein , 1994 ] . . . . . . . . . . . . .. . . . . . B̂0 B̂1 B̂t . . . . . . . . . X0 X1 Xt Algorithm 1 Bandit – Action selection with a stochastic multi-armed bandit Algorithm 2 DrawSample – Sample selection with rejection Rp : X → R map terminal states to the reward for player p. Then ( S , A , Rself , τ , s0 ) is a POMDP , where the opponent ’ s policy τ provides environment state transitions and s0 is the initial infostate . In the rest of this paper , the word “ state ” refers to a world state unless otherwise specified . 5 Deep synoptic Monte Carlo planning . Effective planning algorithms for imperfect information games must model agents ’ choice of actions based on ( belief states derived from ) infostates , not on world states themselves . Deep synoptic Monte Carlo planning ( DSMCP ) approximates infostates with size-limited sets of possible world states in L. It uses those approximations to construct a belief state and as UCT nodes [ Kocsis and Szepesvári , 2006 ] . Figure 3 provides a high-level visualization of the algorithm . A bandit algorithm chooses an action during each node visit , as described in Algorithm 1 . This bandit algorithm is similar to Smooth UCB [ Heinrich and Silver , 2015 ] in that they both introduce stochasticity by mixing in a secondary policy . Smooth UCB empirically approached a Nash equilibrium utilizing the average policy according to action visits at each node . DSMCP mixes in a neural network ’ s policy ( π ) instead . The constant c controls the level of exploration , and m controls how the policy π is mixed into the bandit algorithm . For example , taking m = 0 always selects actions directly with π without considering visit counts , and taking m =∞ never mixes in π . As in Silver et al . [ 2016 ] , π provides per-action exploration values which guide the tree search . Approximate belief states are constructed as subsets B̂ ⊂ L , where each L ∈ B̂ is a set of possible world-states from the opponent ’ s perspective . This “ second order ” representation of belief states allows DSMCP to account for the opponent ’ s uncertainty . Infostates sampled with rejection ( Algorithm 2 ) are used as the “ particles ” in a particle filter which models successive belief states . Sampling is guided by a neural network policy ( τ̂ ) based on the identity of the opponent . To counter particle deprivation , if k consecutive candidate samples are rejected as incompatible with the possible world states , then a singleton sample consisting of a randomly-chosen possible state is selected instead . Algorithm 3 ChooseAction – UCT playouts Given : b , d , ` , nvl , z ∈ Z+ , synopsis func . σ , policy π , opp . policy τ̂ , value func . ν Input : belief distribution B̂ , possible states X , visit counts N , value totals Q π̄ ← average of π ◦ σ over b samples from B̂ while time is left do J ← random sample from B̂ I ← random ` states in X including one in J a0 ← Bandit ( π̄ , Nroot , Qroot ) for t = 0 to d do xt ← the one state in I ∩ J o← observations when at is played on xt I ← { a ( x ) : x ∈ I , a ∈ AI , oself ∼ a ( x ) } J ← { a ( x ) : x ∈ J , a ∈ AJ , oopp∼ a ( x ) } I ← random ` states in I including at ( xt ) J ← random ` states in J including at ( xt ) Kt ← I and µ← π if I is to act Kt ← J and µ← τ̂ if J is to act d += 1 if NKt , at > z for i = 0 to t do NKi , ai += nvl at+1 ← Bandit ( µ ( σ ( Kt ) ) , NKt , QKt ) q ← ν ( σ ( Kt ) ) for i = 0 to t do QKi , ai += q if ρ ( Kt ) =ρ ( Ki ) else −q NKi , ai += 1− nvl return a← argmaxa Qroot Algorithm 4 PlayGame – DSMCP Given : nparticles ∈ Z+ N∗ , ∗ ← 0 // Visits ∀ ( sample , action ) ∈ L ×A Q∗ , ∗ ← 0 // Values ∀ ( sample , action ) ∈ L×A X0 ← { x0 } B̂0 ← { X0 } while the game is not over do t← current turn o← current observation // Track all possible world states Xt← { a ( x ) : x∈Xt−1 , a∈Ax , oself∼a ( x ) } // Filter belief states with the new information for i = t− 1 to 0 do Xi ← { x ∈ Xi : ∃a ∈ Ax such that a ( x ) ∈ Xi+1 } B̂i ← { I ∈ B̂i : I ∩Xi 6= ∅ } // Repopulate belief states with new particles while opponent to act or |B̂t| < nparticles do i← smallest i > 0 s.t . |B̂i| < nparticles I ← DrawSample ( B̂i−1 , Xi , N , Q ) B̂i ← B̂i ∪ { I } if self to act then a← ChooseAction ( B̂t , Xt , N , Q ) Perform action a The tree search , described in Algorithm 3 , tracks an approximate infostate for each player while simulating playouts . Playouts are also guided by policy ( π and τ̂ ) and value ( ν ) estimations from a neural network . A synopsis function σ creates a fixed-size summary of each node as input for the network . The constant b is the batch size for inference , d is the search depth , ` is the size of approximate infostates , nvl is the virtual loss weight , and z is a threshold for increasing search depth . Algorithm 4 describes how to play an entire game , tracking all possible world states . Approximate belief states ( B̂t ) are constructed for each past turn by tracking nparticles elements of L ( from the opponent ’ s point of view ) with an unweighted particle filter . Each time the agent receives a new observation , all of the ( past ) particles that are inconsistent with the observation are filtered out and replenished , starting with the oldest belief states .

Summary: This paper presents Deep Synoptic Monte Carlo Planning, the technique behind the 2020 champion Reconnaissance Blind Chess (RBC) program Penumbra.  This method samples possible world states to approximate information sets, which it then abstracts uses synopses.  These abstracted information sets are then used monte-carlo planning to determine actions to take in the game.  The approach, with some RBC specific tweaks, is evaluated in the RBC domain and shown to be very successful.