Deep Synoptic Monte-Carlo Planning in Reconnaissance Blind Chess

1 Introduction . Choosing a Nash equilibrium strategy is rational when the opponent is able to identify and exploit suboptimal behavior [ Bowling and Veloso , 2001 ] . However , not all opponents are so responsive , and computing a Nash equilibrium is intractable for many games . This paper presents deep synoptic Monte Carlo planning ( DSMCP ) , an algorithm for large imperfect information games that seeks a best-response strategy rather than a Nash equilibrium strategy . When opponents use fixed policies , an imperfect information game may be viewed as a partially observable Markov decision process ( POMDP ) with the opponents as part of the environment . DSMCP treats playing against specific opponents as related offline reinforcement learning ( RL ) problems and exploits predictability . Importantly , the structure of having opponents with imperfect information is preserved in order to account for their uncertainty . DSMCP uses sampling to break the “ curse of dimensionality ” [ Pineau et al. , 2006 ] in three ways : sampling possible histories with a particle filter , sampling possible futures with upper confidence bound tree search ( UCT ) [ Kocsis and Szepesvári , 2006 ] , and sampling possible world states within each information state uniformly . It represents information states with a generally-applicable stochastic abstraction technique that produces a “ synopsis ” from sampled world states . This paper assesses DSMCP on reconnaissance blind chess ( RBC ) , a large imperfect information chess variant . 2 Background . Significant progress has been made in recent years in both perfect and imperfect information settings . For example , using deep neural networks to guide UCT has enabled monumental achievements in abstract strategy games as well as computer games [ Silver et al. , 2016 , 2017a , b , Schrittwieser et al. , 2019 , Wu , 2020 , Tomašev et al. , 2020 ] . This work employs deep learning in a similar fashion . Recent advancements in imperfect information games are also remarkable . Several programs have reached superhuman performance in Poker [ Moravčík et al. , 2017 , Brown and Sandholm , 2018 , 2019 , Brown et al. , 2020 ] . In particular , ReBeL [ Brown et al. , 2020 ] combines RL and search by converting 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . imperfect information games into continuous state space perfect information games with public belief states as nodes . This approach is powerful , but it relies on public knowledge and fails to scale to games with hidden actions and substantial private information , such as RBC . Information set search [ Parker et al. , 2006 , 2010 ] is a limited-depth algorithm for imperfect information games that operates on information states according to a minimax rule . This algorithm was designed for and evaluated on Kriegspiel chess , which is comparable to RBC . Partially observable Monte Carlo planning ( POMCP ) [ Silver and Veness , 2010 ] achieves optimal policies for POMDPs by tracking approximate belief states with an unweighted particle filter and planning with a variant of UCT on a search tree of histories . In practice , POMCP can suffer from particle depletion , requiring a domain-specific workaround . This work combines an unweighted particle filter with a novel information state abstraction technique which increases sample quality and supports deep learning . Smooth UCT [ Heinrich and Silver , 2015 ] and information set Monte Carlo tree search ( ISMCTS ) [ Cowling et al. , 2012 ] may be viewed as multi-agent versions of POMCP . These two algorithms for playing extensive-form games build search trees ( for each player ) of information states . These two algorithms and DSMCP all perform playouts from determinized states that are accurate from the current player ’ s perspective , effectively granting the opponent extra information . Still , Smooth UCT approached a Nash equilibrium by incorporating a stochastic bandit algorithm into its tree search . DSMCP uses a similar bandit algorithm that mixes in a learned policy during early node visits . While adapting perfect information algorithms has performed surprisingly well in some imperfect information settings [ Long et al. , 2010 ] , the theoretical guarantees of variants of counterfactual regret minimization ( CFR ) [ Neller and Lanctot , 2013 , Brown et al. , 2018 ] are enticing . Online outcome sampling ( OOS ) [ Lisy et al. , 2015 ] extends Monte Carlo counterfactual regret minimization ( MCCFR ) [ Lanctot et al. , 2009 ] by building its search tree incrementally and targeting playouts to relevant parts of the tree . OOS draws samples from the beginning of the game . MCCFR and OOS are theoretically guaranteed to converge to a Nash equilibrium strategy . Specifically , CFR-based algorithms produce mixed strategies while DSMCP relies on incidental stochasticity . Neural fictitious self-play ( NFSP ) [ Heinrich and Silver , 2016 ] is an RL algorithm for training two neural networks for imperfect information games . Experiments with NFSP employed compact observations embeddings of information states . DSMCP offers a generic technique for embedding information states in large games . Dual sequential Monte Carlo ( DualSMC ) [ Wang et al. , 2019 ] estimates belief states and plans in a continuous domain via sequential Monte Carlo with heuristics . 3 Reconnaissance blind chess . Reconnaissance blind chess ( RBC ) [ Newman et al. , 2016 , Markowitz et al. , 2018 , Gardner et al. , 2020 ] is a chess variant that incorporates uncertainty about the placement of the opposing pieces along with a private sensing mechanism . As shown in Figure 1 , RBC players are often faced with thousands of possible game states , and reducing uncertainty increases the odds of winning . Game rules Pieces move in the same way in RBC as in chess . Players can not directly observe the movement of the opposing pieces . However , at the beginning of each turn , players may view the ground truth of any 3×3 patch of the board . The information gained from the sensing action remains private to that player . Players are also informed of the location of all captures , but not the identity of capturing pieces . When a requested move is illegal , the move is substituted with the closest legal move and the player is notified of the substitution . For example , in Figure 2 ( a ) , if Black attempted to move the rook from h8 to f8 , the rook would capture the queen on g8 and stop there instead . Players are always able to track the placement of their own pieces . Capturing the opposing king wins the game , and players are not notified about check . Passing and moving into check are legal . Official competition This paper introduces the program Penumbra , the winner of the official 2020 RBC rating competition . In total , 34 programs competed to achieve the highest rating by playing public games . Ratings were computed with BayesElo [ Coulom , 2008 ] , and playing at least 100 games was required to be eligible to win . Figure 2 shows ground truth positions from the tournament in which Penumbra voluntarily put its queen in danger . Players were paired randomly , but the opponent ’ s identity was provided at the start of each game which allowed catering strategies for specific opponents . However , opponents were free to change their strategies at any point , so attempting to exploit others could backfire . Nonetheless , Penumbra sought to model and counter predictable opponents rather than focusing on finding a Nash equilibrium . Other RBC programs RBC programs have employed a variety of algorithms [ Gardner et al. , 2020 ] including Q-learning [ Mnih et al. , 2013 ] , counterfactual regret minimization ( CFR ) [ Zinkevich et al. , 2008 ] , online outcome sampling ( OOS ) [ Lisy et al. , 2015 ] , and the Stockfish chess engine [ Romstad et al. , 2020 ] . Another strong RBC program [ Highley et al. , 2020 , Blowitski and Highley , 2021 ] maintains a probability distribution for each piece . Most RBC programs select sense actions and move actions in separate ways while DSMCP unifies all action selection . Savelyev [ 2020 ] also applied UCT to RBC and modeled the root belief state with a distribution over 10,000 tracked positions . Input to a neural network consisted of the most-likely 100 positions , and storing a single training example required 3.5MB on average which was large enough to hinder training . This work represents training examples with compact synopses which are less than 1kB without compression . 4 Terminology . Consider the two-player extensive-form game with agents P = { self , opponent } , actions A , “ ground truth ” world states X , and initial state x0 ∈ X . Each time an action is taken , each agent p ∈ P is given an observation op ∈ O that matches ( ∼ ) the possible world states from p ’ s perspective . For simplicity , assume the game has deterministic actions such that each a ∈ A is a function a : X → X defined on a subset of world states X ⊂ X . Define Ax as the set of actions available from x ∈ X . An information state ( infostate ) s ∈ S for agent p consists of all observations p has received so far.1 Let X s ⊂ X be the set of all world states that are possible from p ’ s perspective from s. In general , Xs contains less information than s since some ( sensing ) actions may not affect the world state . Define a collection of limited-size world state sets L = { L ⊂ Xs : s ∈ S , |L| ≤ ` } , given a constant ` . Let ρ : X → P indicate the agent to act in each world state . Assume that Ax = Ay and ρ ( x ) = ρ ( y ) for all x , y ∈ X s and s ∈ S. Then extend the definitions of actions available A∗ and agent to act ρ over sets of world states and over infostates in the natural way . A policy π ( a|s ) is a distribution over actions given an infostate . A belief state B ( h ) is a distribution over action histories . Creating a belief state from an infostate requires assumptions about the opponent ’ s action policy τ ( a|s ) . Let 1An infostate is equivalent to an information set , which is the set of all possible action histories from p ’ s perspective [ Osborne and Rubinstein , 1994 ] . . . . . . . . . . . . .. . . . . . B̂0 B̂1 B̂t . . . . . . . . . X0 X1 Xt Algorithm 1 Bandit – Action selection with a stochastic multi-armed bandit Algorithm 2 DrawSample – Sample selection with rejection Rp : X → R map terminal states to the reward for player p. Then ( S , A , Rself , τ , s0 ) is a POMDP , where the opponent ’ s policy τ provides environment state transitions and s0 is the initial infostate . In the rest of this paper , the word “ state ” refers to a world state unless otherwise specified . 5 Deep synoptic Monte Carlo planning . Effective planning algorithms for imperfect information games must model agents ’ choice of actions based on ( belief states derived from ) infostates , not on world states themselves . Deep synoptic Monte Carlo planning ( DSMCP ) approximates infostates with size-limited sets of possible world states in L. It uses those approximations to construct a belief state and as UCT nodes [ Kocsis and Szepesvári , 2006 ] . Figure 3 provides a high-level visualization of the algorithm . A bandit algorithm chooses an action during each node visit , as described in Algorithm 1 . This bandit algorithm is similar to Smooth UCB [ Heinrich and Silver , 2015 ] in that they both introduce stochasticity by mixing in a secondary policy . Smooth UCB empirically approached a Nash equilibrium utilizing the average policy according to action visits at each node . DSMCP mixes in a neural network ’ s policy ( π ) instead . The constant c controls the level of exploration , and m controls how the policy π is mixed into the bandit algorithm . For example , taking m = 0 always selects actions directly with π without considering visit counts , and taking m =∞ never mixes in π . As in Silver et al . [ 2016 ] , π provides per-action exploration values which guide the tree search . Approximate belief states are constructed as subsets B̂ ⊂ L , where each L ∈ B̂ is a set of possible world-states from the opponent ’ s perspective . This “ second order ” representation of belief states allows DSMCP to account for the opponent ’ s uncertainty . Infostates sampled with rejection ( Algorithm 2 ) are used as the “ particles ” in a particle filter which models successive belief states . Sampling is guided by a neural network policy ( τ̂ ) based on the identity of the opponent . To counter particle deprivation , if k consecutive candidate samples are rejected as incompatible with the possible world states , then a singleton sample consisting of a randomly-chosen possible state is selected instead . Algorithm 3 ChooseAction – UCT playouts Given : b , d , ` , nvl , z ∈ Z+ , synopsis func . σ , policy π , opp . policy τ̂ , value func . ν Input : belief distribution B̂ , possible states X , visit counts N , value totals Q π̄ ← average of π ◦ σ over b samples from B̂ while time is left do J ← random sample from B̂ I ← random ` states in X including one in J a0 ← Bandit ( π̄ , Nroot , Qroot ) for t = 0 to d do xt ← the one state in I ∩ J o← observations when at is played on xt I ← { a ( x ) : x ∈ I , a ∈ AI , oself ∼ a ( x ) } J ← { a ( x ) : x ∈ J , a ∈ AJ , oopp∼ a ( x ) } I ← random ` states in I including at ( xt ) J ← random ` states in J including at ( xt ) Kt ← I and µ← π if I is to act Kt ← J and µ← τ̂ if J is to act d += 1 if NKt , at > z for i = 0 to t do NKi , ai += nvl at+1 ← Bandit ( µ ( σ ( Kt ) ) , NKt , QKt ) q ← ν ( σ ( Kt ) ) for i = 0 to t do QKi , ai += q if ρ ( Kt ) =ρ ( Ki ) else −q NKi , ai += 1− nvl return a← argmaxa Qroot Algorithm 4 PlayGame – DSMCP Given : nparticles ∈ Z+ N∗ , ∗ ← 0 // Visits ∀ ( sample , action ) ∈ L ×A Q∗ , ∗ ← 0 // Values ∀ ( sample , action ) ∈ L×A X0 ← { x0 } B̂0 ← { X0 } while the game is not over do t← current turn o← current observation // Track all possible world states Xt← { a ( x ) : x∈Xt−1 , a∈Ax , oself∼a ( x ) } // Filter belief states with the new information for i = t− 1 to 0 do Xi ← { x ∈ Xi : ∃a ∈ Ax such that a ( x ) ∈ Xi+1 } B̂i ← { I ∈ B̂i : I ∩Xi 6= ∅ } // Repopulate belief states with new particles while opponent to act or |B̂t| < nparticles do i← smallest i > 0 s.t . |B̂i| < nparticles I ← DrawSample ( B̂i−1 , Xi , N , Q ) B̂i ← B̂i ∪ { I } if self to act then a← ChooseAction ( B̂t , Xt , N , Q ) Perform action a The tree search , described in Algorithm 3 , tracks an approximate infostate for each player while simulating playouts . Playouts are also guided by policy ( π and τ̂ ) and value ( ν ) estimations from a neural network . A synopsis function σ creates a fixed-size summary of each node as input for the network . The constant b is the batch size for inference , d is the search depth , ` is the size of approximate infostates , nvl is the virtual loss weight , and z is a threshold for increasing search depth . Algorithm 4 describes how to play an entire game , tracking all possible world states . Approximate belief states ( B̂t ) are constructed for each past turn by tracking nparticles elements of L ( from the opponent ’ s point of view ) with an unweighted particle filter . Each time the agent receives a new observation , all of the ( past ) particles that are inconsistent with the observation are filtered out and replenished , starting with the oldest belief states .

Summary: The paper describes an approach to planning in imperfect information games called DSMCP. The algorithm is the basis for a system that won the 2020 reconnaissance blind chess competition. The algorithm views the game as a POMDP by assuming a fixed policy for the opponent. Infostates are represented compactly using a particle filter where each particle represents a set of world states with a maximum size. These are further compressed as "synopses", which are conjunctions or disjunctions of Boolean features over all the states in the set. The synposes are then input to a neural network that has been trained to imitate one of a set of possible policies including several other RBC programs. Policy action choices are mixed with UCB action choices to create a search tree. Saliency of different synopses is found to be somewhat correlated with decreased performance when removing that feature.