Playing Nondeterministic Games through Planning with a Learned Model

1 INTRODUCTION . While the AlphaZero algorithm achieved superhuman performance in a variety of challenging domains , it relies upon a perfect simulation of the environment dynamics to perform precision planning . MuZero , the newest member of the AlphaZero family , combines the advantages of planning with a learned model of its environment , allowing it to tackle problems such as the Atari suite without the advantage of a simulator . This paper presents Nondeterministic MuZero ( NDMZ ) , an extension of MuZero to stochastic , two-player , zero-sum games of perfect information . In it , we formalize the element of chance as a player in the game , determine a policy for the chance player via interaction with the environment , and augment the tree search to allow for chance actions . As with MuZero , NDMZ is trained end-to-end in terms of policy and value . However , NDMZ aims to learn two additional quantities : the player identity policy and the chance player policy . With the assumption of perfect information , we change the MuZero architecture to allow agents to determine whose turn it is to move and when a chance action should occur . The agent learns the fixed distribution of chance actions for any given state , allowing it to model the effects of chance in the larger context of environmental dynamics . Finally , we introduce new node classes to the Monte Carlo Tree Search , allowing the search to accommodate chance in a flexible manner . Our experiments using Nannon , a simplified variant of backgammon , show that NDMZ approaches AlphaZero in terms of performance and attains a high degree of dynamics accuracy . 2 PRIOR WORK . 2.1 EARLY TREE SEARCH . The classic search algorithm for perfect information games with chance events is expectiminimax , which augments the well-known minimax algorithm with the addition of of chance nodes ( Mitchie , 1966 ; Maschler et al. , 2013 ) . Chance nodes assume the expected value of a random event taking place , taking a weighted average of each of its children , based on the probability of reaching the child , rather than the max or min . Chance nodes , min nodes , and max nodes are interleaved depending on the game being played . The * -minimax algorithm extends the alpha-beta tree pruning strategy to games with chance nodes ( Ballard , 1983 ; Knuth & Moore , 1975 ) . 2.2 MODEL-FREE REINFORCEMENT LEARNING . A notable success of model-free reinforcement learning for nondeterministic games is TD-Gammon , an adaptation of Sutton & Barto ( 2018 ) ’ s TD-Lambda algorithm to the domain of backgammon ( Tesauro , 1995 ) . Using temporal difference learning , a network is trained to produce the estimated value of a state . When it is time to make a move , each legal move is used to produce a successor state , and the move with the maximum expected outcome is chosen . Agents produced by TD-Gammon were able to compete with world-champion backgammon players , although it was argued that the stochastic nature of backgammon assisted the learning process ( Pollack & Blair , 1996 ) . 2.3 MONTE CARLO TREE SEARCH . The Monte Carlo Tree Search ( MCTS ) algorithm attracted interest after its success with challenging deterministic problems such as the game Go , and it has been adapted for stochastic games in an number of different ways ( Coulom , 2006 ; Browne et al. , 2012 ) . A version of MCTS by Van Lishout et al . ( 2007 ) called called McGammon ( Monte Carlo Backgammon ) was capable of finding expert moves in a simplified backgammon game . At a chance node , the next move is always chosen randomly ; at choice nodes , the Upper Confidence bound applied to Trees ( UCT ) algorithm is used in the selection phase to choose player actions ( Kocsis & Szepesvári , 2006 ) . The McGammon approach was adapted by Yen et al . ( 2014 ) to the game of Chinese Dark Chess and formalized as Nondeterministic Monte Carlo Tree Search ( NMCTS ) . There are two node types in NMCTS : deterministic state nodes and nondeterministic state nodes . Nondeterministic state nodes contain at least one inner node ; each inner node has its own visit count , win count , and subtree . Upon reaching a nondeterministic state node , an inner node is chosen using “ roulette wheel selection ” based on the probability distribution of possible resulting states . NMCTS was designed to suit the nature of Chinese Dark Chess , in which both types of state nodes may exist on the same level of the search tree ; this is in contrast to backgammon , in which homogeneous layers of chance nodes and choice nodes are predictably interleaved . Alternative MCTS approaches have been adapted for densely stochastic games , wherein the the branching factor at chance nodes is so great that successor states at chance nodes are unlikely to ever be resampled . These include Double Progressive Widening and Monte Carlo * -Minimax Search ( Chaslot et al . ; Lanctot et al. , 2013 ) . In this paper , however , we are concerned with games that are not densely stochastic . 2.4 ALPHAZERO AND MUZERO . MCTS served as the foundation of the AlphaGo and AlphaZero algorithms , the latter of which attained superhuman performance in the deterministic domains of chess , Go , and shogi ( Silver et al. , 2016 ; b ; a ) . AlphaZero substitutes neural network activations for random rollouts to produce policy and value evaluations , employs the Polynomial Upper Confidence Trees ( PUCT ) algorithm during search , and trains solely via self-play reinforcement learning ( Rosin , 2011 ) . Hsueh et al . ( 2018 ) adapted AlphaZero for a simplified , solved version of the nondeterministic game of Chinese Dark Chess , finding that AlphaZero is capable of attaining optimal play in a stochastic setting . A notable limitation of AlphaZero is that the agent relies upon a perfect simulator in order to perform its lookahead search . The MuZero algorithm frees AlphaZero of this restriction , creating a model of environmental dynamics as well as game-playing strategies through interactions with the environment and self-play ( Schrittwieser et al. , 2019 ) . Until recently , the state of the art in complex single-player domains such as Atari games involved model-free reinforcement learning approaches such as Q-learning and the family of algorithms that followed ( Mnih et al. , 2015 ; Çalışır & Pehlivanoğlu , 2019 ) . Despite the burden of learning enviornmental dynamics , MuZero leverages its precision-planning capabilities to achieve an extremely strong degree of performance on these problems . Given parameters θ and a state representation at timestep k , AlphaZero employs a single neural network fθ with a policy head and a value head : pk , vk = fθ ( sk ) . In contrast , the MuZero model employs three networks for representation , prediction , and dynamics . A set of observations is passed into the representation function hθ to produce the initial hidden state : s0 = hθ ( ( o1 , . . . , ot ) ) . The dynamics function recurrently takes a previous hidden state and an action image to produce an immediate reward and the next hidden state : rk , sk = gθ ( sk−1 , ak ) . Any hidden state produced in such fashion may be passed into the prediction function fθ to produce a policy vector and scalar value : pk , vk = fθ ( sk ) . With this model , the agent may search over future trajectories a1 , ... , ak using past observations o1 , ... , ot ; in principle any MDP planning algorithm may be used given the rewards and states produced by the dynamics function . MuZero uses a MCTS algorithm similar to AlphaZero ’ s search , wherein the representation function is used to generate the hidden state for the root node . Each child node then has its own hidden state , produced by the dynamics function , and prior probability and value , produced by the prediction function . These values are used in the tree search to select an action at+1 ∼ πt via PUCT . 3 NONDETERMINISTIC MUZERO . 3.1 DEFINITIONS . We consider here an extension of the MuZero algorithm for two-player , zero-sum games of perfect information with chance events . Following Lanctot et al . ( 2019 ) , we will characterize such games as extensive-form games with the tuple 〈N , A , H , Z , u , ι , S〉 , wherein • N = { 1 , 2 , c } includes the two rival players as well as the special chance player c ; • A is the finite set of actions that players may take ; • H is the set of histories , or sequence of actions taken from the start of the game ; • Z ⊆ H is the subset of terminal histories ; • u : Z → ∆nu ⊆ Rn , where ∆u = [ umin , umax ] , is the utility function assigning each player a utility at terminal states ; • ι : H → N is the player identity function , such that ι ( h ) identifies the player to act at history h ; • S is the set of states , in which S is a partition of H such that each state s ∈ S contains histories h ∈ s that can not be distinguished by ι ( s ) = ι ( h ) where h ∈ s. • The legal actions available at state s are denoted as A ( s ) ⊆ A ; • Zero-sum games are characterized such that ∀z ∈ Z , ∑ i∈N ui ( z ) = 0 ; and • A game of perfect information is one with only one history per state : ∀s ∈ S , |s| = 1 . A chance node ( or chance event ) is a history h such that ι ( h ) = c. A policy π : S → ∆ ( A ( s ) ) , where ∆ ( X ) represents the set of probability distributions over X , describes agent behavior . An agent acts by sampling an action from its policy : a ∼ π . A deterministic policy is one where the distribution over actions has probability 1 on one action and zero on others . A policy that is not necessarily deterministic is called stochastic . The chance player always plays with a fixed , stochastic policy πchance . A transition function T : S × A → ∆ ( S ) defines a probability distribution over successor states s′ when choosing action a from state s. Because states are simply sequences of previous actions , a transition function can equivalently be represented using intermediate chance nodes between the histories of the predecessor and successor states h ∈ s and h′ ∈ s′ . The transition function is then determined by πchance . As we are concerned with perfect information games , we shall use s interchangeably with h throughout this text , such that s may refer to the single history h contained within it . For ease of notation we refer to the chance policy πchance as ψ . 3.2 MODIFICATIONS . First , we shall extend N such that N = { 1 , 2 , c , d } includes the new terminal player d. The terminal player is to act when a terminal state is reached , such that ∀z ∈ Z , ι ( z ) = d. We shall refer to nonchance , nonterminal players as choice players . Let us characterize the full action set A as the union of the choice player action set Aπ , the chance player action set Aψ , and the terminal player action set Ad as follows : A = Aπ ∪ Aψ ∪ Ad . Let us introduce the special no-op action , which is produced when it is not a player ’ s turn to move , such that ∀n ∈ N and ∀s ∈ S , An ( s ) = { no-op } when ι ( s ) 6= n. The action set Ad of the terminal player contains the no-op action as well as the end action , which represents ending the game : Ad = { no-op , end } . We can see that for any given state , the set of legal chance actions and the set of legal choice actions are both subsets of the set of all actions : ∀s ∈ S , Aψ ( s ) ⊆ A , and ∀s ∈ S , Aπ ( s ) ⊆ A . To provide input sk−1 , ak to the dynamics function gθ , the original MuZero encodes the action performed as an image and stacks it with the current hidden state . As with any choice player action , we may then encode any chance action a ∈ Aψ taken from ψ ( s ) as an image and stack it with a previous hidden state to produce input for gθ . For an agent to properly model a stochastic environment , it must determine when chance events occur as well as the distributions of their effects . The problem then is to learn an approximation of the player identity function ι ( s ) as well as the chance player policy ψ ( s ) . We then add two quantities that our model , conditioned by parameters θ , must predict : the chance policy ck ≈ ψt+k and the player identity policy ik ≈ ιt+k in addition to the quantities already present in MuZero : the policy pk ≈ πt+k ( which we now term the choice policy ) , value vk ≈ zt+k , and reward rk ≈ ut+k . Two modifications are made to the MuZero network architecture . The prediction function now produces the choice policy , chance policy , and value : pk , ck , vk = fθ ( sk ) , while the dynamics function now produces the reward , next hidden state , and player identity policy : rk , sk , ik = gθ ( sk−1 , a ) , where the length of ik = |N| . We use the no-op action as a training target for the choice policy and chance policy when it is not that player ’ s turn to move . Adding the L2 regularization term , we define our loss function as follows : lt ( θ ) = K∑ k=0 lr ( ut+k , r k t ) + l v ( zt+k , v k t ) + l p ( πt+k , p k t ) + l c ( ψt+k , c k t ) + l i ( ιt+k , i k t ) + c||θ||2 ( 1 )

Summary: This paper introduces NDMZ, short for nondeterministic MuZero, a deep reinforcement learning algorithm for model-based RL that doesn't use the rules of the game to perform search. The paper's contribution is mostly focused on describing how to construct the algorithm, and experimental results are provided at the end. A good analogy is that of a player that must play a (physical) board game by not only making decisions, but also acting out the game: producing random events, such as die rolls, and moving pieces on the board.