Playing Nondeterministic Games through Planning with a Learned Model

1 INTRODUCTION . While the AlphaZero algorithm achieved superhuman performance in a variety of challenging domains , it relies upon a perfect simulation of the environment dynamics to perform precision planning . MuZero , the newest member of the AlphaZero family , combines the advantages of planning with a learned model of its environment , allowing it to tackle problems such as the Atari suite without the advantage of a simulator . This paper presents Nondeterministic MuZero ( NDMZ ) , an extension of MuZero to stochastic , two-player , zero-sum games of perfect information . In it , we formalize the element of chance as a player in the game , determine a policy for the chance player via interaction with the environment , and augment the tree search to allow for chance actions . As with MuZero , NDMZ is trained end-to-end in terms of policy and value . However , NDMZ aims to learn two additional quantities : the player identity policy and the chance player policy . With the assumption of perfect information , we change the MuZero architecture to allow agents to determine whose turn it is to move and when a chance action should occur . The agent learns the fixed distribution of chance actions for any given state , allowing it to model the effects of chance in the larger context of environmental dynamics . Finally , we introduce new node classes to the Monte Carlo Tree Search , allowing the search to accommodate chance in a flexible manner . Our experiments using Nannon , a simplified variant of backgammon , show that NDMZ approaches AlphaZero in terms of performance and attains a high degree of dynamics accuracy . 2 PRIOR WORK . 2.1 EARLY TREE SEARCH . The classic search algorithm for perfect information games with chance events is expectiminimax , which augments the well-known minimax algorithm with the addition of of chance nodes ( Mitchie , 1966 ; Maschler et al. , 2013 ) . Chance nodes assume the expected value of a random event taking place , taking a weighted average of each of its children , based on the probability of reaching the child , rather than the max or min . Chance nodes , min nodes , and max nodes are interleaved depending on the game being played . The * -minimax algorithm extends the alpha-beta tree pruning strategy to games with chance nodes ( Ballard , 1983 ; Knuth & Moore , 1975 ) . 2.2 MODEL-FREE REINFORCEMENT LEARNING . A notable success of model-free reinforcement learning for nondeterministic games is TD-Gammon , an adaptation of Sutton & Barto ( 2018 ) ’ s TD-Lambda algorithm to the domain of backgammon ( Tesauro , 1995 ) . Using temporal difference learning , a network is trained to produce the estimated value of a state . When it is time to make a move , each legal move is used to produce a successor state , and the move with the maximum expected outcome is chosen . Agents produced by TD-Gammon were able to compete with world-champion backgammon players , although it was argued that the stochastic nature of backgammon assisted the learning process ( Pollack & Blair , 1996 ) . 2.3 MONTE CARLO TREE SEARCH . The Monte Carlo Tree Search ( MCTS ) algorithm attracted interest after its success with challenging deterministic problems such as the game Go , and it has been adapted for stochastic games in an number of different ways ( Coulom , 2006 ; Browne et al. , 2012 ) . A version of MCTS by Van Lishout et al . ( 2007 ) called called McGammon ( Monte Carlo Backgammon ) was capable of finding expert moves in a simplified backgammon game . At a chance node , the next move is always chosen randomly ; at choice nodes , the Upper Confidence bound applied to Trees ( UCT ) algorithm is used in the selection phase to choose player actions ( Kocsis & Szepesvári , 2006 ) . The McGammon approach was adapted by Yen et al . ( 2014 ) to the game of Chinese Dark Chess and formalized as Nondeterministic Monte Carlo Tree Search ( NMCTS ) . There are two node types in NMCTS : deterministic state nodes and nondeterministic state nodes . Nondeterministic state nodes contain at least one inner node ; each inner node has its own visit count , win count , and subtree . Upon reaching a nondeterministic state node , an inner node is chosen using “ roulette wheel selection ” based on the probability distribution of possible resulting states . NMCTS was designed to suit the nature of Chinese Dark Chess , in which both types of state nodes may exist on the same level of the search tree ; this is in contrast to backgammon , in which homogeneous layers of chance nodes and choice nodes are predictably interleaved . Alternative MCTS approaches have been adapted for densely stochastic games , wherein the the branching factor at chance nodes is so great that successor states at chance nodes are unlikely to ever be resampled . These include Double Progressive Widening and Monte Carlo * -Minimax Search ( Chaslot et al . ; Lanctot et al. , 2013 ) . In this paper , however , we are concerned with games that are not densely stochastic . 2.4 ALPHAZERO AND MUZERO . MCTS served as the foundation of the AlphaGo and AlphaZero algorithms , the latter of which attained superhuman performance in the deterministic domains of chess , Go , and shogi ( Silver et al. , 2016 ; b ; a ) . AlphaZero substitutes neural network activations for random rollouts to produce policy and value evaluations , employs the Polynomial Upper Confidence Trees ( PUCT ) algorithm during search , and trains solely via self-play reinforcement learning ( Rosin , 2011 ) . Hsueh et al . ( 2018 ) adapted AlphaZero for a simplified , solved version of the nondeterministic game of Chinese Dark Chess , finding that AlphaZero is capable of attaining optimal play in a stochastic setting . A notable limitation of AlphaZero is that the agent relies upon a perfect simulator in order to perform its lookahead search . The MuZero algorithm frees AlphaZero of this restriction , creating a model of environmental dynamics as well as game-playing strategies through interactions with the environment and self-play ( Schrittwieser et al. , 2019 ) . Until recently , the state of the art in complex single-player domains such as Atari games involved model-free reinforcement learning approaches such as Q-learning and the family of algorithms that followed ( Mnih et al. , 2015 ; Çalışır & Pehlivanoğlu , 2019 ) . Despite the burden of learning enviornmental dynamics , MuZero leverages its precision-planning capabilities to achieve an extremely strong degree of performance on these problems . Given parameters θ and a state representation at timestep k , AlphaZero employs a single neural network fθ with a policy head and a value head : pk , vk = fθ ( sk ) . In contrast , the MuZero model employs three networks for representation , prediction , and dynamics . A set of observations is passed into the representation function hθ to produce the initial hidden state : s0 = hθ ( ( o1 , . . . , ot ) ) . The dynamics function recurrently takes a previous hidden state and an action image to produce an immediate reward and the next hidden state : rk , sk = gθ ( sk−1 , ak ) . Any hidden state produced in such fashion may be passed into the prediction function fθ to produce a policy vector and scalar value : pk , vk = fθ ( sk ) . With this model , the agent may search over future trajectories a1 , ... , ak using past observations o1 , ... , ot ; in principle any MDP planning algorithm may be used given the rewards and states produced by the dynamics function . MuZero uses a MCTS algorithm similar to AlphaZero ’ s search , wherein the representation function is used to generate the hidden state for the root node . Each child node then has its own hidden state , produced by the dynamics function , and prior probability and value , produced by the prediction function . These values are used in the tree search to select an action at+1 ∼ πt via PUCT . 3 NONDETERMINISTIC MUZERO . 3.1 DEFINITIONS . We consider here an extension of the MuZero algorithm for two-player , zero-sum games of perfect information with chance events . Following Lanctot et al . ( 2019 ) , we will characterize such games as extensive-form games with the tuple 〈N , A , H , Z , u , ι , S〉 , wherein • N = { 1 , 2 , c } includes the two rival players as well as the special chance player c ; • A is the finite set of actions that players may take ; • H is the set of histories , or sequence of actions taken from the start of the game ; • Z ⊆ H is the subset of terminal histories ; • u : Z → ∆nu ⊆ Rn , where ∆u = [ umin , umax ] , is the utility function assigning each player a utility at terminal states ; • ι : H → N is the player identity function , such that ι ( h ) identifies the player to act at history h ; • S is the set of states , in which S is a partition of H such that each state s ∈ S contains histories h ∈ s that can not be distinguished by ι ( s ) = ι ( h ) where h ∈ s. • The legal actions available at state s are denoted as A ( s ) ⊆ A ; • Zero-sum games are characterized such that ∀z ∈ Z , ∑ i∈N ui ( z ) = 0 ; and • A game of perfect information is one with only one history per state : ∀s ∈ S , |s| = 1 . A chance node ( or chance event ) is a history h such that ι ( h ) = c. A policy π : S → ∆ ( A ( s ) ) , where ∆ ( X ) represents the set of probability distributions over X , describes agent behavior . An agent acts by sampling an action from its policy : a ∼ π . A deterministic policy is one where the distribution over actions has probability 1 on one action and zero on others . A policy that is not necessarily deterministic is called stochastic . The chance player always plays with a fixed , stochastic policy πchance . A transition function T : S × A → ∆ ( S ) defines a probability distribution over successor states s′ when choosing action a from state s. Because states are simply sequences of previous actions , a transition function can equivalently be represented using intermediate chance nodes between the histories of the predecessor and successor states h ∈ s and h′ ∈ s′ . The transition function is then determined by πchance . As we are concerned with perfect information games , we shall use s interchangeably with h throughout this text , such that s may refer to the single history h contained within it . For ease of notation we refer to the chance policy πchance as ψ . 3.2 MODIFICATIONS . First , we shall extend N such that N = { 1 , 2 , c , d } includes the new terminal player d. The terminal player is to act when a terminal state is reached , such that ∀z ∈ Z , ι ( z ) = d. We shall refer to nonchance , nonterminal players as choice players . Let us characterize the full action set A as the union of the choice player action set Aπ , the chance player action set Aψ , and the terminal player action set Ad as follows : A = Aπ ∪ Aψ ∪ Ad . Let us introduce the special no-op action , which is produced when it is not a player ’ s turn to move , such that ∀n ∈ N and ∀s ∈ S , An ( s ) = { no-op } when ι ( s ) 6= n. The action set Ad of the terminal player contains the no-op action as well as the end action , which represents ending the game : Ad = { no-op , end } . We can see that for any given state , the set of legal chance actions and the set of legal choice actions are both subsets of the set of all actions : ∀s ∈ S , Aψ ( s ) ⊆ A , and ∀s ∈ S , Aπ ( s ) ⊆ A . To provide input sk−1 , ak to the dynamics function gθ , the original MuZero encodes the action performed as an image and stacks it with the current hidden state . As with any choice player action , we may then encode any chance action a ∈ Aψ taken from ψ ( s ) as an image and stack it with a previous hidden state to produce input for gθ . For an agent to properly model a stochastic environment , it must determine when chance events occur as well as the distributions of their effects . The problem then is to learn an approximation of the player identity function ι ( s ) as well as the chance player policy ψ ( s ) . We then add two quantities that our model , conditioned by parameters θ , must predict : the chance policy ck ≈ ψt+k and the player identity policy ik ≈ ιt+k in addition to the quantities already present in MuZero : the policy pk ≈ πt+k ( which we now term the choice policy ) , value vk ≈ zt+k , and reward rk ≈ ut+k . Two modifications are made to the MuZero network architecture . The prediction function now produces the choice policy , chance policy , and value : pk , ck , vk = fθ ( sk ) , while the dynamics function now produces the reward , next hidden state , and player identity policy : rk , sk , ik = gθ ( sk−1 , a ) , where the length of ik = |N| . We use the no-op action as a training target for the choice policy and chance policy when it is not that player ’ s turn to move . Adding the L2 regularization term , we define our loss function as follows : lt ( θ ) = K∑ k=0 lr ( ut+k , r k t ) + l v ( zt+k , v k t ) + l p ( πt+k , p k t ) + l c ( ψt+k , c k t ) + l i ( ιt+k , i k t ) + c||θ||2 ( 1 )

Summary: The paper extends MuZero for nondeterministic domains (NDMZ). Compared to MuZero NDMZ also learns a function that determines who is to act (player 1,2 or chance) and a distribution of chance outcomes. This makes it possible to employ MCTS search adjusted to handle nondeterministic nodes on top of a tree constructed by NDMZ's neural nets.