AlphaZero-based Proof Cost Network to Aid Game Solving

1 INTRODUCTION . There are two main goals in the pursuit of strong game-playing agents . The first is to push the boundaries of artificial intelligence since games can be seen as simplified models of the real world . The second involves finding game-theoretic values , or outcomes given optimal play , for various games ( van den Herik et al. , 2002 ) . These two closely related yet separate goals are commonly referred to as playing and solving1 games , respectively . For playing , researchers have found success in building super-human level game-playing agents for many games , including Go ( Silver et al. , 2017 ; 2016 ) , Chess , Shogi ( Silver et al. , 2018 ) , and Atari games ( Schrittwieser et al. , 2020 ; Mnih et al. , 2015 ) . Achievements for solving include checkers ( Schaeffer et al. , 2007 ) , Hex up to board sizes of 9x9 ( Pawlewicz & Hayward , 2013 ; Henderson et al. , 2009 ) , Go for board sizes up to 5x6 ( van der Werf & Winands , 2009 ) , 15x15 Gomoku ( Allis , 1994 ) , among others . Previously , researchers used hand-crafted heuristics with game-specific knowledge , combined with search methods such as Monte Carlo tree search ( MCTS ) ( Winands et al. , 2008 ) , proof number search ( PNS ) ( Allis et al. , 1994 ) , depth-first proof number search ( DFPN ) ( Nagai , 2002 ; 1999 ) , and threat-space search ( Allis et al. , 1993 ) , to solve problems by massively pruning away unnecessary branches ( Schaeffer et al. , 2007 ; van der Werf et al. , 2003 ; Allis et al. , 1996 ; Allis , 1994 ) . With the great success of AlphaGo and AlphaZero , there have been attempts to combine neural networks with previous solvers to further improve game solving . For example , Gao et al . ( 2017 ) incorporated an AlphaGo-like neural network into a state-of-the-art Hex solver to solve 8x8 Hex openings more quickly , which demonstrated that neural networks can have a positive impact when combined with previous techniques and heuristics . Meanwhile , McAleer et al . ( 2018 ) also trained an AlphaZero- ∗These authors contributed equally . 1A solution to a game is an exhaustive strategy that guarantees the game-theoretic value against all opposing actions from the game ’ s initial position ( Allis , 1994 ) . like agent to solve the Rubik ’ s Cube . In addition , game solving techniques were also shown to be viable for non-game applications ( Kishimoto et al. , 2019 ; Segler et al. , 2018 ) . However , a limitation of using these techniques is that the networks trained with the objective of strong play may not be well-suited for obtaining game-theoretic values . Since the goal of a strong player is to play moves that maximize win rates ( or the highest confidence for winning ) , as a heuristic it often does not choose moves that result in the fewest nodes in the search tree when solving . As an example , Agostinelli et al . ( 2019 ) pointed out that AlphaZero-like systems tend to solve problems with longer solutions . For games with a singular end state , e.g . Rubik ’ s cube and n-puzzles , they proposed an approach that trains a cost-to-go value function that approximates the cost of finding the shortest-path solution , which can be viewed as a kind of prioritized-sweeping ( Moore & Atkeson , 1993 ) algorithm , starting from the end state ( goal ) . However , singular end states are often only exploitable for puzzles . Indeed , in two-player zero-sum games such as Go , Hex , Gomoku , Connect6 , and many others , the game may end in too many configurations to enumerate practically . This paper presents a novel approach to solving these problems , while still tending to choose moves that lead to the fastest solutions . In our approach , we propose a new heuristic , referred to as the proof cost . We give two concrete examples of how proof cost can be designed . The first acts as a “ shortest path ” heuristic to a proof , while the second predicts solution tree sizes . Then , we use AlphaZero training to approximate this heuristic . The resulting network is referred to as the Proof Cost Network ( PCN ) , which can be used as a heuristic with any AND/OR tree search algorithm to improve solving efficiency . Experiments were conducted on 15x15 Gomoku and 9x9 Killall-Go problems . The results show that PCN outperforms AlphaZero when used as a heuristic with MCTS-based and Focused DFPN ( Gao et al. , 2017 ; Henderson et al. , 2009 ) solvers , which are both commonly used to solve games . 2 BACKGROUND . 2.1 THE GAME-THEORETIC VALUE AND PROOF TREES . For a one or two-player game , the game-theoretic value of a game state is the best outcome that can be obtained by all players if they play optimally . In this paper , we focus on two-player games , and simplify the outcome to only include win/loss ; draws are considered losses for both players2 . A game state is said to be solved when the game-theoretic value for the state is obtained , and proved/disproved if the player to play for the state has a winning/losing outcome . In a proof , the winning player needs to ensure a winning response to all possible actions by the losing player . A proof tree ( Pijls & de Bruin , 2001 ; Stockman , 1979 ) is an AND-OR search tree representing a strategy of answering actions such that ( a ) all terminal nodes are wins , ( b ) the winning player ( the OR-player ) contains at least one winning action , ( c ) all losing player ( the AND-player ) actions are enumerated , and the OR-player needs to have winning responses to each AND-player action . A disproof tree is the dual case of the above ( with win/loss and AND/OR reversed ) ; a solution tree is a general term used to describe both proofs and disproofs . 2.2 MONTE CARLO TREE SEARCH . Monte Carlo tree search ( MCTS ) ( Coulom , 2006 ; Kocsis & Szepesvári , 2006 ) is a heuristic best-first search algorithm that iteratively repeats the following four phases : selection , expansion , evaluation , and backpropagation . In the selection phase , starting from the root node , the algorithm chooses a child node according to a specified selection criterion until a leaf node is reached . Given a state s during selection , AlphaZero selects an action a∗ by the PUCT equation ( Silver et al. , 2018 ; Rosin , 2011 ) : a∗ = arg max a { Q ( s , a ) + cPUCT × P ( s , a ) × √∑ bN ( s , b ) 1 +N ( s , a ) } , ( 1 ) where Q ( s , a ) = W ( s , a ) /N ( s , a ) is the estimated mean win rate , N ( s , a ) is the number of times a is selected at s , W ( s , a ) is the number of wins , cPUCT is a weight for exploration , and P ( s , a ) is a 2We can solve the same problem twice , once for each player ; if both outcomes are losses , then we know the outcome is a draw . prior knowledge heuristic . The above selected leaf then expands all children which are added to the tree . The leaf node is then evaluated . In AlphaZero , a two-head network is trained such that it outputs a policy p ( s , a ) , and a value v ( s ) . The policy is a distribution that estimates how likely each child of s will be visited during selection ; p ( s , a ) can then be used as P ( s , a ) in eq . 1 . The scalar v ( s ) is an estimate for the game-theoretic value of s. During backpropagation we move back up the selection sequence andQ ( si , ai ) is updated for all ancestors si . While originally designed for playing , MCTS can also be used to solve games with AND-OR Boolean backpropagation . MCTS has been used to solve games and problems for Lines of Action ( Winands et al. , 2010 ; 2008 ) , Connect4 , Seki in the game of Go ( Cazenave & Saffidine , 2010 ) , and Connect6 ( Wei et al. , 2015 ) . 2.3 PROOF NUMBER SEARCH . Proof number search ( PNS ) ( Allis et al. , 1994 ) is a best-first search algorithm that was designed specifically for solving games . Each node is associated with two numbers : the proof number ( PN ) and the disproof number ( DN ) , which are heuristics that represent the minimum required number of leaf nodes that need to be expanded to prove and disprove the node respectively . PNS largely follows the same four phases as in MCTS . What is called the most-proving node ( MPN ) is selected during the selection phase . Specifically , the child with the lowest PN at each OR node , or the lowest DN at each AND node is selected ; the resulting leaf node is then the MPN that needs to be evaluated . Following the definition of the PN/DN , PNS therefore builds the solution tree by expanding and evaluating the least number of nodes . There are many PNS variants , one of which is the so-called depth-first proof number search ( DFPN ) ( Nagai , 2002 ; 1999 ) , which addresses the large memory requirements for large searches . PNS was originally designed such that all newly expanded nodes initialize PN/DN values to 1/1 . To improve the heuristic , domain knowledge can be used to initialize PN/DN values differently , which speeds up the search for a solution ( Winands & Schadd , 2010 ; Wu et al. , 2010 ; Saito et al. , 2006 ; Kishimoto & Müller , 2005 ; Allis , 1994 ) . In addition , Henderson ( 2010 ) proposed Focused DFPN ( FDFPN ) , which uses heuristics to direct the search toward promising solutions with a technique that is often referred to as progressive widening . Specifically , progressive widening involves prioritizing search on only a portion of all possible moves . The branching factor b at an internal search node is dynamically adjusted by b = bbase + dµ × |blivechildren|e , where bbase is set to 1 , blivechildren is the number of all yet unsolved children of the node , and µ is a weighting constant . With an accurate heuristic , FDFPN can solve problems while searching fewer nodes than unmodified DFPN . Further , Gao et al . ( 2017 ) proposed FDFPN-CNN by incorporating the policy and value outputs from an AlphaGo-based agent as heuristics to solve 8x8 Hex . Namely , the policy was used for move ordering and the value was used to control the branching factor by b = bbase+df ( s ) ×|blivechildren|e and f ( s ) = min { µ , 1 + v ( s ) } . 2.4 THE ALPHAZERO ALGORITHM . AlphaZero ( Silver et al. , 2018 ) is a general reinforcement learning algorithm that can achieve superhuman level playing strength for games without requiring any expert domain knowledge . The training routine consists of two phases : self-play and optimization . In a nutshell , a two-headed neural network that outputs a policy distribution p and a value scalar v is periodically optimized with data that is collected via self-play . Self-play games are generated by running a set number of MCTS simulations with the most recently optimized neural network . In the optimization phase , random positions are sampled from the latest self-play games , where the network is optimized by the following loss function : L = ( z − v ) 2 − πT log p+ c‖θ‖2 , ( 2 ) where z is the ground truth outcome of the game for that position , π is the MCTS search distribution , c is an L2 regularization weight , and θ represents the network parameters .

Summary: This paper uses a modified AlphaZero MCTS training loop to generate proof tree size heuristics, for use in MCTS or proof number search (PNS). The authors demonstrate that both heuristics outperform no heuristic and a existing heuristic method based off of standard search policy and value functions. The ideas and results are a nice step forward in a search domain that is is not always well-addressed by classical value-maximising search.