AlphaZero-based Proof Cost Network to Aid Game Solving

1 INTRODUCTION . There are two main goals in the pursuit of strong game-playing agents . The first is to push the boundaries of artificial intelligence since games can be seen as simplified models of the real world . The second involves finding game-theoretic values , or outcomes given optimal play , for various games ( van den Herik et al. , 2002 ) . These two closely related yet separate goals are commonly referred to as playing and solving1 games , respectively . For playing , researchers have found success in building super-human level game-playing agents for many games , including Go ( Silver et al. , 2017 ; 2016 ) , Chess , Shogi ( Silver et al. , 2018 ) , and Atari games ( Schrittwieser et al. , 2020 ; Mnih et al. , 2015 ) . Achievements for solving include checkers ( Schaeffer et al. , 2007 ) , Hex up to board sizes of 9x9 ( Pawlewicz & Hayward , 2013 ; Henderson et al. , 2009 ) , Go for board sizes up to 5x6 ( van der Werf & Winands , 2009 ) , 15x15 Gomoku ( Allis , 1994 ) , among others . Previously , researchers used hand-crafted heuristics with game-specific knowledge , combined with search methods such as Monte Carlo tree search ( MCTS ) ( Winands et al. , 2008 ) , proof number search ( PNS ) ( Allis et al. , 1994 ) , depth-first proof number search ( DFPN ) ( Nagai , 2002 ; 1999 ) , and threat-space search ( Allis et al. , 1993 ) , to solve problems by massively pruning away unnecessary branches ( Schaeffer et al. , 2007 ; van der Werf et al. , 2003 ; Allis et al. , 1996 ; Allis , 1994 ) . With the great success of AlphaGo and AlphaZero , there have been attempts to combine neural networks with previous solvers to further improve game solving . For example , Gao et al . ( 2017 ) incorporated an AlphaGo-like neural network into a state-of-the-art Hex solver to solve 8x8 Hex openings more quickly , which demonstrated that neural networks can have a positive impact when combined with previous techniques and heuristics . Meanwhile , McAleer et al . ( 2018 ) also trained an AlphaZero- ∗These authors contributed equally . 1A solution to a game is an exhaustive strategy that guarantees the game-theoretic value against all opposing actions from the game ’ s initial position ( Allis , 1994 ) . like agent to solve the Rubik ’ s Cube . In addition , game solving techniques were also shown to be viable for non-game applications ( Kishimoto et al. , 2019 ; Segler et al. , 2018 ) . However , a limitation of using these techniques is that the networks trained with the objective of strong play may not be well-suited for obtaining game-theoretic values . Since the goal of a strong player is to play moves that maximize win rates ( or the highest confidence for winning ) , as a heuristic it often does not choose moves that result in the fewest nodes in the search tree when solving . As an example , Agostinelli et al . ( 2019 ) pointed out that AlphaZero-like systems tend to solve problems with longer solutions . For games with a singular end state , e.g . Rubik ’ s cube and n-puzzles , they proposed an approach that trains a cost-to-go value function that approximates the cost of finding the shortest-path solution , which can be viewed as a kind of prioritized-sweeping ( Moore & Atkeson , 1993 ) algorithm , starting from the end state ( goal ) . However , singular end states are often only exploitable for puzzles . Indeed , in two-player zero-sum games such as Go , Hex , Gomoku , Connect6 , and many others , the game may end in too many configurations to enumerate practically . This paper presents a novel approach to solving these problems , while still tending to choose moves that lead to the fastest solutions . In our approach , we propose a new heuristic , referred to as the proof cost . We give two concrete examples of how proof cost can be designed . The first acts as a “ shortest path ” heuristic to a proof , while the second predicts solution tree sizes . Then , we use AlphaZero training to approximate this heuristic . The resulting network is referred to as the Proof Cost Network ( PCN ) , which can be used as a heuristic with any AND/OR tree search algorithm to improve solving efficiency . Experiments were conducted on 15x15 Gomoku and 9x9 Killall-Go problems . The results show that PCN outperforms AlphaZero when used as a heuristic with MCTS-based and Focused DFPN ( Gao et al. , 2017 ; Henderson et al. , 2009 ) solvers , which are both commonly used to solve games . 2 BACKGROUND . 2.1 THE GAME-THEORETIC VALUE AND PROOF TREES . For a one or two-player game , the game-theoretic value of a game state is the best outcome that can be obtained by all players if they play optimally . In this paper , we focus on two-player games , and simplify the outcome to only include win/loss ; draws are considered losses for both players2 . A game state is said to be solved when the game-theoretic value for the state is obtained , and proved/disproved if the player to play for the state has a winning/losing outcome . In a proof , the winning player needs to ensure a winning response to all possible actions by the losing player . A proof tree ( Pijls & de Bruin , 2001 ; Stockman , 1979 ) is an AND-OR search tree representing a strategy of answering actions such that ( a ) all terminal nodes are wins , ( b ) the winning player ( the OR-player ) contains at least one winning action , ( c ) all losing player ( the AND-player ) actions are enumerated , and the OR-player needs to have winning responses to each AND-player action . A disproof tree is the dual case of the above ( with win/loss and AND/OR reversed ) ; a solution tree is a general term used to describe both proofs and disproofs . 2.2 MONTE CARLO TREE SEARCH . Monte Carlo tree search ( MCTS ) ( Coulom , 2006 ; Kocsis & Szepesvári , 2006 ) is a heuristic best-first search algorithm that iteratively repeats the following four phases : selection , expansion , evaluation , and backpropagation . In the selection phase , starting from the root node , the algorithm chooses a child node according to a specified selection criterion until a leaf node is reached . Given a state s during selection , AlphaZero selects an action a∗ by the PUCT equation ( Silver et al. , 2018 ; Rosin , 2011 ) : a∗ = arg max a { Q ( s , a ) + cPUCT × P ( s , a ) × √∑ bN ( s , b ) 1 +N ( s , a ) } , ( 1 ) where Q ( s , a ) = W ( s , a ) /N ( s , a ) is the estimated mean win rate , N ( s , a ) is the number of times a is selected at s , W ( s , a ) is the number of wins , cPUCT is a weight for exploration , and P ( s , a ) is a 2We can solve the same problem twice , once for each player ; if both outcomes are losses , then we know the outcome is a draw . prior knowledge heuristic . The above selected leaf then expands all children which are added to the tree . The leaf node is then evaluated . In AlphaZero , a two-head network is trained such that it outputs a policy p ( s , a ) , and a value v ( s ) . The policy is a distribution that estimates how likely each child of s will be visited during selection ; p ( s , a ) can then be used as P ( s , a ) in eq . 1 . The scalar v ( s ) is an estimate for the game-theoretic value of s. During backpropagation we move back up the selection sequence andQ ( si , ai ) is updated for all ancestors si . While originally designed for playing , MCTS can also be used to solve games with AND-OR Boolean backpropagation . MCTS has been used to solve games and problems for Lines of Action ( Winands et al. , 2010 ; 2008 ) , Connect4 , Seki in the game of Go ( Cazenave & Saffidine , 2010 ) , and Connect6 ( Wei et al. , 2015 ) . 2.3 PROOF NUMBER SEARCH . Proof number search ( PNS ) ( Allis et al. , 1994 ) is a best-first search algorithm that was designed specifically for solving games . Each node is associated with two numbers : the proof number ( PN ) and the disproof number ( DN ) , which are heuristics that represent the minimum required number of leaf nodes that need to be expanded to prove and disprove the node respectively . PNS largely follows the same four phases as in MCTS . What is called the most-proving node ( MPN ) is selected during the selection phase . Specifically , the child with the lowest PN at each OR node , or the lowest DN at each AND node is selected ; the resulting leaf node is then the MPN that needs to be evaluated . Following the definition of the PN/DN , PNS therefore builds the solution tree by expanding and evaluating the least number of nodes . There are many PNS variants , one of which is the so-called depth-first proof number search ( DFPN ) ( Nagai , 2002 ; 1999 ) , which addresses the large memory requirements for large searches . PNS was originally designed such that all newly expanded nodes initialize PN/DN values to 1/1 . To improve the heuristic , domain knowledge can be used to initialize PN/DN values differently , which speeds up the search for a solution ( Winands & Schadd , 2010 ; Wu et al. , 2010 ; Saito et al. , 2006 ; Kishimoto & Müller , 2005 ; Allis , 1994 ) . In addition , Henderson ( 2010 ) proposed Focused DFPN ( FDFPN ) , which uses heuristics to direct the search toward promising solutions with a technique that is often referred to as progressive widening . Specifically , progressive widening involves prioritizing search on only a portion of all possible moves . The branching factor b at an internal search node is dynamically adjusted by b = bbase + dµ × |blivechildren|e , where bbase is set to 1 , blivechildren is the number of all yet unsolved children of the node , and µ is a weighting constant . With an accurate heuristic , FDFPN can solve problems while searching fewer nodes than unmodified DFPN . Further , Gao et al . ( 2017 ) proposed FDFPN-CNN by incorporating the policy and value outputs from an AlphaGo-based agent as heuristics to solve 8x8 Hex . Namely , the policy was used for move ordering and the value was used to control the branching factor by b = bbase+df ( s ) ×|blivechildren|e and f ( s ) = min { µ , 1 + v ( s ) } . 2.4 THE ALPHAZERO ALGORITHM . AlphaZero ( Silver et al. , 2018 ) is a general reinforcement learning algorithm that can achieve superhuman level playing strength for games without requiring any expert domain knowledge . The training routine consists of two phases : self-play and optimization . In a nutshell , a two-headed neural network that outputs a policy distribution p and a value scalar v is periodically optimized with data that is collected via self-play . Self-play games are generated by running a set number of MCTS simulations with the most recently optimized neural network . In the optimization phase , random positions are sampled from the latest self-play games , where the network is optimized by the following loss function : L = ( z − v ) 2 − πT log p+ c‖θ‖2 , ( 2 ) where z is the ground truth outcome of the game for that position , π is the MCTS search distribution , c is an L2 regularization weight , and θ represents the network parameters .

Summary: As we know that, from AlphaGo to AlphaZero, less and less expert knowledge is used. In this paper, however, the authors make AlphaZero becomes faster by modifying the training target of the AlphaZero algorithm, such that it prioritizes solving the game quickly, rather than winning, and training knowledge-based networks. Experimental evaluation is provided on 15x15 Gomoku and 9x9 Killall-Go problems.