Three-Head Neural Network Architecture for AlphaZero Learning
Authors: Chao Gao
Martin Mueller
Ryan Hayward
Hengshuai Yao
Shangling Jui
Venue:  (2020)
Keywords: alphazero
reinforcement learning
two-player games
heuristic search
deep neural networks

Abstract: The search-based reinforcement learning algorithm AlphaZero has been used as a general method for
mastering two-player games Go, chess and Shogi. One crucial ingredient in AlphaZero (and its predecessor AlphaGo Zero) is the two-head network architecture that outputs two estimates --- policy and value --- for one input game state. The merit of such an architecture is that letting policy and value learning share the same representation substantially improved generalization of the neural net.   
A three-head network architecture has been recently proposed that can learn a third action-value head on a fixed dataset the same as for two-head net. Also, using the action-value head in Monte Carlo tree search (MCTS) improved the search efficiency. 
However, effectiveness of the three-head network has not been investigated in an AlphaZero style learning paradigm. 
In this paper, using the game of Hex as a test domain, we conduct an empirical study of the three-head network architecture in AlpahZero learning. We show that the architecture is also advantageous at the zero-style iterative learning. Specifically, we find that three-head network can induce the following benefits: (1) learning can become faster as search takes advantage of the additional action-value head; (2) better prediction results than two-head architecture can be achieved when using additional action-value learning as an auxiliary task.

Under review as a conference paper at ICLR 2020
THREE-HEAD NEURAL NETWORK ARCHITECTURE FOR ALPHAZERO LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The search-based reinforcement learning algorithm AlphaZero has been used as a general method for mastering two-player games Go, chess and Shogi. One cru- cial ingredient in AlphaZero (and its predecessor AlphaGo Zero) is the two-head network architecture that outputs two estimates - policy and value for one input game state. The merit of such an architecture is that letting policy and value learning share the same representation substantially improved generalization of the neural net. three-head network architecture has been recently proposed that can learn a third action-value head on fixed dataset the same as for two-head net. Also, using the action-value head in Monte Carlo tree search (MCTS) improved the search efficiency. However, effectiveness of the three-head network has not been investigated in an AlphaZero style learning paradigm. In this paper, using the game of Hex a test domain, we conduct empirical study of the three- head network architecture in AlpahZero learning. We show that the architecture is also advantageous at the zero-style iterative learning, producing neural network models stronger than those from the two-head counterpart in the same MCTS.
INTRODUCTION
Computational advancements of deep neural networks (Bengio 2009; LeCun et al., 2013) have led progresses in number of scientific areas, ranging from image classification (Krizhevsky al. 2012). speech recognition (Graves et al. 2013), natural language processing (Collobert et al. 2011 playing Atari games (Mnih et al. 2015). to the prominent achievement of AlphaGo (Silver et al. 2016). AlphaGo Zero (Silver et al. 2017b) and AlphaZero (Silver et al. 70 2018) learning sys- tems for playing multiple classic two-player alternate-turn perfect-informatie
zero-sum games
games of which kind have been research object as old as the subject of artificial intelligence itself. Arguably, the accomplishments of AlphaGo and its successors represent result of continual effort from two lines of research directions. The heuristic search (or sometimes Al planning) community studies various intelligent search strategies (Pearl 1984) (e.g.. best-first search) given that space of states can be generated gradually by description of the problem (e.g.. rules of Go), the goal is to find a desired solution using smaller computational cost by exploring promising regions preferably; the study of machine learning attempts to summarize general concept from experience data, therefore enabling probably correct prediction even in unseen scenarios.
The combination of both learning and search techniques dates back to Samuel's studies in check- ers (Samuel 1959), where two learning methods, role learning and generalization learning, were studied and integrated into look-ahead minimax search; the later development leveraging advance- ments from both learning and search techniques (e.g., training on games produced by stronger play- ers with more expressive function approximation for better generalization learning and using better aB pruming (McCarthy et al., (006) in search) achieved better results (Samuel 1967). Noticeably, the learning techniques are only used to construct an evaluation function for checkers positions from historical playing records either generated by human or computer players. However, in certain games (e.g., chess) where a large body of human players exist, manually constructing a heuristically correct evaluation function is also possible. Therefore, rather than attempting to automatic construct an evaluation functions, much early research effort focus on analyzing formal properties of search algorithms given certain assumption the correctness of an existing evaluation function (Knuth & Moore 1975: Pearl 1980a). proposing new search routines that may lead to more efficient prun- ing (Pear) 1980b: Plaat et al. 1996), devising procedures when the assumed correctness of an

---

Under review as a conference paper at ICLR 2020
evaluation failed to consign in practice (Harris 1975). Indeed, optimizing using minimax upon an approximated evaluation has been regarded as a "deadly sin" in the eye of statistics, since such an optimization could only compound the approximation error, therefore resulting pathological behav- ior of search (Nau 1982), i.e., the deeper the search the worse of the obtained evaluation. How- ever, in practice, pathological behavior was seldom observed in number of games. Pearl (T983) presented several observations that may explain the phenomenon, such as the approximation error tends decrease as search goes deeper, the ubiquitous of perfect evaluation in some branches of the search, the fact that the search space is a directed acyclic graph rather than a tree, and the rel- searching by minimax continued to
atively small branching factor of the game of interest. triumph as computers become faster and more general or game-specific searching techniques are introduced, culminating to the successes of champion playing strength against human players in games such as checkers (Schaeffer et al. 1992), chess (Camphell et al. 2002), Othello (Buro 1998) and backgammon (Tesauro 1995).
It follows that game Al research began to switch attention several more challenging games where reliable human-directed evaluation function is difficult to construct and the branching factor sig- nificantly larger, such as the game of Go (Mäller 2002) and the game of Hex (Van Rijswijek 2002a).
Coulom (2066) proposed a Monte Carlo tree search (MCTS) for computer Go; it uses Monte Carlo simulation sidestep the need of static evaluation function. From the heuristic search perspec- tive, MCTS can be regarded as an instance of the general best-first search depicted by Pearl (184).. Different from a pruning. which relies on an accurate move ordering for preferable exploration, best-first search iteratively adjusts its search effort by summarizing the new information backed from leaf evaluations, at the expense of explicitly storing the whole search tree in memory. To select a Ieaf node to expand and evaluation from the search tree, a UCT search can be used (Kocsis & Szepesvári, 2006) for a balance between exploration and exploitation. Notably, unlike a3 minimax search, MCTS does not apply hard-min or max when backing up approximated leaf evaluation but uses average to summarize all previous evaluations while preserving asymptotic convergence. The supe- riority of MCTS is also manifested by its flexibility of integrating learned heuristic knowledge into the search (Gelly& & Silver (007).. That is, by incorporating an existing move-ranking function for better leaf node selection (i.e., progressive bias by Chaslot et al. (2008) PUCT by Rosin (201) ) the search can focus on only promising moves therefore effectively reducing the high branching factor; by leveraging more informed pattern-based Monte Carlo playout (Coulom) (007). better leaf evaluation can be obtained and backed up therefore further enforcing the best-first behavior, although in practice stronger playout policy for leaf evaluation does not always result in stronger performance given the same finite computation of MCTS (Silver & Tesauro 2009).
Continual effort were made towards adding more accurate prior knowledge to MCTS. The suc- cesses of deep neural networks in other AI domains, such as image classification (Krizhevsky et al.) 2012), revived the application of neural networks for approximate human's 's evaluation in Computer Go (Maddison et al. 2015) Clark& & Storkey 2015; Tian & Zhu 2013), leading to the development of AlphaGo Fan (Silver et al. 2016) which uses policy network for providing prior knowledge in move selection and value network for leaf node evaluation in MCTS- it became the first program defeating human professional player in the game of Go. Subsequently improved versions AlphaGo Lee defeated more pronounced world champion player in public match. A even stronger version called AlphaGo Master was strong enough that it never lost any game against any top human play- in set of online match. The commonality of these programs is that the neural networks used in search were all first trained on human professional game records. Later, it was found that even without using human data to initialize the neural network weights, closed loop learning paradigm could still learn MCTS players stronger than AlphaGo Lee and AlphaGo Master (after doubling neural net size): the paradigm is named as AlphaGo Zero (Silver et al. 20176). By further removing other Go specific encoding in AlphaGo Zero, a more general version AlphaZero (Silver et al.) 2017a; 2018) - was applied chess, Shogi and Go, producing strong players in all these games after separately training on each of them. We summarize the major differences between AlphaGo Master, AlphaGo Zero and AlphaZero in Figure T
However, one limitation of the Zero algorithms (AlphaGo Zero and AlphaZero) is that a large com- putation cost could be needed because they use search to produce millions of training games, where each move in game was produced by calling neural networks hundreds of times for building the search-tree tree- AlphaZero used thousands of TPUs in its week-long training on the game of Go Sil-
ver et al.] 2018). A three-head neural network architecture was proposed in (Gao et al. 2018b)
2

---

Under review as a conference paper at ICLR 2020
Fanling Cleems
Neural Network Model
Neural Network Model
Lainiinniie
Gating
Narii
PV-MCTS Workews
Trainnng
Paraliel PV-MCTS
Thaiinn
Prodkt MOrEs Waxiem
Taxking
Solfpheg
Solpply
Suljae
Glane Bulke
Game Buller
Gamee Buffer
(a) AlphaGo Master Silver al. (b) AlphaGo Zero (iiverr et al. (c) AlphaZero 2017b 2016)
Silver et
2017b)
2017a 2018)
Figure 1: Flowcharts for AlphaGo Master, AlphaGo Zero and AlphaZero: each of them is simpli- fication of its predecessor; that is, AlphaCio Zero removes the need human data in training, while in AlphaZero gafing is further discarded (gating refers playing tournament using existing neural models in PV-MCTS thus a model parameter will passed down to selfplay workers only when its win-percentage exceeds a certain threshold, e.g., 55%). Notice that AlphaGo Zero and AlphaZero used asynchronous training, i.e., selfplay game generation and neural network training were con- ducted simultaneously given a sufficient number of games have been stored in the buffer. Detailed differences on search and learning, such as hyperparameters, are omitted in these diagrams.
which can learn an additional action-value head on search-produce training dataset, akin to the two-head network used in AlphaGo Master, AlphaGo Zero and AlphaZero. In this paper, we con- duct empirical comparative study on AlphaZero style learning using two- and three-head neural networks. We find that, in comparison to the two-head counterpart, head network can also lead to improvement in AlphaZero style learning paradigm.
2 ANATOMY OF ALPHAGO ZERO AND ALPHAZERO
We provide anatomy AlphaGo Zero and AlphaZero, along with discussion with their predeces- sors AlphaGo Fan, AlphaGo Lee and AlphaGo Master. See Appendix A preliminaries on Monte Carlo tree search.
2.1 POLICY VALUE MONTE CARLO TREE SEARCH
MCTS an essential component AlphaGo Zero and AlphaZero since it is the algorithm upon which training data for the neural network produced; that is, the policy network is trained by mimicking the move probabilities generated by MCTS, while the value network is to learn to predict game result generated by MCTS players.
two-head valicy-value network fo provides two estimates given one input game-state, i.e., <- fo(s) where is a gtme-state, p is policy over actions and is estimate of the optimal state-value of 8. After one evaluation, policy and value estimates are added to the Monte Carlo tree search (PV-MCTS) in the following way. At each iteration of MCTS, after leaf node is selected, the game state of that leaf node is then passed to neural net for inference, p, v <- fo()). The value estimate v backpropaged each action a from s, the newly created child node stores: (NN(s, a) = 0, W a) = 0, Q(s, 0, P(s,a) a) ). where (s, a) the visit count of a at 8, W(s, a) the number of wins, Q(s, the accumulated action-value (equivalent to negated V(s') is acting a at results s' s'), and P(s,a) is the prior probability for action a. AlphaGo Fan used an expansion threshold nus > O, therefore a leaf' node is expanded only when it has been visited more than Ylthh times. AlphaGo Zero and AlphaZero used an expansion threshold of 0 because for each selected leaf node, to back up an estimate, neural network has to be called; however, the two-head neural network provides two estimates p and v; naturally, to not waste GPU computation, probabilities in p has to be saved in newly created child nodes by node expansion.
the search tree. The leaf node subsequently expanded, for
3

---

Under review as a conference paper at ICLR 2020
Scarch Param.
Description
Learning Param.
Description
Tmmet Cpuct
of MCTS iterations PUCT coefficient
dan
depth of neural net width of neural net initial learning rate how to schedule learning rate Lg regularization in loss function mini-batch size input planes to neural net
Wnn B
&
root dirichlet noise parameter
move select dithering threshold
dither temperature
reuse subtree not?
B
root fraction for dirichlet noise
Iun
Table 1: Search and Learning parameters in AlphaGo Zero and AlphaZero. In AlphaGo Zero, Amats = 1600 while in AlphaZero Timots 800. PUCT coefficient AlphaGo Zero were not 0.03 in Go; dither selection threshold ry of MCTS was 30 with temperature 1.0; weight for dirichlet noise e 0.25. Note that in (Silver et al.) 2018), dynamic PUCT coefficient with base 1.25 was used; however, as noted in (Silver et al. 2018), during training, due to small visit count of each game state, the coefficient is almost static.
explicitly specified, studies by Tian et al. (2019) suggests Cpuct = 1.5.
2.2 CLOSED LOOP LEARNING
As in Figure [ AlphaGo Zero and AlphaZero can summarized as a closed loop learning paradigm. The predecessor AlphaGo Master might also be called closed loop learning based on description revealed (Silver et al., 2017b). It begins by initializing neural network weights either with supervised learning on human data or just with random weights. Then, the neural network is plugged into an MCTS and passed to a number of parallel workers for producing selfplay games. These games are stored in external buffer, and subsequently updated neural network model is produced by retraining the neural network on game examples sampled from the buffer. Even though AlphaGo Zero and AlphaZero naturally suggests a synchronous paradigm, where at each iteration the learning worker should wait until all selfplay worker finish all their games before optimizing the neural network. Whether the asynchronous implementation is superior to the synchronous execution, or how advantageous the other side, asynchronicity introduces an
run in asynchronous manner, our summarization in Figure
could be, has never been formally investigated. additional difficulty for reproducibility (Henderson et al. (018)) due to the fact that executing results are dependent on the relative speed of the selfplay and learning workers in that specific execution. We therefore believe that the synchronous implementation of AlphaZero is worth investigating as has never been formally investigated before. For example, model-free reinforcement learning algorithm A3C (Mnih et al., 2016), asynchronicity conjectured one reason for the observed good performance, other implementation found that the synchronous version A2C (OpenAl 2017) achieve equivalent results A3C.
2.3 SEARCH AND LEARNING HYPERPARAMETERS
Due to the possibility of different hyperparameter choices, AlphaZero represents family of algo- rithms, rather than a single and rigidly defined procedure. We classify the hyperparameters into three categories: (1) search parameters, (2) learning parameters, (3) exccution parameters. While (3) could be specified by referring to the specific game and available computation hardware, we summarize the (1) and (2) in Table Observing that the learning parameters are in large accordance with many other deep learning applications (Goodfellow et al.) 2016), we emphasize that search hyperparameters used to configure MCTS not typically employed in heuristic search, making the setting of these parameters difficult.
2.4 THEORETICAL JUSTIFICATION
In terms of reinforcement learning, the algorithmic framework of AlphaGo Zero and AlphaZero has been summarized
approximated policy iteration (Bertsekas 2011) (or generalized policy iteration as (Sutton & Barto 2017)), where the policy improvement is due to running PV-MCTS at each state, and policy evaluation is achieved by regression with respect to the search-based selfplay game result (Silver et al.) 20176). However, such an interpretation is essentially high-level: does not provide detailed explanation convergence properties of the algorithm, nor does it give explicit
4

---

Under review as a conference paper at ICLR 2020
and practical guidance how to select better hyperparameters for search, neural network design and training. Another interpretation names the whole framework as an Expert Iteration (Exlt) (Anthony et al., 2017) by drawing connection to Systems and II psychological theories on human mind the fast intuition by deep neural network and the slow thinking by tree search (Evans & Stanovich, 2013). The same drawback remains; Exlt does not provide detailed guidance hyperparameter choices either.
3 POTENTIAL MERITS OF THREE-HEAD NEURAL NETWORK ARCHITECTURE
A three-head network 0 provides three estimates from three heads: p, q. fo(s). The additional head simply a two-layer network appended to the common representation shared by p and v heads: the first layer uses a single 1 x convolution filter, and the second layer uses layer of N x N fully-connected units with tanh function to transform the output into action values. For brevity, we shall call three-head network as 3HNN, and 2HNN for the two-head architecture. A 3HNN can be optimized using the following loss function:
((+A- (105) o(s))² + (s. + a(e, + max(-25.0) Σ (s, ((e,)'))²
L(fo:D)=
Σ (s.D
IA(+)
d'EA(o)
R1
(min +(s))))) xx" [s) + c|10112
R3
where e is a constant for the level of L2 regularization; DD is a set of game examples produced by search; (s, a) is a state-action pair from example set 2g the game result with respect to the player to-play at s that can be extracted from D; and A(s) is the set of actions at In comparison to the loss function used in training 2HNN, three additional value error functions, R1, R2 and R3, are added. RI can be extracted from 'D directly; R2 is produced by data- autta-uggmenttioonn method; and R3 an optimal consistence penalty between state- and action-value heads; see (Gao et al., (0). We present the following potential merits that three-head neural network may bring to MCTS and AlphaZero learning.
More efficient MCTS. One advantage that has been exploited by 3HNN in (Gao et al. 2018b) is allowing node expansion threshold nu > 0. Because when new child node is first created, like the a(s,a) Pa. the action-value can be initialized to Q(s, a) ạa- Therefore, in future MCTS iteration, a leaf value estimate can be obtained simply by querying Q(s, a) without expanding and evaluating the node using fo. When node is expanded, state-value head is backed up immediately. This improves the search efficiency of MCTS allows more iterations of search given the same computation budget.
Auxiliary task for better neural net learning. Yet how deep neural networks manage to generalize is much unknown except some intuitive explanation (Zhang et al.) (2017), many empirical evidence seem to suggest that adding auxiliary tasks (Jaderberg et al. 2016) during neural net training generally produces better results. In AlphaGo, was found that simply combining policy and value network learning into two-head architecture produces an 600 Elo gain (Silver et al. 2017b).
Improved data efficiency. Using 3HNN also improves the data efficiency in the sense that on the same dataset, one more estimator is learned, and if such an advantage is fully used in search, is expected that few number of training samples might be required achieve similar results as using 2HNN.
4 EXPERIMENTS
In this section, we present experiments conducted on the game of Hex. We first give a brief review on previous research in this game, then describe the testing data used to measure leamning progress.
5

---

Under review as a conference paper at ICLR 2020
Finally, we present experiment results and comparisons of 2HNN and 3HNN in the same AlphaZero learning.
4.1 THE GAME OF HEX
The game of Hex (Hayward & Toft 2019) can be played on arbitrary N X hexagonal board, where two players Black and White each is given two opposing borders; at each move, one player places a stone of their color on the board. Usually, Black starts first, then the game continues in alternate turns; the goal for each player is to form monochromatic chain connecting their opposing borders. The game became well-known after Nash (1952), who proved that there is a first-play winning strategy for any board size NxN though explicit winning strategies are unknown.
The game was first studied from an Al perspective in 1950s by Shannon (1953), who designed an electronic resistance simulated evaluation to play Hex deex-afee after observing that devising an evaluation using expert knowledge could be difficult in this game (unlike, e.g., chess). However, progresses have been slow in developing strong algorithms for the game. Using two-distance evaluation extracted from graph properties of Hex, together with advanced implementation of a& search, in late 1990s Queenbee (Van Rijswijck] 2002b) became the first program achieving novice strength on small board sizes. The discovery of hierarchical algebra (implemented as H-Search (Anshele- vich 2002)) to bottom up build connection strategies resulted moderate strong playing players (e.g.. Hexy AAshelevichh 2000) and Six (Hayward) 2006)) using aB search. The benefits of H-Search are manifold: (I) it can be used for early endgame state detection; (2) it often induces signifi- cant pruning due to mustplay (Hayward et al. 2004); and (3) it induces simplified Hex graph where more accurate evaluations - e.g., electronic resistance - can be extracted from. Yet, as suggested in (Berge) 1981, Haaward & Van Rijswijek| 2006), it is possible identify more prun- ing by manually analyzing the graph properties of Hex; the study of inferior cells are towards this end. Notably, both pruning from H-Search and inferior cells are provably correct knowledge to the game, thus optimal strategy is always preserved. Continual effort in identifying more local in- ferior patterns (Henderson) 2010). better H-Search implementations (Pawlewicz et al.] (015), and better parallel exhaustive search (Pawlewicz & Hayward 2013) have subsequently led to board sizes from 7x7 (Hayward et al.) 2004), 8x8 (Henderson et al., 2009), and 9x9 (Pawlewicz & Hayward, 2013) being solved] These knowledge computation have been implemented in an open source codebase benzene (Arneson et al. 2010b). For playing, applying H-search and inferior cells im- proves the application of MCTS to the game, resulting program MoHex (Arneson et al.. 2010a) (available inside of benzene). Like Go, later improved version MoHex 2.0 (Huang et al. 2013 Pawlewicz et al. 2015) add more accurate heuristic knowledge to in-tree selection and leaf evalua- tion by pastern-based Monte Carlo playout. Recent MoHex versions further incorporate knowledge due neural networks in its node selection and evaluation (Gao et al., [017a (018b) adding neural network for both player and solver leads to neurobenzene where how to train these neural nets configurable, and we shall use this platform to conduct our study.
4.2 IMPLEMENTATION
We focus on small board size 9x9 Hex, taking advantage of the existing solver (Henderson,) 2010 Pawlewicz et al. 2015 Pawlewicz & Hayward (2013) to measure the progress of AlphaZero learn- ing. However, since H-search and inferior cell analysis could induce too much pruning on such small board size, we configure MoHex to turn off these knowledge computation during training, simulating pure MCTS player whose performance is majorly influenced by the quality of the neu- ral net knowledge learned from playing data. We use synchronous implementation that follows the rightmost workflow in Figure[[ We use the neural network architecture exactly as in (Gao et al., 2018b), consisting of 10 residual blocks where each block is built with two convolutional layers each is with 32 x 3 filters. See Appendix for implementation details and parameter choices.
4.3 TEST DATA
To measure the progress of neural net learning, we use two datasets.
Note that solving 9x9 requires months of parallel computation

---

Under review as a conference paper at ICLR 2020
Figure 2: AlphaZero-2HNN versus AlphaZeeo-3HNN learning progresses measured in training time. MCTS-3HNN performs 800 iteration MCTS faster than MCTS-2HNN because it uses the default expand threshold of 10, while MCTS-2HNN uses 0. Consequently, 3HNN leads to faster AlphaZero learning than that of 2HNN. For test set Te, because final move is random, we only test the value error on these examples (rightmost).
A publicly available 9x9 Hex game produced by MoHex 2.0 in relatively strong set- ting (Gao et al. 2018a). In such configuration, due to extensive H-Search and inferior cell pruning, the player is already close to optimal and can usually conduct several hundred thousand simulations of MCTS on 9 x 9 board size. The test set contains 149362 examples. We call this test set Tí. We use this set of examples to test the policy-head by measuring the move prediction accuracy, state- and action-vall -value heads by measuring the mean squared error (MSE).
A set of randomly sampled but perfect labeled (using the solver in benzene) game states. This set contains 8485 examples. We call this set T2. This dataset is only used to measure the MSEs of state- and action-value heads, because measuring move prediction accuracy moves produced by random player is meaningless.
4.4 RESULTS
To distinguish, 2IphaZero-2HNN and AlphaZero-3HNN for AlphaZero learning with two- and three-head architecture respectively. We first use the following parameter choices:
The maximum number of MCTS simulations per move is set to Mmmtts = 800 for both A AphNaZero-2HNN and AlphaZero-3HNN.
For in-tree selection, as suggested in Gao et al. 201 7a; 2018b), we use the default method that uses a weighted combination of prior probability of neural net, RAVE and UCT explo- ration term, where the weight for prior probability Gpb 2.47 (served as Cpuct when PUCT formula is used).
Dirichlet noise parameter a = 0.03 as AlphaGo Zero and AlphaZero for Go.
We set move selection dithering threshold rj = 10 due to the short game length in 9x9 Hex.
For AlphaZero-3HNN, we use the default expansion threshold nıah = 10; for AlphaZero- 2HNN, f4.h = 0 as in AlphaZero.
We then run 1ophaZero-2HNN and AlphaZero-3HNN both for 100 hours. Figure 2 shows the comparative results. For policy learning, both methods achieved similar prediction accuracy on the game examples (Ti) produced by strong search algorithms. For the value learning, AlphaZero- 3HNN learned much faster than AlphaZero-2HHe ZHNN both on games states produced by strong player (Ti) as well as examples produced by random play (72). On Tz, there is gap between the accuracy of action-value and state-value heads; this suggests that the action-valle learning is correlated to policy learning.
From Figure 2 we also observe that one iteration AlphaZero-2HND is about to 6 times slower than AlphaZero-3HN The reason is that the MCTS-2HNN builds much deeper search tree than MCTS- (that uses default expand threshold of 10) using the same 800 simulations. We there- fore conduct second using the following parameter specifications:
7

---

Under review as a conference paper at ICLR 2020
2500 2003
Figure 3: AlphaZero-2HNS NHNN versus AlphaZZro-33NN. The only difference is that MCTS-315 --NNN
used Tmmtt 800 with default expand threshold of 10, while MCTS-2HNN is with Ameta = 160 with expand threshold of 0. AlphaZero-3HNN and AlphaZeno-2HNN respectively took 90 and 97
hours to complete 80 iterations.
Mou (16))
Proit Aerara
Dandiaa
Decelao:
Figure 4: AlphaZero- versus AlphaZero-3HHB -HNN. R2 in 3HNN is removed.
We reduce nmets to 160 reduce computation time in each iteration.
To further encourage exploration, move selection dithering threshold 1 30.
We change the in-tree selection formula to PUCT and set Cpuct = 1.5 according to (Tian et al. 2019).
We set the Dirichlet noise parameter 0.15 as observe that the branching factor of 9x9 Hex is about 5 times smaller than 19x 19 Go.
Figure 3 red) shows the results obtained by AlphaZero-2HNN learning. In this setting, A AlphaZero-HHN learned much better state-value head: it achieved better results than the in Figure[2] However, AlphaZero- 2HNN Figure was using search parameters that are largely different from the AlphaZero-3HNN in Figure 2 To have fair comparison, we then train AlphaZero33HN using the same in-tree formula and Cyuct 0.15, 0.15, and include the result to Figure 3
AphaZero-3HNN and AlphaZero-2HNN
77 30 and predictions than AlphaZero-2HNN on both test sets. For the move prediction accuracy, we found that while the top identical accuracy of 91 %. Since the moves of MoHex2.0 have no guarantee of being optimal, is still not clear if the 2HNN models are stronger.
Maaaaerro-HNN did not obtain better
accuracy of 3HNN is lower, for k
3HNN and 2HNN have almost
We suspect that the reason for the worse value prediction of 3HNN is that the R2 term in equation 3 assumes the player always taking the best move in the training games. This does not hold as there is a dithered selection (1) = 30). To see the effect of R22, we conduct another training of AlphaZero- BHNN using the same parameters as in Figure 3 except that R2 is removed. The obtained result and its comparison to AlphaZero- - is shown in Figure 4 which shows that the state-value head of 3HNN converged to smaller errors as measured on Tj and Tz- The side effect is that, on Tz, the gap between state- and action-value predictions increased.
4.5
MATCH PERFORMANCE
We now directly compare the strength of MCTS-2HNN and MCTS-3HNT using the models in Figures and
8

---

Under review as a conference paper at ICLR 2020
70-
MCTS-3INN ielld vs MOTS 1s MITS-SINNN-E 160 vss MCTSS -1UNN-166 50%%
MCTS-3HNN- 160 vs MCTS-2HNN-JO
50%
ha
m 11 1//
Win
Win
04-
10
20
90
60
50
20
70
Iteration
Fteratonn
(a) Using models as in Figure
(b) Using models as Figure
Figure 5: Match results of 3HNN vs 2HNN. SCCTS-HHNN-I--O means MCTS-3HNN using 1s per move with default expand threshold of 10. per move with expand threshold of O. MCTS-2HNN/HHNN. 160 means using 2HNN/3HNN with 160 simulations and expand threshold of 0. Taking the last 20 iteration results, we report the mean win-rates for MCTS-3HNN-1S-44
means using 2HNN with 1s
(left figure), and MCTS-3HNN-160 (right) are respectively 52.73 + 1.04, 51.25 + 1.05 and 56.54 + 0.83, with 95% confidence. For MCTS, as in training, we use Cpact 1.5 but set 77 0.
MCTS-3HNN
Following a common practice (Anthony et al. 2017 Gao et al. 20188) Anthony et al., 2019 Huang al. 2013) in Hex, we conduct the match by iterating all opening moves. Thus, each match consists of 162 games (for each opening cell, play 2 games with each player starts first and second with no swap rule). Figure 5 contains the results of 3HNN when competing against 2HNN; it shows that, overall, MCTS-3HND defeated MCTS-2HNN In Figure Sal MCTS-3HNN-e10 seems obtained better result than let al, 2018b): delayed node allows more simulations of MCTS and consequently it can lead to better move selection for playing. For Figure 5b) the advantage of 3HNN became visible after around iteration 50, consistent with Figure 4 Note that the test by iterating all openings tend to skew the win-rate 50% because some openings are very weak or very strong for the first-player: for Figure if taking only 5 openings (a2... a6) for the test, the mean win-rate of MCTS-3HNN 160 becomes 60.46 4.37 (95% confidence).
this discovery is consistent with the result reported in (Gao
Finally, we test our final iteration-88 80 models of 3HNN and 2HNN by playing against MoHex 2.0. The same as (Anthony al. 2019), we MCTS-2HNT and MCTS-3NN use Nynets 800, and MoHex 2.0 use Tomets 10000. MCTS-3HNN and 2HTNN respectively achieved mean win- rates 91.57 + 1.8 and 89.7 + 3.317 (both with 95% % confidence); see Table In comparison, the Zero implementation PGS-EXIT in (Anthony et al. 2019) achieved a win-rate of 58% against MoHex2.0- 10000 for 800-simulation search. This indicates the high quality of our AlphaZero implementation for both 3HNN and 2HNN
5 RELATED WORK
The high demand on computation makes reproducing the result or studying the behavior of AlphaGo Zero and AlphaZero significantly difficult. Meanwhile such studies are highly relevant, because the understanding for most parts of these algorithms is still in intuition-level. Following research projects to such end include Leela Zero (Pascutto 2018). and ELF OpenGo (Tian et al. (219)) they either use crowd-source for computation or have accessible to thousands of GPUs for parallel selfplay game generation. Some discoveries reported in (Tian et al. 2019) are: (1) the final perfor- mance is almost always bounded by the neural network capacity, i.e., improvement can always be achieved by increasing neural network size as more games are accumulated; (2) the PUCT param- eter in PV-MCTS seems important; (3) learning (strength of PV-MCTS) has high variance even in
Fach of these openings took more than month solve (Pawlewicz Hayward 2013).

---

Under review as a conference paper at ICLR 2020
the later stage; (4) selfplay Elo (Elo 1978) improvement does not always correspond to real playing improvement of the MCTS player the monotonic increase of Elo score did not capture the fluctu- ation in strength; and (5) an important tactic called "Ladder" in Go was never fully master even by increasing MCTS iterations.
AlphaZero style learning has also been studied in games other than Go, chess and Shogi. In par- ticular, in the game of Hex, [Anthony et al.] (2017) proposed an expert iteration paradigm, applied to 9x9 Hex, and obtained PV-MCTS player that is able to defeat 2011 I version of MoHex (Ar- nesso et al. 2010a); the subsequent work (Anthony et al. 2019) explored other search algorithms in expert iteration framework, though on 9x9 Hex these algorithms did not produce results better than an AlphaZero implementation that uses MCTS as the search algorithm. Using policy net- work for move selection and value network for move evaluation, Takada et al.) (2019) proposed minimax search algorithm and applied it to the game of Hex; starting from random weights, by training the neural network solely using search-based reinforcement learning for a few months, the final minrch based player DeepEzo is able to defeat MoHex 2.0 on 13x 13 in a large margin with 30 seconds per move. However, in formal tournament competition, DeepEzo failed to defeat recent version MoHex3HNX that uses three-head neural network whose weights were trained on games produced by MoHex 2.0 as well as MoHex3HNN selfplay; see (Gao et al. 2019). Model-free reinforcement learning have also bee applied to the game of Hex, but they did not produce competi- tive playing strength with search algorithms; see (Young et al.) 2016) Gao et al. 20). ) In the game of NoGo, Lan et al. (2019) explored the idea of dynamically increasing neural network size during the process of AlphaZero training, and showed that a Monte Carlo tree search with multiple small and large networks could accelerate the learning.
CONCLUSIONS
Using the game of Hex as an object, we have presented an empirical study on synchronous Alp- haZero learning with two- and three-head neural networks. Our results suggest that the three-head network is indeed compatible with the zero-style closed loop learning paradigm, consistent to the conjecture made in (Gao al. (018b);; moreover, in the game of Hex, the new architecture also enabled faster search and yielded better performance.
On the other side, the adjustment we have made on search for using the new action-value head is rather limited: only a non-zero expansion threshold is adopted to take advantage of the available action-value estimates. Other opportunities, such as combining the estimates of state-value and action-value heads, could be studied. Furthermore, it is not clear if MCTS is the best choice for search at all. Unlike a:3, MCTS does not directly apply hard-min and -max upon estimated value functions which tends to amplify the approximation error. Early research (Nau 1982, Pearl 1983) have also suggested some alternative methods that do not have such issues. For example, the product propagation (PP) (Pear] 1983) have long been conjectured as promising alternative (or better than?) to minimax, while in practice PP requires to estimate the probabilities of win/lose which are not instantly available. Previous studies (Kaindl al., 2015) typically normalize a static evaluation function into probabilistic estimates, and in practice if PP is superior to minimax is unclear. It remains to experimentally verify if neural networks could be used for provide better probabilistic estimates.
Playing and solving algorithms for two-player games have seen separate developments. While early work in playing majorly focus on more memory-efficient depth-first procedures that use a and B bounds for pruning, the development of solving algorithm (e.g., proof number search (Allis) 1994)
quickly switched to best-first paradigm as they are more tolerant to heuristic estimates backpropa- gated from leaf evaluations. MCTS is a best-first search algorithm that has been extensively used for playing but not solving, while proof number search (Allis et al. 19944) variants (e.g., (Gao et al.) 2017b)) are good at fast-solving but do not perform well at playing games. Unifying playing and solving algorithms that can fast solve game states given sufficient computation time and memory while also can provide high-quality moves given limited computation budget is worth further in- vestigating. More preferably, such an unified algorithm should be compatible with AlphaZero style learning, therefore enabling improved performance in both playing and solving as neural network learning progresses on.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Victor Allis, Maarten van der Meulen, and H Jaap Van Den Herik, Proof-number search. Artificial Intelligence, 1611), 1994.
LV Allis. Searching for solutions in games and artificial intelligence. PhD thesis, Universiteit Maastricht, 1994.
Vadim V Anshelevich. Hexy wins hex tourmamen.. ICGA Journal, 23(3):181-184, 2000.
Vadim V Anshelevich. A hicrarchical approach to computer Hex. Artificial Intelligence, 134(1): 101-120, 2002.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. arXiv preprint arXiv: 1705.08839, 2017.
Thomas Anthony, Robert Nishihara, Philipp Moritz, Tim Salimans, and John Schulman. Pol- icy gradient search: Online planning and expert iteration without search trees. arXiv preprint arXiv:1904.03644 2019.
Broderick Arneson, Ryan B Hayward, and Philip Henderson. Monte carlo tree search in hex. IEEE Transactions Computational Intelligence and Al in Games, 2(4):251-258 2010a.
Broderick Ameson, Philip Henderson, Jakub Pawlewicz, Aja Huang, and Chao Gao. benzene- vanilla-cmake, http:////ggtttt.occccc/////////e aa la-cmake) 2010b. Ac- cessed: 2019-08-11.
Yoshua Bengio. Learning deep architectures for ai. Foundations and trends in Machine Learning, (1):1-127, 2009.
Claude Berge. Some remarks about a hex problem. The Mathematical Gardner, pp. 25-27. Springer, 1981.
Dimitri P Bertsekas. Approximate policy iteration: A survey and some methods. Journal of Control Theory and Applications, 93):310-333, 2011.
Michael Buro. From simple features to sophisticated evaluation functions. International Confer-
ence on Computers and Games, pp. 126-145. Springer, 1998.
Murray Campbell, A Joseph Hoane, and Feng-hsiung Hsu. Deep Blue. Artificial intelligence, 134 (1-2):57-83, 2002.
Guillaume M JB Chaslot, Mark HM Winands, JAAP VAN DEN HERIK, Jos WHM Uiterwijk, and Bruno Bouzy. Progressive strategies for monte-carlo search. New Mathematics and Natural Computation, 4(03):343-357, 2008.
Christopher Clark and Amos Storkey. Training deep convolutional neural networks to play Go. In International Conference Machine Learning, pp. 7766-1774 2015.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2433-2537 101
Rémi Coulom. Efficient selectivity and backup operators in Monte-Carlo tree search. In Interna- tional Conference on Computers and Games, pp. 72-83. Springer, 2006.
Rémi Coulom. Computing Elo ratings of move patterns in the game of Go. In Computer Games Workshop, 2007.
Arpad E Elo. The rating of chessplayers, past and present. Arco Pub., 1978.
Jonathan St BT Evans and Keith E Stanovich. Dual-process theories of higher cognition: Advancing the debate. Perspectives on psychological science, 8(3): 2013.
C. Gao, R. Hayward, and M. Müller. Move prediction using deep convolutional neural networks in hex. IEEE Transactions on Games, 11( 2017a.

---

Under review as a conference paper at ICLR 2020
Chao Gao, Martin Müller, and Ryan Hayward. Focused depth-first first proof number search using volutional neural networks for the game of hex. In Proceedings of the Twenty-Sixi International Joint Conference on Artificial Intelligence, IJCAI-17, pp.
2017b.
Chao Gao, Martin Mueller, and Ryan Hayward. Adversarial policy gradient for alternating markov games. In International Conference on Learning Representations, 2018a.
Chao Gao, Martin Müller, and Ryan Hayward. Three-head neural network architecture for monte carlo tree search. In !JCAI, pp. 3762-3768, 2018b.
Chao Gao, Kei Takada, and Ryan Hayward. Hex 2018: Mohex over deepezo. JOURNAL, 41(1):39-42, 2019.
Sylvain Gelly and David Silver. Combining online and offline knowledge in UCT, In Proceedings of the 24th international conference Machine learning, pp. 273-280. ACM, 2007.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649, IEEE, 2013.
Larry R Harris. The heuristic search and the game of chess a study of quiescence, sacrifices, and plan oriented play. In Proceedings the 4th international joint conference Artificial intelligence- Volume 1, pp. 334-339, Morgan Kaufmann Publishers Inc., 1975.
Ryan Hayward. Six wins Hex tournament. ICGA Journal,
2006.
Ryan Hayward, Yngvi Björnsson, Michael Johanson, Morgan Kan, Nathan Po, and Jack van Rijswi- jek. Solving 7x7 Hex: Virtual connections and game-state reduction. In Advances in Computer Games, pp. 261-278. Springer, 2004.
Ryan B Hayward and Bjarne Toft. Hex: The Full Story. CRC Press, 2019.
Ryan Hayward and Jack Van Rijswijck. Hex and combinatorics. Discrete Mathematics, 306 (19-20):5515-2538 2006.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intel- ligence, 2018
Philip Henderson, Broderick Arneson, and Ryan Hayward. Solving 8x8 Hex. Proc. IJCAI, volume 9, pp. 505-510. Citeseer, 2009.
Philip Thomas Henderson. Playing and solving the game of Hex. PhD thesis, University of Alberta, 2010.
Shih-Chiel Huang, Broderick Arneson, Ryan B Hayward, Martin Müller, and Jakub Pawlewicz. Mohex 2.0: a pattern-based MCTS hex player. In International Conference on Computers and Games, pp. 60-71. Springer, 2013.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint, 2016.
Hermann Kaindl, Helmut Horacek, and Anton Scheucher. Product propagation: backup rule better than minimaxing? IEEE Transactions Computational Intelligence and Al in Games, 12):109- 2015.
Donald E Knuth and Ronald W Moore. An analysis of alpha-beta pruning. Artificial intelligence, 6 (4):293-326, 1975.
Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Care Carlo planning. In European conference machine learning, pp. 282-293. Springer, 2006.
12

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Li-Cheng Lan, Wei Li, Ting-Han Wei, I Wu, al. Multiple policy value monte carlo tree search. In IJCAI, 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):46-4444 2015.
Chris J Maddison, Aja Huang, Ilya Sutskever, and David Silver. Move evaluation in Go using deep convolutional neural networks. In International Conference on Learning Representations, 2015.
John McCarthy, Marvin L Minsky, Nathaniel Rochester, and Claude E Shannon. A proposal for the dartmouth summer research project artificial intelligence, august 31, 1955. Al magazine, 27 (4):12-12, 2006.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529-333 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillierap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning, pp. 1928-1937, 2016.
Martin Müller. Computer go. Artificial Intelligence, 134(1-2):145-177 2002.
2002.
John F Nash. Some games and machines for playing them. 1952.
Technical report, Rand Corporation,
Dana S Nau. An investigation of the causes of pathology in games. Artificial Intelligence, 19(3): 257-278, 1982.
OpenAL Openai baselines:
ACKTR and A2C. https://ppena..comm/bloo phselinnssacktr--222 2017. Accessed: 2019-08-30.
Gian-Carlo Pascutto. Leela zero.
-ero
2018. Accessed: 2019-08-18.
Jakub Pawlewicz and Ryan B Hayward. Scalable parallel DFPN search. In International Conference on Computers and Games, pp. 138-150. Springer, 2013.
Jakub Pawlewicz, Ryan Hayward, Philip Henderson, and Broderick Arneson. Stronger Virtual Connections in Hex. IEEE Transactions on Computational Intelligence and Al in Games, 7(2): 156-166, 2015.
Judea Pearl. Asymptotic properties of minimax trees and game-searching procedures. Artificial Intelligence, 14(2):113-138, 1980a.
Judea Pearl. Scout: A simple game-searching algorithm with proven optimal properties. In AAAI, pp. 143-145, 1980b.
Judea Pearl. On the nature of pathology in game searching. Artificial Intelligence, 20(4):427-453, 1983.
Judea Pearl. Heuristics: intelligent search strategies for computer problem solving. Aadison-Wesler Pub. Co., Inc., Reading, MA, 1984.
Aske Plaat, Jonathan Schaeffer, Wim Pijls, and Arie De Bruin. Best-first fixed-depth minimax algorithms. Artificial Intelligence, 87(1-2):255-29 1996.
Christopher Rosin. aulti- bandits with episode context. Annals of Mathematics and Artifi- cial Intelligence, 61(3):203-22 2011.
13

---

Under review as a conference paper at ICLR 2020
A. L. Samuel. Some studies in machine learning using the game of checkers. IBM J. Res. Dev., (3):210-229, July 1959. ISSN 0018-8646, doi: 10.1147/rd.33.02100 URL http: / //dx/xxxppp doi. org/10. 11..1..111111 1ttp//.11177...3.... 33.0210
Arthur L. Samuel. Some studies in machine learning using the game of checkers. ii ii-recent progress. IBM Journal of research and development, 11(6):601-617, 1967.
Jonathan Schaeffer, Joseph Culberson, Norman Treloar, Brent Knight, Paul Lu, and Duane Szafron.
A world championship caliber checkers program. Artificial Intelligence, 53(2-3):273-289) 1992.
Claude E Shannon. Computers and automata. Proceedings of the IRE, (1(110):134411111 1241, 1953.
David Silver and Gerald Tesauro. Monte-carlo simulation balancing. In Proceedings of the 26th
Annual International Conference on Machine Learning, pp. 945-952. ACM, 2009.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwisser Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint 171::177,. 2017a.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354-335) 2017b.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanetot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi. and go through self- oolf-play. Science, 362(6419):1140 1144, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Preliminary Draft, MIT press Cambridge, second edition, 2017.
Kei Takada, Hiroyuki lizuka, and Masahito Yamamoto. Reinforcement learning create value and policy functions using minimax tree search in hex. IEEE Transactions on Games, 2019.
Gerald Tesauro. Temporal difference learning and TD-Gammon. Communications of the ACM, 38 (3):58-68, 1995.
Yuandong Tian and Yan Zhu. Better computer Go player with neural network and long-term predic- tion. In International Conference on Learning Representations, 2015.
Yuandong Tian, Jerry Ma, Qucheng Gong, Shubho Sengupta, Zhuoyuan Chen, James Pinkerton, and Larry Zitnick. Elf opengo: an analysis and open reimplementation of alphazero. In ICML, pp. 6244 6253, 2019.
Jack Van Rijswijek. Computer Hex: Are bees better than fruitflies? Master's thesis, University of Alberta, 2002a.
Jack Van Rijswijck. Search and evaluation in hex. Master of science, University of Alberta, 2002b.
Kenny Young, Gautham Vasan, and Ryan Hayward. Neurohex: A deep q-learning hex agent. In Computer Games, pp. 3-18. Springer, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
14

---

Under review as a conference paper at ICLR 2020
A MONTE CARLO TREE SEARCH WITH DELAYED NODE EXPANSION
Monte Carlo tree search (MCTS) best-search algorithm that repeatedly conducts the following steps:
1. Selection. Given existing search and single starting node sn, leaf node st. in the tree is selected by series of action-selection starting from an action.
2. Evaluation. The leaf node SL is evaluated, either by Monte Carlo playout or an external evaluation function (e.g., neural network).
3. Expansion. The leaf node 81, may expanded and so, all child nodes of sr. will be created and added to the search tree.
4. Backpropagation. The evaluation sr obtained at step is backpropgated to the search
tree.
complete execution of these steps is called one simulation (or iteration) of MCTS. Delayed node expansion refers to an implementation of step 3. where the leaf node sL will be expanded only when it has been visited at least nesh number times. Conversely, the non-delayed expansion = 0.
implementation expands when it has been selected the first time, i.e., equivalently,
three-head network can used for delayed node expansion, because every leaf node the search tree contains an value estimate that was stored upon its creation.
B IMPLEMENTATION DETAILS
For understanding convenience, we document how to configure the MCTS in MoHex for various use. Most of them have been implemented in benzene except that those involving neural networks and AlphaZero are newly added.
B.1 MOHEX TURN OFF KNOWLEDGE COMPUTATION
param mohex perform pre search 0 param_mohar mhowledge_threshhoe "999999"
B.2 SETTING IN-TREE SELECTION FORMULA
Two formulas have been implemented, the RAVE formula:
N(s) + wR(ss + cpb (1 - (((s.)))) (1..0)(s +00 N(ssa))
p(s.a)
(N(s,0)+1
and a PUCT formula:
P(s E ()
1+ (8,
They can be configured using the following commands.
param_mohex use_rave
param_mohea progressive_biat 1.5
param_mohex htt_bbassconstaan 0.0
param_mohex root_Girichlet_prin 0.15
param_mohex moveselect count
param_mohts
dit.eecee_dtthertheehhhoddd 30
karam_noher extend unstable seasstable_serrchh
If use__m___ Lave is true, a in-tree selection using RAVE will be used, otherwise the PUCT formula will be used. e is set by uet bias constant. and w is dynamic RAVE weight. The prior probability P(s, a) will set to P(s, a) + (1 - P(,, a) + eDir(a: where = 0.25 root_dirichlet prio * otherwise dirichlet noise will be used.
15

---

Under review as a conference paper at ICLR 2020
B.3 MOHEX TURN OFF PLAYOUT
paran_mohex use_playout_conster 0.0
By such, MoHex solely uses neural net estimates for leaf node evaluation.
B.4 MOHEX LOAD NN MODELS
mn mnad path to nn _o__nn__mmm on_modee
By such, MoHex can load trained neural net model into its search.
B.5 MOHEX SETS NUMBER OF ITERATIONS
param_mohex max_games 800
By such, MoHex will conduct 800-iteration search.
B.6
MOHEX SETS EXPAND THRESHOLD
param_mohex expand_threshole 0
By such, MoHex will expand leaf at its first visit.
B.7 NEURAL NET TRAINING
The neural network trained using stochastic gradient descent with momentum 0.9. The batch size is 128, initial learning rate is 0.005, with an annealing factor 0.9 each iteration until a minimum learning rate 0.000005 reached. Note that initial learning rate is order of smaller than that of AlphaZero because our batch-size is 32 times smaller. At each iteration, we train the neural network for 5 epochs and then pass the obtained model MCTS workers for next iteration game generation. L2 regularization is set to c 0.0001.
The neural network we used is exactly the same in (Gao et al. 2018b). containing 10 residual blocks, each with 32 filters.
B.8 MCTS SELFPLAY
We use 60 parallel MCTS workers for selfplay, each producing 200 games each iteration. We reuse the subtree in each selfplay game (by configuring paran_mohex Go, Shogi and Chess, for N X Hex, the maximum game length is N therefore we do not specify resign threshold during selfplay, either maximum game length. To accommodate the swap rule in Hex, for each selfplay game, Black starts first by uniformly sampling a cell on the board.
reuse_subte
1). Unlike
B.9 COMPUTATION HARDWARE
We use the same hardware for neural network training and selfplay game generation. Specifically, the computer has an Intel(R) Xeon(R) CPU ES-2690 v4 @ 2.60GHz with 56 processors with 500 GB RAM, and 6 Tesla P100 GPUs each with 16 GB RAM. During selfplay, the parallel workers are evenly distributed the available GPUs.
B.10 TURN OFF DATA AUGMENTATION IN 3HNN WHEN SELECTED MOVE IS NOT THE BEST
[A(=]] EarEA(S) + q(s,aa))² assumes that in the selfplay game both players are selecting the best actions produced by search. This assumption does not hold when dithering threshold n O. Instead of just removing R2, anther way is remove R2 in training 3HNN when the move played in the example game is not with the highest move probability. This scheme not tested in the paper.
The data augmentation error in Eq. i.c.. max(-c..0)
16

---

Under review as a conference paper at ICLR 2020
Table 2: Detailed match results against MoHex 2.0. Each of games were played by iterating all opening moves; each opening is tried twice with the competitor starts first and second, therefore each match consists of 162 games. We use the final iteration-80 models for 2HNN and 3HNN from Figure The overall results are calculated with 95% confidence. ACTS-22NN and MCTS-3HN used 800 simulations per move with expand threshold of MoHex 2.0 used default setting with 10,000 simulations per move.
Set
3
Overall
Player
MCTS-2HNN 2HNN-800
86.4%
92.0%
90.7%
89.7%
3.32
MCTS-3HNN-800
93.2%
91.4%
90.1% 1
91.57 + 1.77
B.11 MATCH CONFIGURATION
For MoHex 2.0, we use the default setting but restrict to 10000 iteration MCTS, which is equivalent to the following:
param param_mohex use_rave rave param_mohex sec_bias__oonsten 0.22 param_mohex progressive_bias 2.47 param_mohex max_games 10000
For MoHex with 2HNN 3HNN, use the following:
paran_mohex use_rave 0
param_mohex jet_bias_constant 0.0
param_mohex szogressive_bias 1.5
param_mohex max_games n_mets param_mohex use_playort_constan
0.0
The difference for 2HNN and 3HNN is that, for 2HNN, we always set param_mohex sxpand_threshol 0, while for 3HNN, other non-zero thresholds are also used specified in the context. For the test between 2HNN and 3HNN, numot s-mnttssl66 10.0. For test against MoHex 2.0,
m000.
C VISUALIZATION
We show visualization of the learned policy and action-value heads using HexGUI
accesssd by httpt:ttpp........t.. / thub. htapbb/.oommrraabaaaaaa//r/ggggs
17

---

Under review as a conference paper at ICLR 2020
Figure 6: The left subfigure is obtained by inferring the 3HNN model from iteration 80 as in Fig- ure 4 The right subfigure is the ground truth of each cell (black means first player win, assuming black plays first). For each cell, the upper number is the prior probability from policy head, the lower number the action estimate from action-value head. Note that the action-value is with re- spect to the player to play after taking that action. The policy head provided non-zero probabilities only three strongest central openings, while the action-value head provides set of estimates that are mostly consistent with the ground truth of each opening move.
18


Reviews:
[{"review_id": "r1xuzKwGqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies the three-head neural network architecture as well as the corresponding training loss proposed in (Gao et al., 2018b) to alphazero style learning of the Hex game. The paper is mainly an empirical study, and shows that the architecture leads to faster and better learning results for Hex. The evaluation is done on two datasets, one with examples from near-optimal players produced by MoHex 2.0, and the other from randomly sampled but perfectly labelled examples generated by benzene. Performance improvement is evaluated from several different perspectives, including state-value errors, action-value errors and policy prediction accuracies. Finally, the match performance is also reported for competing with MoHex 2.0, one of the state-of-the-art agent for Hex.\n\nGenerally speaking, the paper does a good job in introducing and analyzing the structure of the alphazero learning scheme and the related alphago and alphago zero schemes, and the experiments within the scope of Hex is relatively thorough and the performance improvement is consistent and convincing. \n\nHowever, the description of the three-head neural network in Section 3 is too brief, and without looking at the original paper (Gao et al., 2018b), it is quite hard to understand the motivation of the objectives (especially the definitions and explanations of R1, R2 and R3). \n\nAdditionally, the challenge of applying three-head neural network architecture in the alphazero learning setting is almost not mentioned. In particular, what are the modifications needed compared to the original work (Gao et al., 2018b)? The authors may want to explain clearly how the training scheme is different, and clearly state what the detailed neural network architecture (at least in the appendix) used is, and how they are different from the original alphazero paper and (Gao et al., 2018b). Without these explanations, the significance of the paper would be largely limited to coding and engineering efforts (which are also valuable but not that much in the research sense).\n\nAnother related issue of this paper is that it is not clear (at least to me, who know little about the Hex game) how difficult it is to tackle Hex (compared to Go, Shogi and chess, etc.). The authors may want to elaborate more on this as well to further showcase the significance of the work.\n\nFinally, there are also some inconsistency in the hyper-parameter choices and architecture design. In particular, it is not clear why the authors choose the expansion threshold to 0 in the match performance part, whereas the authors use threshold 10 elsewhere. The turning on and off of the data augmentation in 3HNN in different experiments mentioned in the appendix are also not well explained. \n\nNevertheless, I still value the paper's effort and success in applying a newly proposed approach for a relatively challenging real-world game problem, despite the issues about experimental design and writing mentioned above.\n\nSome minor suggestions: the title of the rightmost plots should better be \"perfectly labelled examples\" instead of \"perfect examples\", and the authors may want to make it clearer which plot corresponds to dataset T1 and which corresponds to T2.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eKL42R_H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper applies three-head neural network (3HNN) architecture in AlphaZero learning paradigm. This architecture was proposed in [1] and the paper builds upon their work. In AlphaGo and AlphaZero 2HNN is used, which predicts policy and value for a given state. 3HNN also predicts action-value Q function. In [1], the three new terms are added to the loss to train such a network, and the network is trained on a fixed dataset. The paper utilizes the same 3HNN idea with the same loss, and the contribution is that 3HNN is trained synchronously with MCTS iterations on an updating dataset (“AplhaZero training style”). Learning speed of 3HNN is shown to be higher than that of 2HNN. The special attention is drawn to varying the threshold expansion parameter, as the 3HNN architecture allows to set it above zero, while 2HNN does not. The approach is demonstrated on the game of Hex. Results are presented on two test datasets: positions drawn from a strong agent’s games and random positions. Labels in both datasets are perfect, obtained by a special solver.\n\nI tend to reject the paper, because the demonstrated results suggest that the models were not tuned well enough. Indeed, the paper claims that the parameters were not tuned. The paper claims using threshold expansion > 0 to be one of the main advantages of 3HNN. However, the best model in the experiment is the one with the parameter equals zero. Overall, for a purely experimental paper, the experiments are too crude.\n\nMain argument\n1.\tIt seems that some of NN models didn’t learn at all:\n           • \tFigure 2, 2HNN model. MSE on both the left and right plots is not improving. Moreover, on the right plot it \n                fluctuates around 1, which is the performance of a random guess.\n           •\tFigure 3, right. MSE of the 3HNN model is not improving. Probably, random positions are too unnatural and \n                nothing similar is presented in the dataset drawn from MCTS iterations.\n2.\tSupmat reveals, that some of the models are in fact learned using the different loss than it is said in the paper. In particular, data augmentation term is sometimes on and sometimes off. A disabling scheme is suggested, depending on the dithering threshold and the number of the moves played before the state s. Some models use the scheme, for others the term is always on. This should be clearly stated in the main text, not in the supmat. \nAlso, there is an experiment in the supmat, when the data augmentation term is always off. The influence of this term is itself interesting, as it is one of the reasons 3HNN is learning q-function at all. However, introduction of this term itself is the contribution in [1]. I suggest to add to the main part of the paper the experiment, comparing three regimes: 1) with the scheme, 2) term always on and 3) term always off. In fact, it is almost done, as all three regimes are used in different figures, but somewhy the final comparison (with other parameters fixed) is not shown and partly concealed in the supmat. It could become methodological improvement of the paper over [1]. \n3.\tWhen the leaf node s is expanded, the v function of the new node s’ = s ∪ a is initialized to predicted q(s, a). It is one of the advantages of 3HNN and allows node expansion threshold. When s’ itself is expanded, how do you merge q(s,a) with backup values during mcts iterations?\n4.\tNothing is said about the architecture of NNs. If the representation of the state is the same as in [1], it should at least be mentioned.\n5.\tHow exactly does figure 2 shows that one iteration AlphaZero-2HNN is 5-6 times slower, than AlphaZero-3HNN. Probably the definition of data point in figure 2 is missing (e.g. one data point corresponds to one MCTS iteration).\n6.\tFigures 3 and 4 basically shows the same experiment, but with different models: AlphaZero-2HNN versus AlphaZero-3HNN with threshold 0 on Figure 3 and AlphaZero-2HNN versus AlphaZero-3HNN with threshold 1 on Figure 4. They could be united in one figure.\n7.\tResults from Figures 3 and 4 suggest, that 3HNN with threshold = 0 is better, than with threshold = 1. However, the paper claims setting threshold > 0 as an advantage. If it allows to save time (as each mcts iteration is faster), maybe they should be compared plotting time on x-axis (like in Figure 2)?\n8.\tPlease provide error bounds in table 2.\n\nAdditional arguments\nArgumentation presented in this section didn’t affect the score. However, it might improve the paper.\n1.\t3HNN predicts q(s,a), which should be equal to v(s’), where s’ = s ∪ a (state after action a is taken in state s). It would be interesting to see how condition q(s,a) == v(s ∪ a) holds during  3HNN models learning. It can be checked on a first dataset (drawn from games), or a special random dataset, containing consecutive positions, could be generated. Probably, this condition could potentially be an additional loss term.\n2.\tAlso, it would be interesting to see how the condition between p and q holds. The higher q(s, a), the lower should be p(a). There is an interesting illustrating figure 7 in supmat, it may be presented in the main text. Also, it could be interesting not only for the first move. \n3.\tThe supmat claims that the motivation for turning off data augmentation term is that it assumes that in the selfplay game both players are selecting the best actions produced by search. Is it connected with the fact, that in the game of Hex all states are either winning or losing, according to theorem proved by Nash? How does this term would work for games with a draw trend, for example chess? In chess, for a lot of states the “ground truth” v(s) would be close to zero and there is no action guaranteeing the win.\n4.\tThe paper claims “Elo (Elo, 1978) improvement does not always correspond to real playing improvement—the monotonic increase of Elo score did not capture the fluctuation in strength”. Citation needed, what fluctuation in strength is not captured? Is it specific to game of Hex? For example, Elo is used as the main measure of total agent strength in AplhaZero papers, as well as by chess community (both chess programs and human players).\n\nMinor comments\n1.\tPage 4, section 2.2: Even though ... . Our summarization -> Even though ... , our summarization.\n2.\tPage 4, table 1: mvoe -> move.\n3.\tPage 7, bullet point above section 4.4: perfect -> perfectly.\n4.\tPage 7, the lowest paragraph. “For the value learning, however, due to fast search, the AlphaZero-3HNN learned much faster than AlphaZero-2HNN both on games states produced by strong player (T1) as well as examples produced by random play (T2).” \nIt is confusing, it seems that 3HNN on datasets T1 and T2, however, it was only tested on these datasets.\n5.\tPage 8: imposing an an auxiliary task -> imposing an auxiliary task.\n6.\tPage 9: “produce playing strength significantly stronger than” – reformulate.\n\n[1] Chao Gao, Martin Müller, and Ryan Hayward. Three-head neural network architecture for monte carlo tree search. In IJCAI, pp. 3762–3768, 2018b.\n\n=====Post Rebuttal=====\nScore updated from 3 to 6.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgHu7bCKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed to use three-head network for AlphaZero-like training. The three-head network is used to predict policy, value and q function after an action is taken. While three-head network is presented by a prior work [1] and is learned via supervised learning on a fixed dataset, this paper mainly applies it to AlphaZero training for the game of Hex 9x9 and shows preliminary results. \n\nWhile the idea is interesting, there are many issues in the experiments and the conclusion is quite indecisive. So I feel that the paper is not ready for publication yet and thus vote for rejection. \n\nWhy we need expansion threshold n_th to be 10? If you keep visiting the same node without expansion, won’t the same node back-propagate the same value (or q) 10 times before expansion? If that’s the case, what’s the difference if we just back-propagate once? Note that if n_th = 0 then prediction of q(s, a) is no-longer necessary (except that predicts q(s, a) becomes an aux task during training, as mentioned in the caption of Fig. 3). \n\nFig. 2 shows that 3HNN trains faster than 2HNN. However, it looks like 2HNN and 3HNN show drastically different training curves, and are probably operating at different regions. In the text, the authors also acknowledge that one iteration of 2HNN is 5-6 times slower than 3HNN, since 2HNN builds a much deeper search tree. This bring about a question: is the performance difference due to unfavorable hyper-parameters on 2HNN (or other factors)? The paper doesn’t answer that. \n\nThe text claims that when n_th = 0, 3HNN performs better than 2HNN, however, the figure shows that 2HNN has lower or comparable MSE than 3HNN. The prediction accuracy is better, though. When n_th = 1, Fig. 4 shows that the 2HNN is doing comparable or better in terms of MSE and Prediction Accuracy than 3HNN (compared to perfect play). This somehow defeats the purpose of using the third head of q(s, a) that only helps when n_th > 0. \n\nIn Table 2, do you have standard derivation? Note that AlphaZero training is not that stable and the performance (in particular the initial performance since the performance might take off earlier or later) against a known bot can vary a lot, the difference between 56% and 63% can be purely due to noise. Also, how is the resulting model compared against MoHex-3HNN [1] and MoHex-CNN [2]? Note that MoHex-3HNN [1] shows 82.4% over MoHex 2.0 on 13x13, but is trained supervisedly, and Table 2 shows slightly better performance. So I am curious their performance comparison. \n\nMinor: \nThe term “iteration” seems to be defined twice with different meanings. It is defined as one MCTS rollout (see Appendix A) and also defined (in Fig 1) as one full synchronization of self-play and training (AlphaGo Zero setting). This causes a lot of confusions. I believe each dot in Fig. 2 is “iteration” in the AlphaGo Zero sense. \n\nFinally, although many hardware information is revealed in the appendix, maybe it would be better if the authors could reveal more details about their AlphaZero-style training, e.g., how long does it take for each move and for each self-play game? How long does it take to wait until all self-play agent returns all games? Is there any synchronization overhead? This could give the audience some idea about the computational bottleneck. \n\nFrom the current number, it seems that 60 self-play processes are run on 56 cores, and each AlphaGo iteration takes approximate 5 hours (read from Fig. 2) with 200 games per self-play process. Assuming there is no synchronization overhead and 1 core per self-play process, this yields 200 games/5 hours per core, which is 1.5 min (or 90s) per game. Since each game has 9x9 = 81 moves, this means that it costs ~1.1 sec per move. Is that correct? \n\n[1] Chao Gao, Martin Muller, and Ryan Hayward. Three-head neural network architecture for Monte Carlo tree search. In IJCAI, pp. 3762–3768, 2018.\n\n[2] Chao Gao, Ryan B Hayward, and Martin Muller. Move prediction using deep convolutional neural networks in Hex. IEEE Transactions on Games, 2017.\n\n=====Post Rebuttal=====\nI really appreciated that the authors have made substantial efforts in improving the paper and adding more experiments. However, the additional change makes the story a bit more convoluted. After substantial parameter tuning on the 2HNN side, It seems that 3HNN is only slightly better than 2HNN (Fig. 5 in the revision, > 50% winrate, but it is not clear how much ELO it is better). Unfortunately, after tuning, 2HNN actually shows comparable performance in terms of speed (updated Fig. 3 and 4, middle columns), which somehow tarnishes the claims of the paper that 3HNN is better than 2HNN. \n\nThe final performance against 10000-rollouts MoHex2.0 is 89.7% (2HNN) versus 91.6% (3HNN), so the performance is slightly better with 3HNN. This number is much better than previous works e.g., PGS-EXIT (Thomas et al. 2019). This indeed shows that the paper does a good job in terms of engineering and performance push (agreed with R1). In my opinion, the paper can be better rewritten as a paper that shows strong performance in Hex, compared to previous works, plus many ablation analysis. \n\nI keep the score. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provide an empirical study of the recent 3-head architecture applied to AlphaZero style learning. They thoroughly evaluate this approach using the game Hex as a test domain.

Initially, reviewers were concerned about how well the hyper parameters for tuned for different methods. The authors did a commendable job addressing the reviewers concerns in their revision. However, the reviewers agreed that with the additional results showing the gap between the 2 headed architecture and the three-headed architecture narrowed, the focus of the paper has changed substantially from the initial version. They suggest that a substantial rewrite of the paper would make the most sense before publication.

As a result, at this time, I'm going to recommend rejection, but I encourage the authors to incorporate the reviewers feedback. I believe this paper has the potential to be a strong submission in the future.


Decision: Reject