Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper applies the three-head neural network architecture as well as the corresponding training loss proposed in (Gao et al., 2018b) to alphazero style learning of the Hex game. The paper is mainly an empirical study, and shows that the architecture leads to faster and better learning results for Hex. The evaluation is done on two datasets, one with examples from near-optimal players produced by MoHex 2.0, and the other from randomly sampled but perfectly labelled examples generated by benzene. Performance improvement is evaluated from several different perspectives, including state-value errors, action-value errors and policy prediction accuracies. Finally, the match performance is also reported for competing with MoHex 2.0, one of the state-of-the-art agent for Hex.

Generally speaking, the paper does a good job in introducing and analyzing the structure of the alphazero learning scheme and the related alphago and alphago zero schemes, and the experiments within the scope of Hex is relatively thorough and the performance improvement is consistent and convincing. 

However, the description of the three-head neural network in Section 3 is too brief, and without looking at the original paper (Gao et al., 2018b), it is quite hard to understand the motivation of the objectives (especially the definitions and explanations of R1, R2 and R3). 

Additionally, the challenge of applying three-head neural network architecture in the alphazero learning setting is almost not mentioned. In particular, what are the modifications needed compared to the original work (Gao et al., 2018b)? The authors may want to explain clearly how the training scheme is different, and clearly state what the detailed neural network architecture (at least in the appendix) used is, and how they are different from the original alphazero paper and (Gao et al., 2018b). Without these explanations, the significance of the paper would be largely limited to coding and engineering efforts (which are also valuable but not that much in the research sense).

Another related issue of this paper is that it is not clear (at least to me, who know little about the Hex game) how difficult it is to tackle Hex (compared to Go, Shogi and chess, etc.). The authors may want to elaborate more on this as well to further showcase the significance of the work.

Finally, there are also some inconsistency in the hyper-parameter choices and architecture design. In particular, it is not clear why the authors choose the expansion threshold to 0 in the match performance part, whereas the authors use threshold 10 elsewhere. The turning on and off of the data augmentation in 3HNN in different experiments mentioned in the appendix are also not well explained. 

Nevertheless, I still value the paper's effort and success in applying a newly proposed approach for a relatively challenging real-world game problem, despite the issues about experimental design and writing mentioned above.

Some minor suggestions: the title of the rightmost plots should better be "perfectly labelled examples" instead of "perfect examples", and the authors may want to make it clearer which plot corresponds to dataset T1 and which corresponds to T2.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposed to use three-head network for AlphaZero-like training. The three-head network is used to predict policy, value and q function after an action is taken. While three-head network is presented by a prior work [1] and is learned via supervised learning on a fixed dataset, this paper mainly applies it to AlphaZero training for the game of Hex 9x9 and shows preliminary results. 

While the idea is interesting, there are many issues in the experiments and the conclusion is quite indecisive. So I feel that the paper is not ready for publication yet and thus vote for rejection. 

Why we need expansion threshold n_th to be 10? If you keep visiting the same node without expansion, won’t the same node back-propagate the same value (or q) 10 times before expansion? If that’s the case, what’s the difference if we just back-propagate once? Note that if n_th = 0 then prediction of q(s, a) is no-longer necessary (except that predicts q(s, a) becomes an aux task during training, as mentioned in the caption of Fig. 3). 

Fig. 2 shows that 3HNN trains faster than 2HNN. However, it looks like 2HNN and 3HNN show drastically different training curves, and are probably operating at different regions. In the text, the authors also acknowledge that one iteration of 2HNN is 5-6 times slower than 3HNN, since 2HNN builds a much deeper search tree. This bring about a question: is the performance difference due to unfavorable hyper-parameters on 2HNN (or other factors)? The paper doesn’t answer that. 

The text claims that when n_th = 0, 3HNN performs better than 2HNN, however, the figure shows that 2HNN has lower or comparable MSE than 3HNN. The prediction accuracy is better, though. When n_th = 1, Fig. 4 shows that the 2HNN is doing comparable or better in terms of MSE and Prediction Accuracy than 3HNN (compared to perfect play). This somehow defeats the purpose of using the third head of q(s, a) that only helps when n_th > 0. 

In Table 2, do you have standard derivation? Note that AlphaZero training is not that stable and the performance (in particular the initial performance since the performance might take off earlier or later) against a known bot can vary a lot, the difference between 56% and 63% can be purely due to noise. Also, how is the resulting model compared against MoHex-3HNN [1] and MoHex-CNN [2]? Note that MoHex-3HNN [1] shows 82.4% over MoHex 2.0 on 13x13, but is trained supervisedly, and Table 2 shows slightly better performance. So I am curious their performance comparison. 

Minor: 
The term “iteration” seems to be defined twice with different meanings. It is defined as one MCTS rollout (see Appendix A) and also defined (in Fig 1) as one full synchronization of self-play and training (AlphaGo Zero setting). This causes a lot of confusions. I believe each dot in Fig. 2 is “iteration” in the AlphaGo Zero sense. 

Finally, although many hardware information is revealed in the appendix, maybe it would be better if the authors could reveal more details about their AlphaZero-style training, e.g., how long does it take for each move and for each self-play game? How long does it take to wait until all self-play agent returns all games? Is there any synchronization overhead? This could give the audience some idea about the computational bottleneck. 

From the current number, it seems that 60 self-play processes are run on 56 cores, and each AlphaGo iteration takes approximate 5 hours (read from Fig. 2) with 200 games per self-play process. Assuming there is no synchronization overhead and 1 core per self-play process, this yields 200 games/5 hours per core, which is 1.5 min (or 90s) per game. Since each game has 9x9 = 81 moves, this means that it costs ~1.1 sec per move. Is that correct? 

[1] Chao Gao, Martin Muller, and Ryan Hayward. Three-head neural network architecture for Monte Carlo tree search. In IJCAI, pp. 3762–3768, 2018.

[2] Chao Gao, Ryan B Hayward, and Martin Muller. Move prediction using deep convolutional neural networks in Hex. IEEE Transactions on Games, 2017.

=====Post Rebuttal=====
I really appreciated that the authors have made substantial efforts in improving the paper and adding more experiments. However, the additional change makes the story a bit more convoluted. After substantial parameter tuning on the 2HNN side, It seems that 3HNN is only slightly better than 2HNN (Fig. 5 in the revision, > 50% winrate, but it is not clear how much ELO it is better). Unfortunately, after tuning, 2HNN actually shows comparable performance in terms of speed (updated Fig. 3 and 4, middle columns), which somehow tarnishes the claims of the paper that 3HNN is better than 2HNN. 

The final performance against 10000-rollouts MoHex2.0 is 89.7% (2HNN) versus 91.6% (3HNN), so the performance is slightly better with 3HNN. This number is much better than previous works e.g., PGS-EXIT (Thomas et al. 2019). This indeed shows that the paper does a good job in terms of engineering and performance push (agreed with R1). In my opinion, the paper can be better rewritten as a paper that shows strong performance in Hex, compared to previous works, plus many ablation analysis. 

I keep the score. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper applies three-head neural network (3HNN) architecture in AlphaZero learning paradigm. This architecture was proposed in [1] and the paper builds upon their work. In AlphaGo and AlphaZero 2HNN is used, which predicts policy and value for a given state. 3HNN also predicts action-value Q function. In [1], the three new terms are added to the loss to train such a network, and the network is trained on a fixed dataset. The paper utilizes the same 3HNN idea with the same loss, and the contribution is that 3HNN is trained synchronously with MCTS iterations on an updating dataset (“AplhaZero training style”). Learning speed of 3HNN is shown to be higher than that of 2HNN. The special attention is drawn to varying the threshold expansion parameter, as the 3HNN architecture allows to set it above zero, while 2HNN does not. The approach is demonstrated on the game of Hex. Results are presented on two test datasets: positions drawn from a strong agent’s games and random positions. Labels in both datasets are perfect, obtained by a special solver.

I tend to reject the paper, because the demonstrated results suggest that the models were not tuned well enough. Indeed, the paper claims that the parameters were not tuned. The paper claims using threshold expansion > 0 to be one of the main advantages of 3HNN. However, the best model in the experiment is the one with the parameter equals zero. Overall, for a purely experimental paper, the experiments are too crude.

Main argument
1.	It seems that some of NN models didn’t learn at all:
           • 	Figure 2, 2HNN model. MSE on both the left and right plots is not improving. Moreover, on the right plot it 
                fluctuates around 1, which is the performance of a random guess.
           •	Figure 3, right. MSE of the 3HNN model is not improving. Probably, random positions are too unnatural and 
                nothing similar is presented in the dataset drawn from MCTS iterations.
2.	Supmat reveals, that some of the models are in fact learned using the different loss than it is said in the paper. In particular, data augmentation term is sometimes on and sometimes off. A disabling scheme is suggested, depending on the dithering threshold and the number of the moves played before the state s. Some models use the scheme, for others the term is always on. This should be clearly stated in the main text, not in the supmat. 
Also, there is an experiment in the supmat, when the data augmentation term is always off. The influence of this term is itself interesting, as it is one of the reasons 3HNN is learning q-function at all. However, introduction of this term itself is the contribution in [1]. I suggest to add to the main part of the paper the experiment, comparing three regimes: 1) with the scheme, 2) term always on and 3) term always off. In fact, it is almost done, as all three regimes are used in different figures, but somewhy the final comparison (with other parameters fixed) is not shown and partly concealed in the supmat. It could become methodological improvement of the paper over [1]. 
3.	When the leaf node s is expanded, the v function of the new node s’ = s ∪ a is initialized to predicted q(s, a). It is one of the advantages of 3HNN and allows node expansion threshold. When s’ itself is expanded, how do you merge q(s,a) with backup values during mcts iterations?
4.	Nothing is said about the architecture of NNs. If the representation of the state is the same as in [1], it should at least be mentioned.
5.	How exactly does figure 2 shows that one iteration AlphaZero-2HNN is 5-6 times slower, than AlphaZero-3HNN. Probably the definition of data point in figure 2 is missing (e.g. one data point corresponds to one MCTS iteration).
6.	Figures 3 and 4 basically shows the same experiment, but with different models: AlphaZero-2HNN versus AlphaZero-3HNN with threshold 0 on Figure 3 and AlphaZero-2HNN versus AlphaZero-3HNN with threshold 1 on Figure 4. They could be united in one figure.
7.	Results from Figures 3 and 4 suggest, that 3HNN with threshold = 0 is better, than with threshold = 1. However, the paper claims setting threshold > 0 as an advantage. If it allows to save time (as each mcts iteration is faster), maybe they should be compared plotting time on x-axis (like in Figure 2)?
8.	Please provide error bounds in table 2.

Additional arguments
Argumentation presented in this section didn’t affect the score. However, it might improve the paper.
1.	3HNN predicts q(s,a), which should be equal to v(s’), where s’ = s ∪ a (state after action a is taken in state s). It would be interesting to see how condition q(s,a) == v(s ∪ a) holds during  3HNN models learning. It can be checked on a first dataset (drawn from games), or a special random dataset, containing consecutive positions, could be generated. Probably, this condition could potentially be an additional loss term.
2.	Also, it would be interesting to see how the condition between p and q holds. The higher q(s, a), the lower should be p(a). There is an interesting illustrating figure 7 in supmat, it may be presented in the main text. Also, it could be interesting not only for the first move. 
3.	The supmat claims that the motivation for turning off data augmentation term is that it assumes that in the selfplay game both players are selecting the best actions produced by search. Is it connected with the fact, that in the game of Hex all states are either winning or losing, according to theorem proved by Nash? How does this term would work for games with a draw trend, for example chess? In chess, for a lot of states the “ground truth” v(s) would be close to zero and there is no action guaranteeing the win.
4.	The paper claims “Elo (Elo, 1978) improvement does not always correspond to real playing improvement—the monotonic increase of Elo score did not capture the fluctuation in strength”. Citation needed, what fluctuation in strength is not captured? Is it specific to game of Hex? For example, Elo is used as the main measure of total agent strength in AplhaZero papers, as well as by chess community (both chess programs and human players).

Minor comments
1.	Page 4, section 2.2: Even though ... . Our summarization -> Even though ... , our summarization.
2.	Page 4, table 1: mvoe -> move.
3.	Page 7, bullet point above section 4.4: perfect -> perfectly.
4.	Page 7, the lowest paragraph. “For the value learning, however, due to fast search, the AlphaZero-3HNN learned much faster than AlphaZero-2HNN both on games states produced by strong player (T1) as well as examples produced by random play (T2).” 
It is confusing, it seems that 3HNN on datasets T1 and T2, however, it was only tested on these datasets.
5.	Page 8: imposing an an auxiliary task -> imposing an auxiliary task.
6.	Page 9: “produce playing strength significantly stronger than” – reformulate.

[1] Chao Gao, Martin Müller, and Ryan Hayward. Three-head neural network architecture for monte carlo tree search. In IJCAI, pp. 3762–3768, 2018b.

=====Post Rebuttal=====
Score updated from 3 to 6.

review_title: Official Blind Review #3

Meta-review: The authors provide an empirical study of the recent 3-head architecture applied to AlphaZero style learning. They thoroughly evaluate this approach using the game Hex as a test domain.

Initially, reviewers were concerned about how well the hyper parameters for tuned for different methods. The authors did a commendable job addressing the reviewers concerns in their revision. However, the reviewers agreed that with the additional results showing the gap between the 2 headed architecture and the three-headed architecture narrowed, the focus of the paper has changed substantially from the initial version. They suggest that a substantial rewrite of the paper would make the most sense before publication.

As a result, at this time, I'm going to recommend rejection, but I encourage the authors to incorporate the reviewers feedback. I believe this paper has the potential to be a strong submission in the future.


Decision: Reject