Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies models trained with AlphaZero for the game of Hex. The paper applies techniques from the NLP literature to investigate whether the learned model learns concepts usually taught to human players. The paper further investigates if the trained agent is able to act according to such concepts (e.g., connect a bridge when that is needed to win the game or to reach an advantageous state of the game). 
strengths_and_weaknesses: Strengths

The paper is well written and the experiments reported are interesting. Although the metrics and procedures come from the NLP literature, this paper is the first to attempt them in models learned with AlphaZero. I find it particularly interesting the Probe procedure, where a linear classifier is trained to detect known concepts of Hex. Interestingly, the concepts studied are detected with very high accuracy and the accuracy improves as AlphaZero further trains the model. This shows an interesting connection between the learned model and concepts that we (humans) deem as important for playing the game. 

There is a chance this paper will inspire others to work on the extraction of such concepts from the learned model. This future research endeavor could be important for those interested in teaching humans the knowledge that is generated with learning systems. 

The paper also makes interesting connections with previous work. I particularly like the connection between the results presented in the paper and those of McGrath et al. [37] on chess regarding where in the neural model that the short and long-term decisions are stored. Such connections facilitate the work of others starting out in this field. 

It was also interesting to see the methodology the paper uses to verify that the learned agent learns the connections in the board of Hex. 

Weaknesses

Some of the results reported in the paper aren't as interesting as the idea of running the experiments. For example, short-term decisions are encoded near the output layers of the neural model. Perhaps this isn't surprising because such decisions are actionable and one would expect the last year of the model (which is also linear) to correctly identify such situations. 

Another result that perhaps isn't particularly interesting is the fact that the studied concepts are first noted in the MCTS search and only later incorporated in the model. This is because MCTS generates the training data for the model, so the natural order is to first generate the data and then train the model. It would be strange if the model learned before MCTS collected the data. 

It is interesting that the concepts 'escape' and 'bottleneck' are first detected in probing to only then being detected in decisions of the agent (Figure 6 of the paper). I list this as a 'weakness' because the paper offers no explanation of why this is the case. This result almost contradicts the idea of MCTS generating training data and the model learning from the training data. This is because the concept can be first noted in the neural model to only then appearing as decisions in the game (behavior). I apologize if I missed the explanation of this phenomenon and would appreciate a response from the authors about it. 

Other comments

Regarding the following: "This highlights a weakness in AlphaZero and a risk: Some concepts may be provable and useful to people, but deemed less important by AZ." The agent is probably playing dead cells because it has either won or lost the game and it doesn't matter if it plays on those cells. If this is the cases, then this isn't a weaknesses of AlphaZero, but rather a challenge for systems that will attempt to extract information from the learned model to transfer to human learners. 
questions: Why did you define short term as single moves to the end of the game and more than one move as long term? It would be interesting to see a continuum between short and long term. For example, one could argue that 2 moves to the end of the game is still short term. 

Why is B a captured cell in Figure 2 (j)? This should be explained in the main text.

Is Hˆ{(l)} a board or the activation values of the network? I would have appreciated a formal definition of weights and activation values as I am not sure I would be able to implement the Probe classifier with the current description from the paper. For example, do we use the output values of all neurons of all layers as input to the classifier? Do we use the logits or the values after activation? 
limitations: The experiments were performed on a single domain and it isn't clear whether they will generalize to other domains (e.g., security games or even other board games). 

The experiments and results aren't exactly actionable. They are of scientific interest and might inspire others, but it isn't clear what role this paper will have moving forward.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper investigates AlphaZero's internal representations in the game of Hex by using two evaluation techniques: model probing and behavioral tests. They use probing classifiers to determine if agents can encode conceptual information and design behavioral tests to measure whether the agents can use the concepts to win games. 
strengths_and_weaknesses: The paper combines both representational and behavioral approaches to analyze reinforcement learning agents in the game of Hex. Overall, the result is quite interesting and allows the readers an understanding of how AlphaZero agents learn. However, the method used in the paper has already existed and has been widely used in many other board games, such as Chess[1], and Go[2]. The paper simply applies these methods to a different game (Hex). I think the novelty is not enough and the results in this paper will be limited to the whole community.

[1] T. McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, D. Hassabis, B. Kim, U. Paquet, and V. Kramnik. Acquisition of chess knowledge in alphazero. arXiv preprint arXiv:2111.09259,2021.
[2] N. Tomlin et al. Understanding game-playing agents with natural language annotations. To Appear: ACL, 2022.
questions: * The design of behavior tests in this paper is too simple. In the experiment, the test problems are designed to be 1 or 2 moves to win and lose if playing on the wrong move. It is obvious that AlphaZero can play on the right move since it has already achieved superhuman levels. That is to say, regardless of whether the AlphaZero agents have learned the so-called "concept", they will always play on the right move by choosing the move with the highest win rate. Overall, the behavioral test experiment is not convincing and does not clarify whether AlphaZero utilizes the learned concept to do things.
 
* In Figure 4 (a). In [2], selectivity was used to check the complexity of the probing model, to prevent the probing model from being too complex and overfitting. However, in this paper, there is no probing model complexity comparison, only the linear classifier was used. The author should explain what is the meaning of selective in the paper.
 
* In line 213, the paper mentions that they have similar results with McGrath et al. [1].  (short-term was best represented higher in the network and long-term was largely represented lower in the network.)  According to [1], there is no significant evidence that short-term concepts (McGrath et al. [1]’s Fig. 2)[c,e,f] are better represented in higher layers.
 
Other minor issues:
* line 192, 193, what are the definitions for Logit,  z-score? 
* In Figure 6, what do “behavioral:start” and “probing:start” mean? The definitions of each color point are unclear.
* In Supplementary line 34,  what A,B,C refer to?
* In Supplementary Figure7, NCDG -> NDCG

[1] T. McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, D. Hassabis, B. Kim, U. Paquet, and V. Kramnik. Acquisition of chess knowledge in alphazero. arXiv preprint arXiv:2111.09259,2021.
[2] J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743
limitations: The authors have addressed most of the possible limitations and potential negative societal impact in Section 3.4.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors use linear probing to identify pre-defined Hex concepts in the activations of the agent. They use behavioural tests to evaluate whether the concepts that were identified with linear probing are actually enacted by the agent. They identify that AlphaZero does indeed represent concepts, but inconsistently enacts some of them (negative ones - the concepts related to which moves the agent should not play). They find that short term concepts (those related to winning the game) are represented in later layers in the agent and long term concepts (those related to gamestates that are distant from a potential winning state) are represented in middle layers. They also find that the learned embeddings reflect the structure of the game board. 
strengths_and_weaknesses: I'm inclined to believe the authors when they describe their work as concurrent to the (very similar) work of McGrath et al. (2021), which means the author's work is at least somewhat novel. However, the authors sell this work as borrowing methods from NLP, when in fact those methods are linear probes and behavioural studies, methods that are sufficiently general that it can't be accurate to describe them as NLP methods (even the reference the authors use for linear probing is from image classifiers; Alain and Bengio, 2017). Consequently, the discussion in the introduction regarding NLP ought to be changed to reduce the implied links between NLP and the present work. The experiments are also not especially extensive and do little to address the weaknesses of prior work, particularly those of McGrath et al., which is that they rely on a pre-defined set of concepts. Nor is it possible (in Hex) to use this method to improve human play, since perfect play for Hex is known, at least for boards of certain sizes. It would have been interesting and novel to expand on prior work to see whether AlphaZero uses novel concepts in settings where perfect play is unknown.  

Overall the paper is reasonably well presented. However, figure 6 is unclear. Why are there lines connecting concepts in the same group? The authors should use a consistent name for 'structural'/'neighbours' in both the figure, its caption, and the text.

The choice of the game of Hex warrants some explanation, especially given that it dramatically reduces the audience of people who are interested in the game compared with Go or Chess. 

Structural comment: Put long term vs short term (sec 2.1) into the concept taxonomy (sec 2.2)

questions: Why did the authors choose to study the game Hex? How many people play this game? Was the choice because there are known concepts for perfect play? Wouldn't chess or Go have been a more interesting game to study, given its much wider player base and the possibility to identify new concepts learned by superhuman players? The reasons behind this decision should be explained in the paper.
limitations: The authors do little to address the limitations of the method of linear probing and behavioural studies - namely, that it relies on a bank of pre-identified human concepts, consequently making it difficult to understand how the network plays so well beyond concepts humans already know. Nor does it really give us a deep picture about how the network actually achieves these feats of play. The authors adequately address the potential societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors use *probing* and *behavioral* tests to investigate to what extent, and when, AlphaZero-style DNNs learn to use various concepts that humans are also known to use in the board game Hex. 

The analysis shows that the DNN tends to consistently pass many of the tests, for almost all of the evaluated concepts, after about 75% training progress. Negative concepts (which recognise cells that are useless to play in, such as dead cells) are an exception; these appear to still not be fully "understood" even at the end of training, and this can be observed in the agent's behaviour as well where it sometimes does not win as fast as it could, wasting some moves instead. For most concepts, the agent's performance on behavioral tests tends to improve before the performance on probing tests, which suggests that the full MCTS agent may learn how to use them before this knowledge gets encoded into the DNN. In general, it appears that long-term concepts tend to be encoded in middle layers, whereas short-term concepts tend to be encoded closer to the final layers.

The overall methodology used for the analysis described above may be applicable to other games / other RL environments.
strengths_and_weaknesses: Strengths:
1. Interesting and important topic, with potential applications to improve our understanding and ability to interpret what deep RL models have learned (and when they learn it, and what they fail to learn).
2. Well-written, largely easy to follow.
3. Discussions of related work seems good, don't see anything missing there, though my background and awareness of existing work is more so in RL and games than in probing/interpretability angle.

Weaknesses:
1. I believe that one small (but important) part of the paper could use some clarifications in the writing: Section 3.2 (on Representational Probing). I will elaborate below.
2. I think that a couple of claims in the paper may be slightly too strong and need a bit more nuance. I will elaborate below.
3. A lot of the details described in Section 3.3 (Behavioral Tests) seem quite specific to the game of Hex. For the specific case of Hex, we can indeed know how to create such states that (i) contain a concept, (ii) contain that concept in only exactly one place, (iii) make sure that the agent **must** play according to the concept immediately, because otherwise it would lose. I imagine that setting up such specific situations may be much more difficult in many other games or RL environments, and would certainly require highly game-specific knowledge again for such tasks. This seems like a potential limitation (which doesn't seem to be discussed yet).

---

**On weakness (1)**: 

The first sentence that I personally found confusing was "To form the random control, for each board $(H^{(0)}, y)$ in the probing dataset, we consistently map each cell in that board to a random cell, forming $H_s^{(0)}." I guess that "map each cell in that board to a random cell" means creating a random mapping from all cells in the original board to all cells in the control board, in a way such that every original cell maps to exactly one randomly-selected control cell, and every control cell is also mapped to by exactly one original cell. And then, the *value* of each original cell (black/white/empty) is assigned to the control cell that it maps to. I guess this is what is done, and it makes sense, but it's not 100% explicit. I'm afraid that many readers could misunderstand it as simply saying that every cell gets a random value directly.

Then, a bit further down under Implementation Details, it is described how the boards in the probing dataset get constructed. I suspect it would make more sense to actually describe this *before* describing how the matching controls are created.

---

**On weakness (2)**:

(a) The behavioral tests involve states created specifically such that they (i) contain the concept, but also (ii) **demand** that the agent **immediately** plays according to the concept, because it will lose otherwise. In the game of Hex, this means that all of these board states, for all these different concepts, actually include one more new "concept" that is shared across all the tests; a concept that recognises a long chain of enemy pieces that is about to become a winning connection if not interrupted by playing in what is usually just one or two remaining blank cells in between. So, I do not believe that we can say with 100% certainty that all these behavior tests are actually testing for the concept that you intend them to test for. Some or all of them may simply be testing more generally if the agent can recognise when it needs to interrupt the opponent's soon-to-be-winning chain.

(b) "Fig. 5 shows evidence that some information is learned before the model is able to use the concepts." --> I think "evidence" may be too strong here, and would say something more like "Fig. 5 **suggests** that some information **may** be learned [...]". Technically, Fig. 5 just shows that there is generally a long period with no progress on the tests, and after a long time suddenly rapid progress on the tests. To me this indeed suggests that it is *likely* that it is learning something else first, but it is not hard evidence. It could also be that it's just randomly wandering about the parameter space and suddenly gets lucky and makes quick progress then, having learned nothing at all before.

(c) "Behavioral tests can also expose heuristics the model may be using." --> yes, but only if we actually already know that the heuristics exist, and know how to explicitly encode them and create probes for them. They can't teach us any new heuristics that we didn't already know about. So maybe, better phrasing could be something like "Behavioral tests can also confirm whether or not the model may be using certain heuristics."
questions: (1) I'd like to start with a general question if you have any comments or clarifications on the weaknesses I listed above?

(2) For the probing dataset, it is described that random enemy pieces are added to the board. How many? There is no mention of randomly adding friendly pieces. Is it always guaranteed that ultimately the resulting game states constitute positions that could have been obtained through legal gameplay?
limitations: - It may be useful to discuss the apparent limitation that quite a bit of Hex-specific knowledge is used for setting up the probes (discussed in more detail as a weakness above).
- It may be useful to discuss the potential limitation I discussed in more detail above that the behavioral tests may simply all be testing for an agent's ability to recognise when it needs to interrupt an opponent's immediate winning threat.
review_title: null

Meta-review: metareview: This paper uses methods from interpretability to study the knowledge learned by AlphaZero in the game of Hex. In particular, the network's outputs are correlated with various hand-designed features.

The reviewers are all in agreement that there are some interesting contributions. There was some comparison with McGrath et al., but given that this paper is relatively recent and still unpublished, this does not seem like an insurmontable blocker. There was also some debate on whether evaluating on Hex is sufficiently relevant to the research community, or whether the kind of probes used are particularly insightful. I agree that the latter is a weak point of the paper, while the former is a reasonable concern but more minor. Another concern is what we do from this paper onwards -- i.e., how will this research feed into future work? In follow-up discussion, a majority of reviewers argued that there were useful nuggets of knowledge produced by this paper.
confidence: Less certain
Decision: Accept