Evaluation beyond Task Performance: Analyzing Concepts in AlphaZero in Hex
Authors: Charles Lovering
Jessica Zosa Forde
George Konidaris
Ellie Pavlick
Michael Littman
Venue: NeurIPS 2022 Accept (2022)
Keywords: alphazero
deep reinforcement learning
explainability
interpretability
evaluation
concepts
hex
mcts

Abstract: AlphaZero, an approach to reinforcement learning that couples neural networks and Monte Carlo tree search (MCTS), has produced state-of-the-art strategies for traditional board games like chess, Go, shogi, and Hex. While researchers and game commentators have suggested that AlphaZero uses concepts that humans consider important, it is unclear how these concepts are captured in the network. We investigate AlphaZero's internal representations in the game of Hex using two evaluation techniques from natural language processing (NLP): model probing and behavioral tests. In doing so, we introduce several new evaluation tools to the RL community, and illustrate how evaluations other than task performance can be used to provide a more complete picture of a model's strengths and weaknesses. Our analyses in the game of Hex reveal interesting patterns and generate some testable hypotheses about how such models learn in general. For example, we find that the MCTS discovers concepts before the neural network learns to encode them. We also find that concepts related to short-term end-game planning are best encoded in the final layers of the model, whereas concepts related to long-term planning are encoded in the middle layers of the model.

Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex
Charles Lovering"
Jessica Zosa Forde"
George Konidaris Ellie Pavlick Michael L. Littman
Department of Computer Science
Brown University (first)_[ast)) @rrwwn.duu
Abstract
AlphaZero, approach to reinforcement learning that couples neural networks and Monte Carlo tree search (MCTS), has produced state-of-the-ar strategies for traditional board games like chess, Go, shogi, and Hex. While researchers and game commentators have suggested that AlphaZero uses concepts that humans consider important, it is unclear how these concepts are captured in the network. We investigate AlphaZero's internal representations in the game of Hex using two evaluation techniques from natural language processing (NLP): model probing and behavioral tests. In doing so, we introduce new evaluation tools to the RL community, and illustrate how evaluations other than task performance can be used to provide a more complete picture of model's strengths and weaknesses. Our analyses in the game of Hex reveal interesting patterns and generate some testable hypotheses about how such models learn in general. For example, we find that MCTS discovers concepts before the neural network learns to encode them. We also find that concepts related to short-term end-game planning are best encoded in the final layers the model, whereas concepts related to long-term planning are encoded in the middle layers of the model.
Introduction
reinforcement--eammie agent that combines Monte Carlo tree search (7] with deep reinforcement learning. has achieved impressive performance games like Go, chess, shogi, and Hex. Domain experts have observed that AlphaZero uses, but does not master, identifiable game concepts. For example, despite being exceptionally strong overall, AlphaZero appeared unable fully to project the implications of a ladder-an an important concept in the game of Go [60].
AlphaZero 55
Good performance mask flaws in deep learning systems generally 2
49
69
10]
and deep reinforcement-learbne systems particular [66] 77). Evaluating systems in terms of task performance alone makes it impossible know if systems are "right for the right reasons" and difficult to predict how they will generalize to new situations (67 68). Recently, the field of natural language processing (NLP) has begun develop evaluation techniques that go beyond "just" task performance For example, some techniques, probing classifiers, inspect the form of models' internal
3 4.
representations to whether they are consistent with linguistic theory [ 58]; other techniques. generalization to
behavioral tests or challenge sets, evaluate specific types of oft-off-istribute
assess whether models encode human-like inductive biases [3 19
32
36
63]. Although
these techniques are only used over known concepts, often requiring expert domain knowledge to define, the insights generated by these techniques are not only scientifically interesting, but have begun to yield actionable insights on how to employ and improve models. For example, Geva et al.
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) An example winning board for black, which connects the black edges together.
(b) Short- vs Long-term concepts. Left: If black plays A or black immediately wins; the board concept containing and B, bridge (See ( relevant in the short-term. Right: A and can help black win only in the long-term.
Figure 1: To win in Hex, a player must use their pieces to form connecting chain between the edges matching their color (a). Hex boards can be of varying size; we evaluate AlphaZero 9x9 board. In our work, we make a distinction between short- and long-term concepts (b).
[20] found that multilayer perceptrons (MLPs) in transformer layers act key- memories, and Meng et al. 43 leveraged this understanding to manipulate model behavior in a controlled manner.
We demonstrate how these ideas leveraged in NLP can be applied to reinforcement learning, testing the capabilities and limitations of our models. Specifically, we leverage the abeve-dedrribed anal- ysis techniques- pprobing classifiers and behavioral tests to interpret AlphaZero's ' behavior at conceptual level. We use probing classifiers to determine if reeinforeeenntllearnnng agents encode tactical and strategic conceptual information. However, probing performance alone is insufficient: Information may be encoded but not used [37]. To address this issue, we also use behavioral tests, which evaluate an agent's 's decisions in situation tailored to require the understanding of a specific concept.
Given these concept-level evaluation methods, we do an in-depth study of AlphaZero (AZ) trained to play Hex. Hex a board game similar to Go ($2). but provides an opportune test bed for analysis: the game is complex, yet has perfect-play baselines for smaller board sizes. Moreover, concepts and strategies within Hex have been studied by the Hex-playing community, making Hex rich vector for study. Furthermore, the clear behavioral expectations in Hex make it easier interpret and guide future work? In our work, we probe for concepts that are taught to new players Hex, like the bridge (Fig- 2h, and test that the model is able to use them to win games. We investigate how concepts are represented within AZ, when concepts are learned during training, and where concepts represented within AZ's neural network.
Overall, our findings suggest that there some regular dynamics how AZ learns concepts with the game of Hex, and generate interesting predictions about how AZ behaves in general, which could be tested in other games. (In fact, concurrent work on chess 42 and Go [6] already begin to provide convergent evidence that some trends we see here generalize elsewhere.) We introduce a novel way of analyzing deep RL models on which such subsequent work can easily build.
In summary, our main contributions are:
We adapt several evaluation techniques from NLP to the RL setting, and illustrate how evaluations other than task performance can be used to provide a more complete picture of model's strengths and weaknesses.
2. We analyze a poppeerfrrming model from Jones 30 and find that (1) short-term end-game concepts are best represented in the final layers of the network, whereas long-term concepts are best represented in the middle layers of the network; (2) concepts appear to originate with with
MCTS withh MCTS overriding the deep learning policy prediction early in training butin later in training these concepts are incorporated directly into the model's network.
"Again drawing inspiration from work in NLP. Linzen et al. 335 focused on one syntactic phenomenon in simple current neural networks, and the basic insight and methods enabled later studies to make broader claims about learning linguistic structure in many types of models 226 63 64

---

Positive Concepts
Negative Concepts
Internal Concepts
Edge Concepts
Ladder Concepts
(e) Edge 2
(i) Dead cells
(g) Bottleneck
(w) Bridge (b) Crescent
(c) Trupezoid
(d) Spun
(i) Edge
(b) Escape
( Captured cells
Hex templates exemplifying game concepts. Concepts within the game of Hex are templates on the board formed by a player's pieces with known strategic and tactical implications. Positive concepts provide the player with the concept with multiple ways to connect the pieces within the concept together, despite possible attacks from the opponent. An example of such a concept is the bridge (a). If white plays move I, black can connect the two pieces of the bridge by playing move 2. Negative concepts change the strategic value of specific open spots of the board, such that the opponent is disincentivized to play those open spots, such move A in (i). Each concept is further described in our Supplementary Material. Arrows indicate that the piece is connected to the opposite side of the board; the lines show the bridge concept within the other concepts.
Figure 2:
2 Concepts in Hex
We study AlphaZero (AZ) agents trained to play Hex [17], game where two players take turns filling cells until one player builds a chain across the board. Hex has well-studied rules, reasonable computational costs, and it can be evaluated against perfect play, making it an ideal experimental vehicle for model probing. Unlike Go, there are no captures; once a cell is filled with a piece, the pieces stays there for the remainder of the game. For example, in Fig. Ta] white must connect pieces from left to right (marked on the edges), and black top to bottom. Hex cannot end in a tie [15]. and given perfect play, black, the first player, will win [17
In Hex, certain templates - patterns of cells have been recognized as useful. Because building up a chain from one side of the board to the other is casily thwarted, a key part of learning how to play Hex is recognizing when it is possible to connect groups of pieces together. We consider these templates to be concepts within the game. Concepts in Hex have different move implications depending on the condition of the board. We discuss two conditions, long- and short-term, in detail and provide additional discussion of conditions in the Supplemenaary Material.
While the properties of concepts are debated [39], here, in a board game setting, we consider a concept to be a useful template that generalizes across board configurations. For our purposes, "understanding" a concept amounts to recognizing and leveraging its implications during gameplay ($2.2).
2.1 Long-term vs short-term Concepts
We define a concept in Hex to be short-term if its use is sufficient to win the game. Concepts are typically short-term when there exists connection between the concept and the player's board edges. For example, in Fig. [b] moves A and belong to the bridge concept (see Fig. 2p) In the left board, the board edges are connected, and all that is required for black to win is to play moves A B. Conversely, long-term concepts are insufficient to winning the game immediately used. In the right board of Fig. Tb playing the bridge concept is useful but insufficient for winning immediately; the concept is not connected to the edges of the board. Empirically, we find that whether the concept is short- or long-term has significant impact AlphaZero's 's representations (Fig. 4
'Hex is often played with a "swap rule" that makes the game more even between black and white. See Jones (30] whose implementation we use, for further discussion on the swap rule. did not include it to simplify the game implementation.
3

---

2.2 Concept Taxonomy
From Seymour [54 and King 33,,
we identify the nine concepts that we use in our analysis, summarized in Fig. These concepts fall into four categories, as described below.
Positive Concepts With the goal of Hex being build a chain across the board, it is helpful to recognize when cells are virtually connected, that is, even in response to perfect adversarial play. the few different types of positive concepts. All the positive concepts favor the player that owns the concept on the board.
cells are guaranteed to connect 23 47.. There
Internal concepts are templates that appear within the interior of the board. The bridge (Fig. 2a)) is the simplest such concept. The larger internal templates - crescent, trapezoid, span (Fig. 2 bl.,,d)) provide several possibilities to connect player's 's pieces. Edge concepts concern connecting single cell to a given edge. Ladders in Hex are similar to ladders in Go. There are two different ladder concepts, bottleneck (g) and escapes (h). A bottleneck favors the defender, because the attacking player cannot break through. An escape, however, allows the attacker to break through.
Negative Concepts Negative concepts do not inform one on which actions to play, but rather, which actions not to play. Dead cells (i)g.2i)) cannot impact the outcome of the game regardless of the color with which they are filled. While it is in general difficult to compute if a cell is dead [6], there are some known templates where it is easier to deduce. If a player can make cell dead, that cell captured. Both captured and dead cells should not be filled.
3 Designing Probing Tasks and Behavioral Tests for Hex
To understand the concepts encoded within AlphaZero, probe its internal representations; to evaluate if these concepts are used by AZ, we test its behavior on tailored board configurations. By evaluating AZ across training checkpoints, and across neural network layers, we can build up an understanding of how and where the model recognizes these concepts. Specifically, we evaluate the top-performing agent trained by Jones (30 across 21 training checkpoints. We additionally evaluate other publicly available agents (30 from that differ by width and depth and we report on those results in the Supplementary Material. Code from Jones is available under the MIT License.
3.1 Reproducibility
Furthermore, we release example images of boards created for our probing classifiers and videos of the behavioral tests. The code, results and examples can found at fattps:/bitt.1y//allatttot
Our code and results are publicly available 38
Our repository is also available on GitHub at We report hyperparameters in the Supplementary Material. All error bars presented in plots are one standard deviation above and below the mean. We used NVIDIA GeForce RTX 3090. The total compute across all experiments was about 24 GPU hours. We present a breakdown of the compute in the Supplementary Materials.
https://gt thub.
3.2 Representational Probing
Model probing measures how well a model's learned representations encode a known concept M. To make model probe, one labels examples, such Hex boards, with the presence or absence of concept, such as the bridge concept (see Figure Fig. 2 Next, one collects the model's 's activations for each example, and then trains a linear classifier (the probe) to predict the presence of the concept based on activations. The linear classifier's 's test performance is used to interpret how well the original model encoded the concept.
We train a linear probe for each concept over each layer of AZ's network body. We follow a procedure similar to Tenney et al. [58]: (0) is the state of the board that is used as input into AZ. For each board, record the label y& to indicate the presence absence of the concept i on the board. H E I..L. is the activation of the layer of AZ's network body We then train linear classifiers I p(i,1e
O. per layer to predict the presence vs. absence of a concept in a given board. These classifiers are our concept probes.
WWe never use activations from multiple layers once, layer weights AZ's value/policy outputs.

---

It is important to compare probing results against baseline. We follow Hewitt and Liang [24] 's procedure to measure concept selectivity, the delta between probing accuracy over a concept and random control. To form the random control, for each board H (0) in the probing dataset, we map each cell in that board to a random cell in a consistent manner to form the transposed board, H!!) In this way, the same information is encoded in the original boards, but we expect the shuffled boards to be meaningless in Hex. Next, we train a set of linear probes p!!) I O..L over the control boards H!! to predict y. Now, finally, we can compute the concept selectivity by finding the delta in accuracy between ( and p()), Concept selectivity is the performance of probe beyond the performance of probe control task, adding context for interpreting the results.
Implementation Details. Each concept is defined by a set of boards with vs. without that concept. We train and evaluate probing classifiers over AlphaZero's encoding of these boards. To generate a set of (N = 2500) boards for each concept, we translate the minimal templates across an empty board. Then, we add random enemy pieces to the board. Negative instances of a given concept match the statistics of the positive examples, except that the pieces pertaining to the concept template are randomly moved across the board. This constitutes the long-term version of a concept. To form the short-term version of a concept, we connect the template to the edges of the board.
3.3 Behavioral Tests
Where model probing asks if concepts are represented within the model, behavioral tests asks the model knows how to use the concept in gameplay. To interpret the behavioral tests, they must have clear behavioral expectations. We construct our behavioral tests for positive concepts ($2) to be forced: If AZ understands the concept and plays the expected moves, AZ will win and pass the test; otherwise, AZ will lose the game and fail the test. For negative concepts ($2). we have clear behavioral expectations. Dead and captured cells should never be filled. Thus, the behavioral test for negative cells checks that during a selfplay continuation of a board containing a dead (or captured) cell, the agent does not fill that cell.
Success on these behavioral tests are necessary but are alone insufficient to establish that the model has the concept. A negative result means that AZ is unable to use the concept, whereas positive result means that AZ can use the concept to win games in forced situations. This approach has been used to test large language models. Specifically, we are inspired by Eutingers [13 46) who uses simplified language to ask if the model is able to use a concept like negation or syntax when needed. If the model still fails to use the concept, then this good evidence that the concept is not represented.
Implementation Details. For each concept, we create behavioral tests that comprise a board, forcing moves, and expected moves. (For the dead and captured cells there are no forcing or expected moves, only the moves to avoid). We discuss the motivation for our behavioral test setup and its connection AlphaZero's 's style of gameplay in the Supplementary Material. By way of example, see Fig. To generate a set (N = 100 samples) of boards, we translate the templates to a sampled valid board position. We then connect the concept to the edges of the board. Finally, we add connections for the defending player up to the region of the concept such that if the attacking player fails to complete the behavioral test, the other play would win. Finally, we add the appropriate number of random pieces such that the board position is valid. The behavioral tests determine when/if AZ learns to navigate these situations correctly.
3.4 Considerations
Limitations. While find a consistent relationship between the probing and behavioral tests in Section (4.3 we do not run a counterfactual study. For example, we do not show that mistakes in recognizing concept on given board, lead to mistakes in using that concept. Studying the causal mechanisms of how the concept representations detected by model probing impact downstream model behavior is a rich direction for future work.
In our behavioral tests, all the concepts we tested share the property that they are about to be connected. (They share this property because we connect the concept so that the behavioral expectations are clear, Fig. 3 All the concepts being tested could be interpreted as "interrupt the opponent's 's soon-to- winning chain." However, the behavioral tests of different concepts report different performance levels
5

---

(4)
(b)
(d)
Figure 3: Creating behavioral tests from concept templates. To evaluate AlphaZero's ability utilize concepts during game play, we build synthetic boards where utilizing the strategic advantages of the given concepts allows the player win the game. In this example, demonstrate the building of a behavioral test for the bridge concept (Figure 2b). The minimal template (a) is translated to random position on the board (b). Then both players' pieces are connected their respective edges they need to utilize to win the game (c). Finally, the minimum number of noise pieces necessary to form valid board are added (d). Cells A, are used to define the behavioral test. If white plays A. black must play B to win the game. (Only half a 5x5 board is shown for space.)
1.0
50
Long-term Short-term
Percent of Probes
Accuracy
0.6
0.4
25
Long term Concept
Short term Concept
Selectivity
Selectivity
Selectivity Baseline
Selectivity Baseline
bridge
crescent
trapezoid
span
dead
captured
bridge
crescent
trapezoid
span
edge
escape
battleneck
Layer with Highest Accuracy
(a) AlphaZero successfully encodes long-term and short-term concepts. The selectivity (24,, indicated by the colored bars, are the accuracy of probe trained to identify a concept based on network activations, minus the accuracy of selectivity baseline. The selectivity baseline randomly maps board pieces that the boards not contain structures known be relevant Hex. report selectivity based on the layer with the highest test accuracy.
Long-term concepts best rep- resented the middle layers of the network whereas short-term concepts are best represented in the final lay- of the network. Each distribution shows the layer in which probes had highest accuracies.
Figure 4: Probing performance long- and short-term concepts.
(though admittedly similar), suggesting that the differences of concept are still relevant. Furthermore, the agent would still have had the targeted concepts as constituent of "interrupt the opponent's soon-tee soon-to-be to-be-winning chain."
Societal Impact.
Deep reinforcement learning models such AZ are not interpretable [12]. and
yet are being applied to impactful, real world domains [2] 50 18 5 31
In this work, we start to analyze how AZ. comes to its decisions. Understanding model's decisions is critical for accountability, but does open the door to some avenues for exploitation. If is known that a given algorithm does not reason about concept well, this could be leveraged for ill. However, this risk only strengthens the argument for uncovering such problems and fixing them to prevent such risks.
Results
To understand which concepts AlphaZero (AZ) learns, we examine if its neural network activations encode the concepts (probing tests) and if AZ can use the concepts win games (behavioral tests).
4.1 AlphaZero Recognizes and Uses Concepts
AZ successfully encodes short-term concepts, with high selectivity scores (Fig. 4a). concept scores are also learned, but with slightly lower scores. By the end of training AZ is able to all the positive concepts win games (Fig. 5l).
The long-term

---

AZ improves on the behavioral tests 50% of the way through training. Unsurprisingly, the MCTS passing rates increase before the policy network passing rates (Fig. [5). though it need not have been the case. It is possible for AZ. to represent the concepts before MCTS used them possibly via the signal through the value prediction.
Interestingly, the relative magnitude of the correct action logits initially increases earlier. To specify how we measure this, we need to cover two definitions. First, the logits are the outputs of the modules (MCTS or policy prediction head). Second, we use a Z-score, which reports the number of standard deviations higher given value is than the mean of the population. In our case, the blue line in Fig. 5p. reports the proportion of cases where the Z-score of the correct action is greater than 1. So, this value captures when the correct action becomes more likely throughout training.
The trend in Fig. 5a suggests "pre-conceptual" information is learned, and coalesces (for bridge) 60% of the way through training into actionable understanding of the concept. (In Section 4.4 investigate further and find that this "pre-conceptual" information is not board structure.)
bridge
span
trapezoid
crescent
dead
Behaviaral Test
1.0
Behaviaral Test Behaviaral Test
0.75
0.5
0.50
A P Pase
0.25
edge
bottleneck
escape
5%
50%
100%
captured
Behaviaral Test
1.0
Training Progress
0.75
0.5
MCTS Z Score
0.50
S
Policy Passing
MCTS Passing
Passed
5%
50%
100%
5%
50%
100%
5%
50%
100%
5%
50%
100%
Training Progress
Training Progress
Training Progress
Training Progress
(a)
(b)
Figure 5: (a): AlphaZero learns to use positive concepts. At each checkpoint, present AlphaZero (AZ) with a set of example boards that test its ability to utilize each concept. MCTS and the policy network both select actions that pass our behavioral tests with increasing frequency throughout training. We additionally report the rate at which the logit score of the action that passes our behavioral test is one standard deviation above the mean logit score score 1).
(b): AZ does not fully use the negative concepts. Passed denotes the rate at which AZ avoids the negative concept throughout selfplay rollouts. So, the end of training, AZ plays moves in 25% of our behavioral tests that will not impact the game [6.
AZ also improves behaviorally upon negative concepts. However, it does not reach a perfect passing rate (Fig. 5p). The probing performance for the negative concepts, shown in Fig. 4a is also lower than for other concepts. This aligns with evidence that AZ. wastes moves at the end of the game. This highlights a weakness in AlphaZero and risk: Some concepts may be "provable" and useful to people, but "deemed" less important by AZ an agent that plays remarkably well. We will have to consider this dynamic as people begin to try to learn concepts from machines.
4.2 Long-term and Short-term Concepts are Best Represented in Different Layers
Fig. 4b highlight that short and long-term concepts are best represented at different layers. Long-term concepts. by the end of training, are best represented in the middle layers of the network. Short- term concepts, throughout training, have two conjectures, which are not mutually exclusive, for why this may be the case. We leave the verification/refuuatiion to future work. (1) Short-term concepts generally require more global information and so require more depth; (2) Short-term concepts factor directly into action selection and more proximal to the final task-specific layer of the network.
best represented in the upper layers of the network. We
We find that two and four layer networks demonstrate the same pattern; see our Supplementary Material. Lastly, McGrath et al. (2)'s results in chess are similar, although they did not directly study short- vs long-term concepts. The following is our interpretation of the results shown their plots: "in check" (short-term) was best represented higher in the network and "material imbalance"
7

---

(long-term) was largely represented lower in the network. (Informally categorized, the short-term concepts (McGrath al.]s Fig. 2)cc,e.] are better represented in higher layers than the long- geggtterm concepts (Fig. 2).a.g.h.i).)
4.3 Improvements in Behavioral Tests Occur Before Improvements in Probing Accuracy
Where network body processes board configurations, MCTS directly governs the decision-making procecess. In principle, either module could be the first to discover the game concepts. For instance, the network body could start to represent concepts via updates to the value function, later enabling MCTS to successfully navigate the concept templates. We find that the first improvements in behavioral tests precede the first improvements in probing accuracy (Fig. (6) where we consider first improvement the epoch with 5% increase concepts, especially the internal concepts. Then, as the policy network is trained to match the MCTS logits, the concept representation is absorbed into the network.
the baseline value. MCTS seems discover
find evidence that the structure of the board first learned at about the same time other concepts are learned. In Section 4.4 we discuss how probe for this structure, determining if AZ encodes relative distance between all cells on the board. The last column of Fig. 6 labeled structural, shows that improvements that AZ does not learn concepts according to an obvious order or curriculum, but rather, concepts of differing levels of complexity develop parallel.
this concept occur similar time as other concepts. This suggests
The results shown in Fig. [6 outline the relationship between the concepts as they are learned during training, but does not establish a causal relationship between how the concept representation (as detected by the probing models) impacts how the model uses that concept.
100%
Training Progress
behavioral: start converged probing: start converged structural: start converged
50%
5%
trapezoid bridge crescent
span
edge
escape
bottle.
capturedstructural
Figure 6: Improvements in behavioral tests occur before improvements in probing accuracy, Each point marks the mean checkpoint in training that AZ started to learn (or converge upon) the behavioral (or probing) test. Error bars are standard deviation across random seeds and short- vs long-term. We group the concepts as in Fig. 2 While behavioral tests start to improve before the probing, they both converge near the end of training. Exception: the ladder escape and bottleneck concepts (Fig. (2)) are easy for the probes to detect, consequently having low selectivity (Fig. 4a). perhaps because they occur along the edges of the board, and as result, have fewer possible configurations. The concept structural, ($4.4). which evaluates how well AZ's cell embeddings capture Hex's s neighborhood structure, learned and converges in a similar time frame as the probing task.

---

4.4 AlphaZero's Cell Embeddings Capture the Structure of the Board
Understanding Hex's concepts requires under- standing the board's 's structure. that is, which cells connect to which other cells. AlphaZero (AZ), with its feed forward network architec- ture, does not a priori represent this struc- ture. Fig. [S] suggests that some information learned before the model able to use the concepts. A possibility is that AZ spends the initial portions of training building up a repre- sentation of the board. However, we find no evidence that the neighborhood structure is learned in the initial stages of training, rather, appears to be learned only after the first improvements in behavioral tests (Fig. [
Figure 7: Implicit board structure. Arrows mark the learned nearest neighbors of each cell.
To study if Hex's board is represented in AZ. hypothesized that the structure of Hex's board is implicitly learned by AZ's s first layer. For each Hex cell, we extract a cell embedding from the first layer of AZ Next, we compute the dot-products between each cell embedding. The dot-product score between ground-truth neighbors increases throughout AZ's training.
The nearest-neighhors (according the dot-product scores) nearly match the ground-truth by the 15th of 20 checkpoints, and eventually match the ground-truth before deviating slightly. We evaluate how well the dot product scores align with the ground-truth cell distances via ranking metric, Normalized Discounted Cumulative Gain (NDCG). The NDCG first improves about 50% of the way through training highlighted in Fig. 6 Details the learned structure are in the Supplementary Material.
5 Related Work
Applying MCTS and Deep Learning to the Game of Hex. Jones 33), whose trained agents
use, studied scaling laws between various parameters, finding a mathematical relationship between compute. board size, and agent performance. This type of result can help others make informed tradeoffs when training woard-game-plant both use value functions in concert with MCTS to play Hex, MoHex (27] combines programmatic connection detection, pattern matching, and MCTS, attacking Hex for the 9x9 grid.
agents. Where Gao et al. [16] and Takada et al. (57]
AlphaZero, anecdotally, uses concepts, but only achieves partial mastery. Domain experts have observed that AlphaZero (55) and related agents that combine Monte Carlo tree search [ with deep reinforcement learning use identifiable concepts within board games. Michael Redmond identifies several common gameplay concepts demonstrated by AlphaZero [T]] in AlphaZero's 's Go matches against Lee Sedol. Silver et al. (56) noted that common human-player corner moves in Go Go-joseki josseii-eerrr were used by AlphaZero during self elff training before being abandoned, presumably other more effective moves were found. Additional analysis provided by Tian et al. (60) note that Elf OpenGo only partially mastered ladder sequences within the game. Chess commentator Antonio Radié (51) detailed how AlphaZero used zugzwang
(65] in the course of defeating Stockfish (52)
Experts have already incorporated some of AlphaZero's 's innovations into their play [45] 53
Probing neural networks for human concepts. A range of linguistic concepts have been detected in NLP models using probing models g
59 25]. For example, Tenney et al. [59], with probing models, found that BERT appears best encode linguistic concepts in the same order those concepts might be processed by the typical NLP pipeline. There has been some discussion on what good probing accuracy signifies. Hewitt and Liang (24] calls for baseline controls, and to measure only the gain in accuracy compared to these baselines. Voita and Titov 62 found that measures
48
40
AZ's first layer takes a flattened hhhot encoding of board and matrix multiplies with weight matrix. The embedding of each cell can be spliced out of that weight matrix. We only extract the cell embeddings for the current player. This constitutes structural probe [25] A structural probe tests the relationship between neural network activations (or weights).

---

beyond accuracy, namely Minimum Description Length (MDL) (MDL)-which which is used to measure how easy it detect given
more stable results.
Understanding reinforeement-terringg agents trained to play board games. Sadler and Regan [53] cover how some particulars from AZ's gameplay has impacted high-level chess. Concurrent to our work, McGrath et al. 42] also looked at how AZ acquires game knowledge (which we term concepts) in chess. Using similar probing techniques, and the original (and larger) AZ. they find that it learns to encode many of the prototypical chess concepts. Related, but not focused on model understanding per se, Jhamtani et al. (28] collected an annotated set of chess games. This type of resource is similar to what is used by McGrath et al. 442 for their probing task. In our work, we focus on controlled programmatic generation of boards with (and without) concepts present. While these board instances may not occur during selfplay, the human annotated games are also off-distribution with respect to AZ. Beyond finding similar results in a different game (Hex vs chess), we also leveraged behavioral tests, tying the concept representation to agent behavior.
6 Discussion
Our analyses suggest that AlphaZero (AZ. learned to both represent and use concepts that humans consider important when playing Hex. However, we found that AZ is sometimes either blind or agnostic a winning move. This had interesting ramifications. For instance, the negative concepts, especially dead cells, were not as well-encoded or used compared to other concepts. This may be because AZ has no "qualms" with playing wasted moves, like dead cells, if doing so doesn't 't change the outcome of the game.
layerwise analysis showed that the same concept represented most strongly in different layers. depending on its context: short-term concepts that inform actions at the end of the game are encoded in the upper layers of the model, whereas long-ter mergteerm concepts are absorbed deeper into the network. This absorption mirrors how AZ's policy head was trained to predict the policy outputs of MCTS.
Combining both representational and behavioral approaches to analyze reinforcement-leanee learning agents allows for a fuller understanding of how agents learn. Studying the representations of concepts allows us ask (and answer) rich of questions about where those concepts reside. Studying the behavior of the model with respect to a given concept tests that this representation can be translated into action. Behavioral tests can also determine whether the model may be using known heuristics. They are complementary approaches. In future work, applying causal approach to the study of agents' policies will further illustrate how well these agents understand these concepts.
Acknowledgements
We would like to thank the reviewers across different iterations of this work; they helped us clarify work and findings. This research conducted using computational resources and services at the Center for Computation and Visualization, Brown University. Charles Lovering was supported in part by the DARPA GAILA program. This work received support from the ONR PERISCOPE MURI award N00014-17-1-2999.
References
[1] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. ICLR Workshops, 2017.
[2] M. A. Badgeley, J. R. Zech, L. Oakden-Rayner, B. S. Glicksberg, M. Liu, W. Gale, M. V. McConnell, B. Percha, T. M. Snyder, and J. T. Dudley, Deep learning predicts hip fracture using confounding patient and healthcare variables. NPJ Digit Med, 2:31, Apr. 2019.
[3] Y. Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational
Linguistics, 48(1):207-22 2022.
[4] Y. Belinkov, S. Gehrmann, and E. Pavlick. Interpretability and analysis in neural nlp. In Proceedings of the 58th annual meeting of the association for computational linguistics: futorial abstracts, pages 1-5, 2020.
10

---

[5] M. G. Bellemare, S. Candido, P. S. Castro, J. Gong, M. C. Machado, S. Moitra, S. Ponda, and Z. Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(78366:77-82, Dec. 2020.
[6] Y. Björnsson, R. Hayward, M. Johanson, and J. van Rijswijck. Dead cell analysis in Hex and the Shannon game, Graph Theory in Paris, pages 45-59. Springer, 2006.
[7] B. Brügmann. Monte Carlo go. Technical report, Max Planck Institute of Physics, 1993.
[8] A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni. What you can cram into a single $&.#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-21 (66-21333 36, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/11/18--111 P18-1198
URL httth::ttp/ett wwwweb..wwwbbb............ org/ anthology/
[9] A. F. Cooper, Y. Lu, J. Forde, and others. Hyperparameter optimization is deceiving us, and how to stop Adv. Neural Inf. Process. Syst. 2021.
[10] A. F. Cooper, E. Moss, B. Laufer, and others. Accountability in an algorithmic society: relationality, responsibility, and robustness in machine learning. 2022 ACM Conference on, 2022.
[11] DeepMind. Match 1 I-Google DeepMind challenge match: Lee Sedol AlphaGoo Mar. 2016.
[12] Doshi- Velez and Kim. Towards rigorous science of interpretable machine learning. arXiv,, Feb. 2017.
[13] A. Ettinger. What bert is not: Lessons from new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48, 2020.
[14] J. Z. Forde, A. F. Cooper, K. Kwegyir-Aggrey, and others. Model selection's 's disparate impact Real-World deep learning applications. arXiv preprint arXiv, 2021.
[15] D. Gale. The game of hex and the brouwer Fixed-Point theorem. Am. Math. Mon., 86(10): 818-827, Dec. 1979.
[16] C. Gao, R. Hayward, and M. Müller. Move prediction using deep convolutional neural networks in hex. IEEE Transactions on Games, 10(4):336-343 0333, 2017.
[17] M. Gardner. The 2nd Scientific American Book of Mathematical Puzzles and Diversions. Simon & Schuster, 1961.
[18] J. Gauci, E. Conti, Y. Liang, K. Virochsiri, Y. He, Z. Kaden, V. Narayanan, X. Ye, Z. Chen, and Fujimoto. Horizon: Facebook's 's open source applied reinforcement learning platform. Nov. 2018.
[19] J. Gauthier, J. Hu, E. Wilcox, P. Qian, and R. P. Levy. Syntaxgym: An online platform for targeted evaluation of language models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page 70-76, 2020.
[20] M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.1499 14913, 2020.
[21] Gu, Holly, Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipula- tion with asynchronous off-policy updates. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3389-3396. ieeexplore.ieee.org. May 2017.
[22] S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. Smith. Annotation artifacts in natural language inference data. arXiv preprint 1XXv:1803300222. 2018.
[23] R. Hayward, Y. Björnsson, M. Johanson, M. Kan, N. Po, and Van Rijswijek. Solving 7x 7 hex with domination, fill-in, and virtual connections. Theoretical Computer Science, 349(2): 123-139, 2005.

---

[24] J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNN.-IICCNLP)) pages 2733-2743, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1 1275.
Uttpttps:: www. aclweb. org/ argg/nntologyy///// /D99-1/11111 1275
[25] J. Hewitt and C. D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4129-4133 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 1111665//1/N19-14199 URL https:/ /www. aclweb. org/ anthology/ N19-1419
[26] J. Hu, J. Gauthier, P. Qian, E. Wilcox, and R. Levy. A systematic assessment of syntactic generalization in neural language models. In Proc. of NAACI. (ttp:ttt:://wwtt aclweb arg/anthology/220 2000 acl-main. 158
[27] S.-C. Huang, B. Arneson, R. B. Hayward, M. Müller, and J. Pawlewicz. MoHex 2.0: A Pattern-Based MCTS hex player. In Computers and Games, pages 60-71. Springer International Publishing, 2014.
[28] H. Jhamtani, V. Gangal, E. Hovy, G. Neubig, and T. Berg- KinngKKikkattikk.. Learning to generate move-by-move commentary for chess games from large-scale social forum data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1661-1671, 2018.
[29] A. Jones. LICENSE at master andyljones/boardlaw, 2021.
[30] A. L.. Jones. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03/1 2021.
[31] Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool,
R. Bates, A. Židek, A. Potapenko, et Highly accurate protein structure prediction with alphafold. Nature, 596(7873) 2021.
[32] N. Kim and T. Linzen. Compositionality as directional consistency in sequential neural networks In Workshop on Context and Compositionality Biological and Artificial Neural Systems, 2019.
[33] D. King. Hall of hexagons. https://wwxt www.:r.kiiggoooo org. uk uk/hhexagoss/indxx html 2004. Accessed: 2021-11-12
[34] B. Lake and M. Baroni. Generalization without systematicity: On the compositional skills of scquence-t0-scqqeence recurrent networks. In Proc. of ICML, 2018. URL ltttps///axxi org/ arg/abs/1711.00355
[35] T. Linzen, F. Dupoux, and Y. Goldherg. Assessing the ability of LSSMS learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016. doi: 10.1162/tacl_ Latiaalla__01111__11_11111111111 00115. httpstts:: hwtp::/wwwwwwwwww.cop.bb org org/ wttg://anthlloggg11-111222
[36] T. Linzen, E. Dupoux, and Y. Goldberg. Assessing the Ability of LSTMs to Learn Syntax- Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4: 521-535, 2016. URL http://accwwb. org anthologyy////// 001-1037
[37] C. Lovering, R. Jha, T. Linzen, and E. Pavlick. Predicting inductive biases of pre-trained models, 2021.
[38] A. Lucic, M. Bleeker, S. Bhargav, J. Forde, K. Sinha, J. Dodge, S. Luccioni, and R. Stojnic. Towards reproducible machine learning research in natural language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 7-11, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[39] E. Margolis, S. Laurence, et al. Concepts: core readings. Mit Press, 1999.
[40] R. Marvin and T. Linzen. Targeted syntactic evaluation of language models. arXiv preprint arXiv:1808.00331 2018.
12

---

[41] R. T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics natural language inference. In 57th Annual Meeting of the Association for Compu- national Linguistics, ACI. 2019, pages (ACL), 2020.
Association for Computational Linguistics
[42] T. McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, D. Hassabis, B. Kim, U. Paquet, and V. Kramnik, Acquisition of chess knowledge in alphazero. arXiv preprint 2021.
[43] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.05262, 2022.
[44] A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.-. Lee, E. Johnson, O. Pathak, A. Nazi, J. Pak, A. Tong, K. Srinivasa, W. Hang, E. Tuncer, Q. V. Le, J. Laudon, R. Ho, R. Carpenter, and Dean. A graph placement methodology for fast chip design. Nature, 594(7862)2077-21,, June 2021.
[45] H. Nielsen. The exciting impact of a game changer: When Magnus met AlphaZero. New In Chess, 2019.
[46] L. Pandia and A. Etringer. Sorting through the noise: Testing robustness of information processing in pre-trained language models. In EMNLP (1), 2021.
[47] J. Pawlewicz, R. Hayward, P. Henderson, and B. Arneson. Stronger virtual connections in hex. IEEE Transactions on Computational Intelligence and Al Games, 7(2):156-16) 2014.
[48] A. Poliak, A. Haldar, R. Rudinger, E. Hu, E. Pavlick, A. S. White, and B. Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. arXiv preprint arXiv: 1804 08207, 2018
[49] A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme. Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.0104 2018
[50] M. Popova, O. Isayev, and A. Tropsha. Deep reinforcement learning for de novo drug design. Science Advances, 4(7)xeaap7885, July 2018.
"immortal [51] A. Radić. AlphaZero's "immortaa zugzwang game" against Stockfish, 2017. URL https http///bb..cccmmaaaaaa/////////p
[52] T. Romstad, M. Costalba, J. Kiiski, and others. Stockfish, 2008
[53] M. Sadler and N. Regan. Game changer. AlphaZero's Groundbreaking Chess Strategies and the Promise of AI. Alkmaar. The Netherlands. New in Chess, 2019.
[54] M. Seymour, Hex: Strategy Guide. 2019. URL http::/www.meeymouu..oomm wawpuuuua.aacaebbbeeeeeooeoe hexstrat.html
[55] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587): 484-489. Jan. 2016.
[56] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144 1144, Dec. 2018.
[57] K. Takada, H. lizuka, and M. Yamamoto. Reinforcement learning for creating evaluation function using convolutional neural network in hex. In 20/7 Conference on Technologies and Applications of Artificial Intelligence (TAAI), pages 196-201. IEEE, 2017.
13

---

[58] I. Tenney. D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593- 4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ 1452. URL htapga/://cc.anhhooogyy.oo Ptr/ 1452
[59] I. Tenney, Xia, B. Chen, A. Wang, A. Poliak, R. T. McCoy, Kim, B. V. Durme, Bowman, D. Das, and E. Pavlick. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019. URL https://openrreiiet http:///pppppneeeeeeeeeeeerrrrrr
[60] Y. Tian, Ma, Q. Gong, S. Sengupta, Z. Chen, J. Pinkerton, and L. Zitnick. Elf opengo: An analysis and open reimplementation of alphazero. In International Conference on Machine Learning, pages 6244-6253, 2019.
[61] N. Tomlin et al. Understanding game-playing agents with natural language annotations. To Appear: ACL, 2022.
[62] E. Voita and I. Titov. Information-theoree probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183-196, Online, Nov. 2020. Association for Computational Linguistics. doi: 10. 10111886//222000 120o6665551222000mmmmmmmm.m....
URL taattpp://maa http://wwwwwwwww.../////// aclweb. org/ wwgganthology/202
emn smnlp-maing 14
[63] A. Warstadt, A. Parrish, H. Liu, A. Mohananey, W. Peng, S.-F. Wang, and S. R. Bowman. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377-392, 2020.
[64] A. Warstadt, Y. Zhang, X. Li, H. Liu, and Bowman. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proc. of EMNLP, 2020. URL / /wwww www.eb.rrg//eoo...ooooo..oo..o.t htt///athholoo/o//ee emnn0mmain...n......mm
[65] E. Winter, html 1997. Accessed: 2021-11-12.
Zugzwang.
[66] S. Witty, J. K. Lee, E. Tosch, A. Atrey, M. Littman, and Jensen. Measuring and characterizing generalization in deep reinforcement learning. arXiv preprint arXiv:1812.02888, 2018.
[67] S. Witty, J. K. Lee, E. Tosch, A. Atrey, K. Clary, M. L. Littman, and D. Jensen. Measuring and characterizing generalization in deep reinforcement learning. Applied Al Letters, 2(4):e45, Dec. 2021.
[68] J. Zech, J. Forde, J. J. Titano, D. Kaji, A. Costa, and E. K. Oermann. Detecting insertion, substitution, and deletion errors radiology reports using neural sequence-to-sequeenc models. Ann Transl Med, 7(11):233, June 2019.
[69] J. R. Zech, J. Z. Forde, and M. L. Littman. Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging. Dec. 2019.
[70] A. Zhang, Y. Wu, and J. Pineau. Natural environment benchmarks for reinforcement learning. arXiv preprint 1xXxvv1811.00 2018
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 3.4
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
See
Section 3.4
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
14

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 3.2
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report one standard deviation above and below the mean in Fig. 4a and Fig. 6
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Our hardware used and compute estimates is described in Section 3.1 We will include breakdown of compute costs in the Supplementary Materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We acknowledge the work of Jones (30) in Section 3
(b) Did you mention the license of the assets? [Yes] See Section 3
(c) Did you include any assets either in the supplemental material or as URL? [Yes] See Section 3.1
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 3.4
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "yfMXBfLapX", "reviewer": "Reviewer_8eDq", "summary": "The authors use linear probing to identify pre-defined Hex concepts in the activations of the agent. They use behavioural tests to evaluate whether the concepts that were identified with linear probing are actually enacted by the agent. They identify that AlphaZero does indeed represent concepts, but inconsistently enacts some of them (negative ones - the concepts related to which moves the agent should not play). They find that short term concepts (those related to winning the game) are represented in later layers in the agent and long term concepts (those related to gamestates that are distant from a potential winning state) are represented in middle layers. They also find that the learned embeddings reflect the structure of the game board. ", "questions": "Why did the authors choose to study the game Hex? How many people play this game? Was the choice because there are known concepts for perfect play? Wouldn't chess or Go have been a more interesting game to study, given its much wider player base and the possibility to identify new concepts learned by superhuman players? The reasons behind this decision should be explained in the paper.", "limitations": "The authors do little to address the limitations of the method of linear probing and behavioural studies - namely, that it relies on a bank of pre-identified human concepts, consequently making it difficult to understand how the network plays so well beyond concepts humans already know. Nor does it really give us a deep picture about how the network actually achieves these feats of play. The authors adequately address the potential societal impact of their work. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I'm inclined to believe the authors when they describe their work as concurrent to the (very similar) work of McGrath et al. (2021), which means the author's work is at least somewhat novel. However, the authors sell this work as borrowing methods from NLP, when in fact those methods are linear probes and behavioural studies, methods that are sufficiently general that it can't be accurate to describe them as NLP methods (even the reference the authors use for linear probing is from image classifiers; Alain and Bengio, 2017). Consequently, the discussion in the introduction regarding NLP ought to be changed to reduce the implied links between NLP and the present work. The experiments are also not especially extensive and do little to address the weaknesses of prior work, particularly those of McGrath et al., which is that they rely on a pre-defined set of concepts. Nor is it possible (in Hex) to use this method to improve human play, since perfect play for Hex is known, at least for boards of certain sizes. It would have been interesting and novel to expand on prior work to see whether AlphaZero uses novel concepts in settings where perfect play is unknown.  \n\nOverall the paper is reasonably well presented. However, figure 6 is unclear. Why are there lines connecting concepts in the same group? The authors should use a consistent name for 'structural'/'neighbours' in both the figure, its caption, and the text.\n\nThe choice of the game of Hex warrants some explanation, especially given that it dramatically reduces the audience of people who are interested in the game compared with Go or Chess. \n\nStructural comment: Put long term vs short term (sec 2.1) into the concept taxonomy (sec 2.2)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M4s2sdovUhN", "reviewer": "Reviewer_gsoX", "summary": "The paper investigates AlphaZero's internal representations in the game of Hex by using two evaluation techniques: model probing and behavioral tests. They use probing classifiers to determine if agents can encode conceptual information and design behavioral tests to measure whether the agents can use the concepts to win games. ", "questions": "* The design of behavior tests in this paper is too simple. In the experiment, the test problems are designed to be 1 or 2 moves to win and lose if playing on the wrong move. It is obvious that AlphaZero can play on the right move since it has already achieved superhuman levels. That is to say, regardless of whether the AlphaZero agents have learned the so-called \"concept\", they will always play on the right move by choosing the move with the highest win rate. Overall, the behavioral test experiment is not convincing and does not clarify whether AlphaZero utilizes the learned concept to do things.\n \n* In Figure 4 (a). In [2], selectivity was used to check the complexity of the probing model, to prevent the probing model from being too complex and overfitting. However, in this paper, there is no probing model complexity comparison, only the linear classifier was used. The author should explain what is the meaning of selective in the paper.\n \n* In line 213, the paper mentions that they have similar results with McGrath et al. [1].  (short-term was best represented higher in the network and long-term was largely represented lower in the network.)  According to [1], there is no significant evidence that short-term concepts (McGrath et al. [1]’s Fig. 2)[c,e,f] are better represented in higher layers.\n \nOther minor issues:\n* line 192, 193, what are the definitions for Logit,  z-score? \n* In Figure 6, what do “behavioral:start” and “probing:start” mean? The definitions of each color point are unclear.\n* In Supplementary line 34,  what A,B,C refer to?\n* In Supplementary Figure7, NCDG -> NDCG\n\n[1] T. McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, D. Hassabis, B. Kim, U. Paquet, and V. Kramnik. Acquisition of chess knowledge in alphazero. arXiv preprint arXiv:2111.09259,2021.\n[2] J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743", "limitations": "The authors have addressed most of the possible limitations and potential negative societal impact in Section 3.4.", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper combines both representational and behavioral approaches to analyze reinforcement learning agents in the game of Hex. Overall, the result is quite interesting and allows the readers an understanding of how AlphaZero agents learn. However, the method used in the paper has already existed and has been widely used in many other board games, such as Chess[1], and Go[2]. The paper simply applies these methods to a different game (Hex). I think the novelty is not enough and the results in this paper will be limited to the whole community.\n\n[1] T. McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, D. Hassabis, B. Kim, U. Paquet, and V. Kramnik. Acquisition of chess knowledge in alphazero. arXiv preprint arXiv:2111.09259,2021.\n[2] N. Tomlin et al. Understanding game-playing agents with natural language annotations. To Appear: ACL, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JvQQNGcoHs5", "reviewer": "Reviewer_YogY", "summary": "This paper studies models trained with AlphaZero for the game of Hex. The paper applies techniques from the NLP literature to investigate whether the learned model learns concepts usually taught to human players. The paper further investigates if the trained agent is able to act according to such concepts (e.g., connect a bridge when that is needed to win the game or to reach an advantageous state of the game). ", "questions": "Why did you define short term as single moves to the end of the game and more than one move as long term? It would be interesting to see a continuum between short and long term. For example, one could argue that 2 moves to the end of the game is still short term. \n\nWhy is B a captured cell in Figure 2 (j)? This should be explained in the main text.\n\nIs Hˆ{(l)} a board or the activation values of the network? I would have appreciated a formal definition of weights and activation values as I am not sure I would be able to implement the Probe classifier with the current description from the paper. For example, do we use the output values of all neurons of all layers as input to the classifier? Do we use the logits or the values after activation? ", "limitations": "The experiments were performed on a single domain and it isn't clear whether they will generalize to other domains (e.g., security games or even other board games). \n\nThe experiments and results aren't exactly actionable. They are of scientific interest and might inspire others, but it isn't clear what role this paper will have moving forward. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is well written and the experiments reported are interesting. Although the metrics and procedures come from the NLP literature, this paper is the first to attempt them in models learned with AlphaZero. I find it particularly interesting the Probe procedure, where a linear classifier is trained to detect known concepts of Hex. Interestingly, the concepts studied are detected with very high accuracy and the accuracy improves as AlphaZero further trains the model. This shows an interesting connection between the learned model and concepts that we (humans) deem as important for playing the game. \n\nThere is a chance this paper will inspire others to work on the extraction of such concepts from the learned model. This future research endeavor could be important for those interested in teaching humans the knowledge that is generated with learning systems. \n\nThe paper also makes interesting connections with previous work. I particularly like the connection between the results presented in the paper and those of McGrath et al. [37] on chess regarding where in the neural model that the short and long-term decisions are stored. Such connections facilitate the work of others starting out in this field. \n\nIt was also interesting to see the methodology the paper uses to verify that the learned agent learns the connections in the board of Hex. \n\nWeaknesses\n\nSome of the results reported in the paper aren't as interesting as the idea of running the experiments. For example, short-term decisions are encoded near the output layers of the neural model. Perhaps this isn't surprising because such decisions are actionable and one would expect the last year of the model (which is also linear) to correctly identify such situations. \n\nAnother result that perhaps isn't particularly interesting is the fact that the studied concepts are first noted in the MCTS search and only later incorporated in the model. This is because MCTS generates the training data for the model, so the natural order is to first generate the data and then train the model. It would be strange if the model learned before MCTS collected the data. \n\nIt is interesting that the concepts 'escape' and 'bottleneck' are first detected in probing to only then being detected in decisions of the agent (Figure 6 of the paper). I list this as a 'weakness' because the paper offers no explanation of why this is the case. This result almost contradicts the idea of MCTS generating training data and the model learning from the training data. This is because the concept can be first noted in the neural model to only then appearing as decisions in the game (behavior). I apologize if I missed the explanation of this phenomenon and would appreciate a response from the authors about it. \n\nOther comments\n\nRegarding the following: \"This highlights a weakness in AlphaZero and a risk: Some concepts may be provable and useful to people, but deemed less important by AZ.\" The agent is probably playing dead cells because it has either won or lost the game and it doesn't matter if it plays on those cells. If this is the cases, then this isn't a weaknesses of AlphaZero, but rather a challenge for systems that will attempt to extract information from the learned model to transfer to human learners. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-3o_K-8vpr-", "reviewer": "Reviewer_CKwV", "summary": "In this paper, the authors use *probing* and *behavioral* tests to investigate to what extent, and when, AlphaZero-style DNNs learn to use various concepts that humans are also known to use in the board game Hex. \n\nThe analysis shows that the DNN tends to consistently pass many of the tests, for almost all of the evaluated concepts, after about 75% training progress. Negative concepts (which recognise cells that are useless to play in, such as dead cells) are an exception; these appear to still not be fully \"understood\" even at the end of training, and this can be observed in the agent's behaviour as well where it sometimes does not win as fast as it could, wasting some moves instead. For most concepts, the agent's performance on behavioral tests tends to improve before the performance on probing tests, which suggests that the full MCTS agent may learn how to use them before this knowledge gets encoded into the DNN. In general, it appears that long-term concepts tend to be encoded in middle layers, whereas short-term concepts tend to be encoded closer to the final layers.\n\nThe overall methodology used for the analysis described above may be applicable to other games / other RL environments.", "questions": "(1) I'd like to start with a general question if you have any comments or clarifications on the weaknesses I listed above?\n\n(2) For the probing dataset, it is described that random enemy pieces are added to the board. How many? There is no mention of randomly adding friendly pieces. Is it always guaranteed that ultimately the resulting game states constitute positions that could have been obtained through legal gameplay?", "limitations": "- It may be useful to discuss the apparent limitation that quite a bit of Hex-specific knowledge is used for setting up the probes (discussed in more detail as a weakness above).\n- It may be useful to discuss the potential limitation I discussed in more detail above that the behavioral tests may simply all be testing for an agent's ability to recognise when it needs to interrupt an opponent's immediate winning threat.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Interesting and important topic, with potential applications to improve our understanding and ability to interpret what deep RL models have learned (and when they learn it, and what they fail to learn).\n2. Well-written, largely easy to follow.\n3. Discussions of related work seems good, don't see anything missing there, though my background and awareness of existing work is more so in RL and games than in probing/interpretability angle.\n\nWeaknesses:\n1. I believe that one small (but important) part of the paper could use some clarifications in the writing: Section 3.2 (on Representational Probing). I will elaborate below.\n2. I think that a couple of claims in the paper may be slightly too strong and need a bit more nuance. I will elaborate below.\n3. A lot of the details described in Section 3.3 (Behavioral Tests) seem quite specific to the game of Hex. For the specific case of Hex, we can indeed know how to create such states that (i) contain a concept, (ii) contain that concept in only exactly one place, (iii) make sure that the agent **must** play according to the concept immediately, because otherwise it would lose. I imagine that setting up such specific situations may be much more difficult in many other games or RL environments, and would certainly require highly game-specific knowledge again for such tasks. This seems like a potential limitation (which doesn't seem to be discussed yet).\n\n---\n\n**On weakness (1)**: \n\nThe first sentence that I personally found confusing was \"To form the random control, for each board $(H^{(0)}, y)$ in the probing dataset, we consistently map each cell in that board to a random cell, forming $H_s^{(0)}.\" I guess that \"map each cell in that board to a random cell\" means creating a random mapping from all cells in the original board to all cells in the control board, in a way such that every original cell maps to exactly one randomly-selected control cell, and every control cell is also mapped to by exactly one original cell. And then, the *value* of each original cell (black/white/empty) is assigned to the control cell that it maps to. I guess this is what is done, and it makes sense, but it's not 100% explicit. I'm afraid that many readers could misunderstand it as simply saying that every cell gets a random value directly.\n\nThen, a bit further down under Implementation Details, it is described how the boards in the probing dataset get constructed. I suspect it would make more sense to actually describe this *before* describing how the matching controls are created.\n\n---\n\n**On weakness (2)**:\n\n(a) The behavioral tests involve states created specifically such that they (i) contain the concept, but also (ii) **demand** that the agent **immediately** plays according to the concept, because it will lose otherwise. In the game of Hex, this means that all of these board states, for all these different concepts, actually include one more new \"concept\" that is shared across all the tests; a concept that recognises a long chain of enemy pieces that is about to become a winning connection if not interrupted by playing in what is usually just one or two remaining blank cells in between. So, I do not believe that we can say with 100% certainty that all these behavior tests are actually testing for the concept that you intend them to test for. Some or all of them may simply be testing more generally if the agent can recognise when it needs to interrupt the opponent's soon-to-be-winning chain.\n\n(b) \"Fig. 5 shows evidence that some information is learned before the model is able to use the concepts.\" --> I think \"evidence\" may be too strong here, and would say something more like \"Fig. 5 **suggests** that some information **may** be learned [...]\". Technically, Fig. 5 just shows that there is generally a long period with no progress on the tests, and after a long time suddenly rapid progress on the tests. To me this indeed suggests that it is *likely* that it is learning something else first, but it is not hard evidence. It could also be that it's just randomly wandering about the parameter space and suddenly gets lucky and makes quick progress then, having learned nothing at all before.\n\n(c) \"Behavioral tests can also expose heuristics the model may be using.\" --> yes, but only if we actually already know that the heuristics exist, and know how to explicitly encode them and create probes for them. They can't teach us any new heuristics that we didn't already know about. So maybe, better phrasing could be something like \"Behavioral tests can also confirm whether or not the model may be using certain heuristics.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept