Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Authors: Kenneth Li
Aspen K Hopkins
David Bau
Fernanda Viégas
Hanspeter Pfister
Martin Wattenberg
Venue: ICLR 2023 notable top 5% (2023)
Keywords: world representation
GPT

Abstract: Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create "latent saliency maps" that can help explain predictions in human terms.

Published as a conference paper at ICLR 2023
EMERGENT WORLD REPRESENTATIONS: EXPLORING A SEQUENCE MODEL TRAINED ON A SYNTHETIC TASK
Kenneth Li Harvard University
Aspen K. Hopkins
David Bau
Massachusetts Institute of Technology
Northeastern University
Fernanda Viégas Harvard University
Hanspeter Pfister
Martin Wattenberg
Harvard University
Harvard University
ABSTRACT
Language models show surprising range of capabilittis, but the source of their apparent competence is unclear. Do these networks just memorize collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question in synthetic setting by applying variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network. By leveraging these intervention techniques, we produce "latent saliency maps" that help explain predictions.
INTRODUCTION
Recent language models have shown an intriguing range of capabilities. Networks trained on a simple "next-word" prediction task are apparently capable of many other things, such as solving logic puzzles or writing basic code. 2 Yet how this type of performance emerges from sequence predictions remains a subject of current debate.
Some have suggested that training on a sequence modeling task is inherently limiting. The arguments range from philosophical (Bender Koller 2020) mathematical (Merrill al. 2021). common theme is that seemingly good performance might result from memorizing "surface statistics," i.e., a long list of correlations that do not reflect causal model of the process generating the sequence. This issue is of practical concern, since relying on spurious correlations may lead to problems on cut-of-distribution data (Bender al., 2021 Floridi & Chiriatti 2020).
On the other hand, some tantalizing clues suggest language models may do more than collect spurious correlations, instead building interpretable world models -that is, understandable models of the process producing the sequences they are trained on. Recent evidence suggests language models can develop internal representations for very simple concepts, such as color, direction Abdou et al.
(2021) Patel & Pavlick 2022), tracking boolean states during synthetic tasks (Li al. 2021 (see Related Work (section 6) for more detail).
A promising approach to studying the emergence of world models is used by Toshniwal et al. (2021) which explores language models trained on chess move sequences. The idea is to analyze the behavior of standard language modeling architecture in well-understood, constrained setting. The paper finds that these models learn to predict legal chess moves with high accuracy. Furthermore, by analyzing predicted moves, the paper shows that the model appears track the board state. The authors stop short, however, of exploring the form of any internal representations. Such an
" "Correspondence ke 1i8g hww.ard..adddre. edu I 'Codes at hts //qqittttbbb com/ www.entteoohoooooo hothell wo_world "See Srivastava al. 2022 for encyclopedic list of examples.

---

Published as a conference paper at ICLR 2023
investigation will be the focus of this paper. A key motivation is the hope that ideas and techniques learned in this simplified setting may eventually useful in natural- -atuual-angguge settings as well.
1.1 THE GAME OF OTHELLO AS TESTBED FOR INTERPRETABILITY
Toshniwal et al. 2021) 's observations suggest a new approach to studying the representations learned by sequence models. If we think of a board as the "world," then games provide with appealing experimental testbed to explore world representations of moderate complexity. As our setting, we choose the popular game of Othello (Figure 1}, which is simpler than chess. This setting allows to investigate world representations in highly controlled context, where both the task and sequence being modeled are synthetic and well-understood.
As first step, we train a language model (a GPT variant we call Othello-GPT) to extend partial game transcripts (a list of moves made by players) with legal moves. The model has no priori knowledge of the game or its rules. All it sees during training is series of tokens derived from the game transcripts. Each token represents a tile where players place their discs. Note that we do nor explicitly train the model to make strategically good moves or to win the game. Nonetheless, our model is able to generate legal Othello moves with high accuracy.
Our next step is to look for world representations that might be used by the network. In Othello, the "world" consists of the current board position. A natural question is whether, within the model, we can identify representation of the board state involved in producing its next move predictions. To study this question, we train a set of probes, i.e., classifiers which allow us to infer the board state from the internal network activations. This type of probing has become a standard tool for amalyzing neural networks (Alain & Bengio 2016 Tenney et al. 2019 Belinkov 2016).
Using this probing methodology, find evidence for emergent world representation. In particular, we show that a non-linear probe is able to predict the board state with high accuracy (section 3). (Linear probes, however, produce poor results.) This probe defines an internal representation of the board state. We then provide evidence that this representation plays a causal role in the network's 's predictions. Our main tool is intervention technique that modifies internal activations so that they correspond to counterfactual board states.
We also discuss how knowledge of the internal world model can be used as an interpretability tool. Using our inivaationinterventiio technique, we create latent saliency maps, which provide insight into how the network makes given prediction. These maps are built by performing attribution at a high-level setting (the board) rather than a low-level one (individual input tokens or moves).
To sum up. we present four contributions: (1) we provide evidence for an emergent world model in a GPT variant trained to produce legal moves in Othello; (2) we compare the performance of linear and non-linear probing approaches, and find that non-linear probes are superior in this context; (3), we present an intervention technique that suggests that, in certain situations, the emergent world model can be used to control the network's ' behavior; and (4) we show how probes can be used to produce latent saliency maps to shed light on the model's predictions.
"LANGUAGE MODELING" OF OTHELLO GAME TRANSCRIPTS
Our approach for investigating internal representations of language models is to narrow our focus from natural language to a more controlled synthetic setting. We are partly inspired by the fact that language models show evidence of learning to make valid chess moves simply by observing game transcripts in training data (Toshniwal et al. (02T). We choose the game Othello, which is simpler than chess, but maintains a sufficiently large game tree to avoid memorization. Our strategy is to what, if anything, a GPT variant learns simply by observing game transcripts, without any a priori knowledge of rules or board structure.
The game is played on an 8x8 board where two players alternate placing white or black discs on the board tiles. The object of the game is to have the majority of one's color discs on the board at the end of the game. Othello makes a natural testbed for studying emergent world representations since the game tree is far too large memorize, but the rules and state are significantly simpler than chess.
2

---

Published as a conference paper at ICLR 2023
B
C
2 3
6
2
5
7
2
5
7
Figure I: A visual explanation of Othello rules, from left to right: (A) The board is always initialized with four discs (two black, two white) placed in the center of the board. (B) Black always moves first. Every move must flip one or more opponent discs by outflanking- sandwiching- the opponent disc(s). (C) The opponent repeats this process. game ends when there are no more legal moves.
The following subsections describe how we train system with no prior knowledge of Othello to predict legal moves with high accuracy. The system itself is not our end goal; instead, it serves as our object of study.
2.1 DATASETS: "CHAMPIONSIII" AND "SYNTHETIC"
We use two sets of training data for the system, which call "championship" and "synthetic", Each captures different objectives. namely data quality vs. quantity. While limited in size, championship data reflects strategic moves by expert human players. The synthetic data set is far larger, consisting of legal but otherwise random moves.
Our championship dataset produced by collecting Othello championship games from two online sources containing 7, 605 and 132, 921 games, respectively. They are combined and split randomly by 8 into training and validation sets. The games in this dataset were produced by matches where human players presumably made moves with a strategic intent to win. Following this, we generate a synthetic dataset with 20 million games for training and 3, 796, 010 games for validation. We compute this dataset by uniformly sampling leaves from the Othello game tree. Its data distribution is different from the championship games, reflecting no strategy.
2.2 MODEL AND TRAINING
Our goal to study how much Othello-GPT can learn from pure sequence information, so we provide as few inductive biases as possible. (Note the contrast with system like AlphaZero (Silver et al. 2018), where the goal was to win highly competitive chess games.) We therefore use only sequential tile indices imput to model. For example, A4 and H6 indexed the 4th and the 58th word in our vocabulary, respectively. Each game is treated as a sentence tokenized with a vocabulary of 60 words, where each word corresponds to one of the 60 tiles on which players put discs, excluding the 4 tiles in the center (Figure I).
We trained an 8-layer GPT model (Radford et al. 2018 2019 Brown al. 2020 with 8- head attention mechanism and a 512-dimensional hidden space. The training was performed in an autoregressive fashion. For each partial game {yr) the computation process starts from indexing a trainable word embedding consisting of the 60 vectors, each for one word, to get {x:) They are then sequentially processed by 8 multi-head attention layers. We denote the intermediate feature for the /-th token after the 1-th layer as z/. By employing a causal mask, only the features at the immediately preceding layer and earlier time steps z'ze" are visible to z/. Finally, - goes through a linear classifier to predict logits for शτ. We minimize the cross-entropy loss between ground-truth move and predicted logits by gradient descent.
(-1
The model starts from randomly initialized weights, including in the word embedding layer. Though there are geometrical relationships between the 60 words (c.g., C4 is below B4), this inductive bias is not explicitly given to the model but rather left to be learned.
www. vecthel com com and www. ffothel 11 1o.org
3

---

Published as a conference paper at ICLR 2023
2.3 OTHELLO-GPE USUALLY PREDICTS LEGAL MOVES
We now evaluate how well the model's predictions adhere to the rules of Othello. For each game in the validation set, which was not seen during training, and for each step in the game, ask Othello-GPT to predict the next legal move conditioned by the partial game before that move. We then calculate the error rate by checking if the top-1 prediction legal. The error rate 0.01% for Othello-GPT trained on the synthetic dataset and 5.17% for Othello-GPT trained on the championship dataset. For comparison, the untrained Othello-GPT has an error rate of 93.29%. The main takeaway that Othello-GPT does far better than chance in predicting legal moves when trained on both datasets.
potential explanation for these results may be that Othello-GPT is simply memorizing all possible transcripts. To test for this possibility, we created skewed dataset of 20 million games to replace the training set of synthetic dataset. the beginning of every game, there are four possible opening moves: C5, D6, E3 and F4. This means the lowest layer of the game tree (first move) has four nodes (the four possible opening moves). For our skewed dataset, we truncate one of these nodes (C5), which is equivalent to removing quarter of the whole game tree. Othello-GPP trained on the skewed dataset still yields an error rate of 0.02%. Since Othello-GPT has seen none of these test sequences before, pure sequence memorization cannot explain its performance
If the performance of Othello- Gerllo--PTiia is not due to memorization, what is it doing? We now turn to this question by probing for internal representations of the game state.
3 EXPLORING INTERNAL REPRESENTATIONS WITH PROBES
We seek to understand if Othello-GPT computes internal representations of the game state. One standard tool for this task is "probe" (Alain & Bengio 2016 [elinnov|| 2016: Tenney et 2019). A probe is a classifier regressor whose input consists of internal activations of network, and which is trained to predict a feature of interest, e.g., part of speech or parse tree depth (Hewitt & Manning 2019). If we are able to train an accurate probe, it suggests that representation of the feature is encoded in the network's s activations.
In our case, we want to know whether OPh'ls internal activations contain representation of the current board state. To study this question, we train probes that predict the board state from the network's internal activations after a given sequence of moves. Note that the board state-w whhthhr each tile is empty or holds a black or white disc-is generally a nonlinear function of the input tokens. On the other hand, is straightforward write a program to compute this function, it makes a natural probe target
We take the autoregressive features x/ that summarize the partial sequence Vęt as the input to the probe and study results from different layers l. The output ps (= is 3-way categorical probability distribution. We randomly split pairs of internal representation and ground-trum truth tile state by 8 2 into training and validation set. Error rates on validation set are reported. A best random guess yields 52.95%, if the probe always guess the tile is empty.
3.1 LINEAR PROBES HAVE HIGH ERROR RATES
Our first result is that linear classifier probes have poor relative accuracy. Its function can be written as (a softmax 2 where 0 = {W E gFx3y F is the number of dimensions of input 2/- As Table I shows, error rates never dip below 20%. As a baseline, we have included probes trained on randomly initialized network( We when we move to probing a fully-trained network. This result suggests that if there is an internal representation of the board state, it does not have simple linear form.
that there is only marginal improvement in accuracy
*Note that even truncated the game tree may include some board states in the test dataset, since different move sequences can lead to the same board state. However. our goal is to prevent memorization of input data; the network only sees moves, and never sees hoard state directly.
*Classifying tile unoccupied occupied we consider only the 3-way black/ hhec///ttt/emptt classificrs.
be written linear function of the input sequence, thus
'Probes on the randomized network do better than chance; a constant guess of "empty" has 47% error rate. But that performance is not surprise, since it makes sense that some information about moves is preserved even by random network. The key comparison is between the randomized network and the trained network.
4

---

be seen in Published as a conference paper at ICLR 2023
2:'
x''
x²a
Randomized
26.7
27.1
27.6
28.0
28.3
28.5
28.7
28.9
Championship
24.2
23.8
23.7
23.6
23.6
23.7
23.8
24.3
Synthetic 21.9
20.5
20.4
20.6
21.1
21.6
22.2
23.1
Table 1: Error rates (%) of linear probes randomized Othello-GPT and Othello-GPTs trained on different datasets across different layers (a' represents internal representations after the i-th layer).
3.2 NONLINEAR PROBES HAVE LOWER ERROR RATES
Given the poor performance of linear probes, it is natural ask whether a nonlinear probe would have higher accuracy. Moving up one notch of complexity. we apply 2-layer MLP as a probe. This technique has been used successfully in other language model probing work, e.g., Conneaul 20188; Cao et al. (2021); Hernandez & Andreas (202I). Its function can be written as ct al.
(a softmax W ReLU(W2222 222 where 0 {Wi RR MX3 W2 B FFxH). 3. H the number of
hidden dimensions for the nonlinear probes.
The probe accuracy for trained networks, shown in Table 2, is significantly better than the linear probe in absolute terms. By contrast, the baseline (probing randomized network with nonlinear probes) shows almost no improvement over linear case. These results indicate that the probe may be recovering nontrivial representation of board state in the network's activations. In section 4) we describe intervention experiments validating this hypothesis.
x:'
x
2.6
x''
Randomized
25.5
25.4
25.5
25.8
26.0
26.2
26.2
26.4
Championship
12.8
10.3
9.5
9.4
9.8
10.5
11.4
12.4
Synthetic 11.3
7.5
4.8
3.4
2.4
1.8
1.7
4.6
Table 2: Error rates (%) of nonlinear probes on randomized Othello-GPT and Othello-GPTs trained on different datasets across different layers. Standard deviations are reported in Appendix H
VALIDATING PROBES WITH INTERVENTIONAL EXPERIMENTS
Our nonlinear probe accuracies suggest that Othello-GPP computes information reflecting the board state. It's not obvious, however, whether that information causal for the model's predictions. In the following section, we adhere to [elinkkvv (201))'s recommendation, performing a set of interventional experiments to determine the causal relationship between model predictions and the emergent world representations.
To determine whether the board state information affects the network's predictions, we influence internal activations during Othello-GPT's calculation and measure the resulting effects. At a high level, the interventions are as follows: given a set of activations from the Othello-GPT, a probe predicts baseline board state B. We record the move predictions associated with B, then modify these activations such that our probe reports an updated board state B'. Through our protocol, only single tile distinguishes B' from B's board state (an example of which Figure 2}. This small modification results in a different set of possible legal moves for B'. If the new predictions match our expectations for and not the predictions we recorded for B-we conclude the representation had a causal effect on the model.
4.1 INTERVENTION TECHNIQUE
To implement an intervention that changes the predicted state from board position to modified version B' we must decide (a) which layers to modify activations in, and (b) how to modify those activations. The first question subtle. Given the causal attention mechanism of GPT-2, modifying activations for only one layer is unlikely to be effective later layer computations incorporate information from prior board representations unaffected by our intervention. Instead, we select an initial layer Lg then modify it and subsequent layers' activations (see Figure (C)). Our modification uses simple gradient descent method on the probe's 's class score for the particular tile s whose state being modified.
5

---

Published as a conference paper at ICLR 2023
Pre-Intervention
Post-Intervention
A
Intervention
B
Probala By
White
Prediction
100) 2-Layer
Classifer Boundary
Current Stose (to -Cihh)
Black
World State
Cestred Stane
100 (lack))
L
Layer
Layer
Layer
Layer
Timestep
T-1
Timestep
T-1
Timestep
T-1
Timestep
T-1
Figure 2: (A) explains how intervene on board tile. Here, we only want to flip one tile, e.g. E6, from white to black. (B), four views present an Othello game in progression, which can be reliably probed from internal representation The lower left board represents the model's perceived world state prior to intervention. The upper left board shows the model's predictions for legal moves given this state. Post-intervention, the model's world state is updated updsted-FE6's state has been switched from white to black (lower right), leading to a different set of legal move predictions (upper right). Note that two tiles (E6) are highlighted in the world state boards. This is the tile that we "intervene" on, changing from white to black. (C) Shows our proposed intervention scheme. Light blue indicates unmodified activations: dark blue represents activations affected by intervention. Starting from predefined layer, we intervene at the temporally-Be original internal representations with the post-intervention one and resume computation for the next one layer. Part of the misinformation gets corrected (light blue), but we alternate this intervening and computation process until the last layer, from which the next-step prediction is made.
token (shown in (A)). We replace
Figure illustrates an intervention on single feature into x' such that the corresponding board state B is updated to match the desired B'. We observe the effectiveness of these interventions by probing the intervened x' or x at later layers (see Appendix D), as well as the change in next-step prediction (see subseection 4.2). Consistent with the training process of probes po, entropy loss between the grobe-predictee probability distribution and the desired board state, but rather than optimize probe weights 6, we optimize for intervention
cross
x' t @Ccn(po(r), B')
dr
At timestep T', the internal activations of an L-layer Othello-GPT can be viewed as an L x T grid of activation vectors. Our intervention process runs Othello-GPT sequentially, but gradient descent to modify key activation vectors at the last timestep such that their board state class scores change. Note that if change activations only middle layer, activations higher layers are directly affected by pre-ineeveention information. Thus, we sequentially intervene ( at the last timestep, on all activations starting from preset layer I., until the final layer, illustrated Figure
Note that the learning See on the intervention parareparamcters in Appendix G

---

Published as a conference paper at ICLR 2023
Error on Natural Benchmark
Error on Unnatural Benchmark
3.0
2.5
hall inderrreeeeionn foom
1.0
0.5
0.12
0.5
5
7
8
2
3
Number of Intervened Layers
Number of Intervened Layers
Figure 3: Intervention experiment results. Red dashed lines represents average number of errors by testing pre-intervention predictions on post-intervention ground-truths, representing a null interven- tion method for contrast. The shaded area represents the 95% confidence interval.
4.2 EVIDENCE FOR A CAUSAL ROLE FOR THE REPRESENTATION
To systematically determine if this world representation is causal for model predictions, we create an evaluation benchmark set. test case in this set consists of a triple of partial game, targeted board tile, and a target state. For each case, we give the partial game to Othello-GPT and perform the intervention described in the previous section. Le.. we extract the model's activations mid- computation, modify them change the representation of the targeted board tile into the target state, give back the modified world representation and let make prediction with this new world state.
The benchmark set consists of two subsets of 1000 intervention cases: one "natural," one "unnatural." The natural subset consists of positions reachable by legal play. The unnatural subset contains position that are unreachable by legal play. This second subset is designed to be a stringent test, since it is by definition far from anything encountered in the training distribution.
We measure prediction alignment with the intervened world representation a multi- classifica- tion problem, comparing the top-N predictions against the ground truth legal next-move set, where N is the number of legal next-moves after intervention. We then calculate error per case (a sum of false positives and false negatives, shown in Figure 3 X For both benchmarks, T., = 4 (intervening layers) gives the best result: average errors of 0.12 and 0.06 respectively. Compared baseline errors (2.68 and 2.59), the proposed intervention technique is effective even under unnatural board states, suggesting the emergent representations are causal model predictions.
LATENT SALIENCY MAPS: ATTRIBUTION VIA INTERVENTION
The intervention technique of the previous section provides insight into the predictions of Othello- GPT. We can also use it to create visualizations which contextualize Othello -PPt's predictions in terms of the board state. The basic idea is simple. For each tile s on the board B, we ask how much the network's prediction probability for the attributed tile p will change if apply the intervention technique in secteciion 4 to change the state representation of that tile s. This will yield a value per tile E B, positive or negative, corresponding its saliency in the prediction of p (see aalorithhm 1). We then create visualization of the board where tiles are colored according to their saliency for the top-1 prediction for the current board state rather than its input, we call it latent saliency map.
9 Because this map based on the network's 's latent space
Figure 4 shows latent saliency maps for the top-1 predictions for Othello-GPTI trained on the synthetic and championship datasets by intervening from L, = 4. The two diagrams show clear pattern. The synthetic Othello-GPT shows high saliency for precisely those tiles that are required to make a move legal. In almost all cases, other tiles have lower saliency values. Even without knowing how synthetic-GPT was trained, an experienced Othello player might be able to guess its goal. The latent saliency maps for the championship version, however, are more complex. Although tiles that relate directly to legality typically have high values, many other tiles show high saliency as well. This pattern makes sense, too. Expert moves rely on complex global features of the board. The difference
'Qualitative results can be found in Appendix C We report additional metrics in Appendix However, idcally we can attribute any prediction. See Appendix for more discussion.
7

---

Published as a conference paper at ICLR 2023
Predation C
Prediia 57
Preftie Tock., 64
Fallka flak
Prdulia Wake
Pediiia C
Prddicann Wate ut
Pridcee Stckuc
Feeiik Baccal
Proicca Whis 2t
Figure 4: Latent saliency maps: Each subplot shows a different game state, and the top-1 prediction by the model enclosed a black box. Colors (red is high, blue is low) indicate the contribution of square's state this prediction. The contribution is higher when changing the internal representation of this square makes the prediction less likely. The values are normalized by subtracting the mean of the board. (A) Latent saliency maps for Othello-GPP trained the synthetic dataset, where the model learns legal moves. (B) Latent saliency maps for Othello-GPT trained on the championship dataset. Rather than learning rules, this Othello-GPT learns to make strategically good moves.
between the latent saliency maps for the two versions of Othello-GPT suggests that the visualization technique is providing useful information about the two systems.
Algorithm 1: Attribution via Intervention on Othello-GP? GPT
Inputs
B the current board state
p legal next move which we try to attribute
Output:
{S.}.EB assigned sensitivity values for the prediction of
Po < fp(xt-1)
// pre-intervention
probability for p
for 8 E B do
Gt- IIteerventio(zz--1.55
Ps f fp(ā--1)
S, po Pa
RELATED WORK
Our work fits into a general line of investigation into world representations created by sequence models. For example, (Li 2021) fine-tune two language models synthetic natural language
8

---

Published as a conference paper at ICLR 2023
tasks (Long et al. 2016) and find evidence that semantic information about the underlying world state is at least weakly encoded in the activations of the network. More direct evidence of faithful representation of 3D color space comes from Abdou et al. (202I, who examine activations in the BERT model and find a geometric connection to a standard 3D color space. Another study by (Patel & PPavlick] 2022) shows that language models can learn map conceptual domains, e.g.. direction color, onto grounded world representation via prompting techniques (Brown et al. (2020). These investigations operate in natural language domains, but investigate relatively simple world models.
Another related stream of work concerns neural networks that learn board games. There is a long history of work in AI to learn game moves, but in general, these systems have been given some a priori knowledge of the structure of the game. Even one of the most purposse game- playing engines, AlphaZero (Silver et al. 2018), has built-in knowledge of basic board structure and game rules (although, intriguingly. it seems to develop interpretable models of various strategic concepts (McGrath et al. 2021 Forde et al. 2022)) Closer to the work described here-and major motivation for our research-is Toshniwal et al. (2021) which trains language model chess transcripts. They show strong evidence that transtormer networks are building a representation of internal board state, but they stop short investigating what form that representation takes.
The intervention technique secciin 4 follows an approach of steering model output while keeping the model frozen. It is related to the ideas behind plug-and-play controllable text generation for autoregressive (Dathathri et al.) 2019 Qin et al. 2020: Krause et al. 2020) and diffusion (Li et al. 2022) language models by optimizing the likelihood of the desired attribute and the fluency of generated texts at the same time. These methods naturally involve trade-off and require several forward and backward passes to generate. Our proposed intervention method stands out by only working on internal representations and requires only one forward pass. Finally, latent saliency maps can be viewed as a generalization of the TCAV approach Kim et al. (2018); Ghorbani et al. (2019); Koh et al. (2020). In the TCAV setting, attribution is performed via directional derivatives. This is essentially linearization of the gradient-descent optimization used in our attribution maps.
CONCLUSION
Our experiments provide evidence that Othello-GPT maintains representation of game board states that is, the Othello "world"-to produce sequences it was trained on. This representation appears to be nonlinear in an essential way. Further, we find that these representations can be causally linked to how the model makes its predictions. Understanding of the internal representations of sequence model is interesting in its own right, but may also be helpful in deeper interpretations of the network.
We have also described how interventional experiments may be used create "latent saliency map". which gives a picture, in terms of the Othello board, of how the network has made a prediction. Applied to two versions of Othello-GPT that were trained on different data sets, the latent saliency maps highlight the dramatic differences between underlying representations of the Othello-GPT trained on synthetic dataset and its counterpart trained championship dataset.
There are several potential lines of future work. One natural extension would be to perform the same type of investigations with other, more complex games. It would also be interesting compare the strategies learned by a sequence model trained on game transcripts with those of a model trained with a priori knowledge of Othello. One option is to compare latent saliency maps of Othello-GPP with standard saliency maps of an Othello-playing program which has the actual board state as input.
More broadly, would be interesting to study how our results generalize to models trained on natural language. One stepping stone might be to look at language models whose training data has included game transcripts. Will we see similar representation of board state? Grammar engineering tools (Weston et al.) 2015 Hermann et al. 2017] Côté et al., 2018) could help define a synthetic data generation process that maps world representations onto natural language sentences, providing a similarly controllable setting like Othello while closing the distance to natural languages. For more complex natural language tasks, can we find meaningful world representations? Our hope is that the tools described in this paper- nonliner probes, layerwise interventions, and latent saliency maps-ma prove useful in natural language settings.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank members of the Visual Computing Group and the Insight + Interaction Lab Harvard for their early discussions and feedback. We especially thank Aoyu Wu for helping with making part of the figures and other valuable suggestions. We gratefully acknowledge the support of Harvard SEAS Fellowship (to KL). Siebel Fellowship (to AH), Open Philanthropy (to DB), and FTX Future Fund Regrant Program (to DB). This work was partially supported by NSF grant IIS- 1901030.
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode pereeptual structure without grounding? case study in color. arXiv preprint arXiv:2109.06125 2021.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: arXiv: 1610.01644, 2016.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, pp. 1-12, 2016.
Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185-51999 2020.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Steven Cao, Victor Sanh, and Alexander M Rush. Low-complexity probing via finding subnetworks. arXiv preprint 2Xxx:2104.0333 2021.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.0107. 2018
Mlre-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Workshop on Computer Games, pp. 41-75. Springer, 2018.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv: 1912.0269 2019.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681-69) 2020.
Jessica Zosa Forde, Charles Lovering, George Konidaris, Ellie Pavlick, and Michael L Littman. Where, when & which concepts does alphazero learn? lessons from the game of hex. In AAAI Workshop on Reinforcement Learning in Games, volume 2, 2022.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 32, 2019.
Karl Moritz. Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in simulated 3d world. arXiv preprint vrXiv:1706.0655:1 2017.
Evan Hermandez and Jacob Andreas. The -in-dinensional linear geometry of contextualized word representations. arXiv preprint arXiv:2105.077 109, 2021.
10

---

Published as a conference paper at ICLR 2023
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers),
pp. 1139-4138, 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pp. 2668-2677. PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. PMLR, 2020.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.0633 2020.
Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint urXiv:2106.00737, 2021.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- Im improves controllable text generation. arXiv preprini urXiv:2205.142117 2022.
Reginald Long, Panupong Pasupat, and Percy Liang. Simpler context-dependent logical forms via model projections. arXiv preprint arXiv:1606.05377 2016.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, UIrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. arXiv preprint arXiv:2111.00259, 2021.
William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A Smith. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? Transactions of the Association for Computational Linguistics, 9:1047-1060, 2021.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022. URL http:ttt/ppeeeereett
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning. arXiv preprint arXiv:2010.05900 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
David Silver, Thomas Hubert, Julian Schrittwesser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144, 2018.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga- Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04611 2022.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv: /905. /90.099550000......0 2019.
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Learning chess blindfolded: Evaluating language models state tracking. arXiv preprint arXiv:2102. 13249, 2021.
11

---

Published as a conference paper at ICLR 2023
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai- complete question answering: A set of prerequisite toy 2015.
tasks. arXiv preprint
12

---

Published as a conference paper at ICLR 2023
A VISUALIZING THE GEOMETRY OF PROBES
Randomized
Championship
Synthetic
Linear
Nonlinear
Figure 5: Geometry of probe weights. The first row is for linear probe weights; the second row for the second-layer weights of the nonlinear probes. Three columns from left to right mean: probes trained on randomly-initialize GPT, GPTs trained on championship and synthetic datasets respectively. For each subplot, one dot represents one tile on an Othello board, and neighbouring tiles have their two dots connected. Top-left and nightbbotoom corners labelled for reference. Togglable 3D plots can be found in Supplementary Material.
Both linear and nonlinear probes be viewed geometric objects. In the case of linear probes, we can associate each classifier with the normal vector to the separating hyperplane. In the case of nonlinear probes, we treat the second layer of the MIP linear classifier and take the same view. This perspective associates vector to each grid tile, corresponding to the classifier for that grid tile.
A natural question is whether these vectors display any geometric patterns. Figure visualizes their configuration using PCA plots. To make the connection with board geometry clear, we have overlaid a grid in which the vector for a given grid tile is connected to the vectors that represent its neighbors. At left, as baseline, are weights of probes trained on randomized GPTs; the result is somewhat jumbled version of the board's grid. For classifier vectors, however, we see somewhat clearer geometric correlation with the grid shape itself. This shape may reflect set of correlations between neighboring grid cells, and could be an interesting object for future study. One point of interest is that probe geometry for the randomized network does not seem completely random. This may fit with fact that linear probe baseline performance is better than chance, indicating some information about board state can be gleaned from random projections of game move tokens.
B ABLATION ON NONLINEAR PROBE ACCURACIES
Though high accuracies have been observed on nonlinear probes in sectionn 3) we want to develop deeper understanding on them. For example, we wish to understand when, during a game, model has developed world representations of board tiles, where in Othello-GPT that information is stored, how difficult
decode that information, and when the model may forget that information.
As shown in Figure 6 (B), we plot probe accuracies of two-layer probes varying to two different experiment settings: (1) Probe Hidden Units: the number of hidden units H nonlinear probes; (2) Layer: at which layer the representations xf is taken out. With the increase of hidden units, i.e., probe capacity, probe accuracy is higher as it can capture more knowledge from the hidden space. For layer 1, the accuracy peaks at midway, which is aligned with studies natural language (Hewitt & Manning 2019), where linguistic properties are found to best probed mid- hay-layers. The format of how-where plots is similar the Whaa--when-wheee plots in McGrath al. (202T), except ask how many hidden units within our nonlinear probe are necessary to achieve reasonable accuracy given each layer of the GPT instead of looking into number of training epochs.
We are also curious about when (in terms of game progression) these concepts are captured by Othello-GPT. Are these concepts updated immediately after each move? Will they persist or be
13

---

Published as a conference paper at ICLR 2023
Randomized
Championship
Synthetic
Performance Across
Game Progression
0
04
Game Poposim
Gane Pogg
Mloven
(hobs)
Pobii
Accumcy
Performence Across
130
025
Model Layer
015
055
Neben Une.
use
Brots)
Laye
Figure 6: (A) Whut-how-where plots and (B) Whut-how-when plots of nonlinear probes trained on randomly-initialized GPT, on GPT's trained on championship and synthetic datasets respectively. (A) presents probe accuracy across an Othello game progression, while (B) presents accuracy across Othello-GPT layers.
forgotten with newer moves being made? To study this, we divide the data points for probe validation by how many steps the tile has been in its current state and plot what concept can be probed by how powerful probes when in the game progression Figure 6 (A).
For nonlinear probes with moderate number of hidden units, a parabolic accuracy curve is shown: concepts are best captured when they have existed for some time but not too long. This tells us: (I) forgetting happens when Othello-GPP changes its world representations; (2) there is period of uncertainty before changes in board states are updated.
C PREDICTION HEATMAPS OF COUNTERFACTUAL BOARD STATES
In Figure 7 we can see one case of how intervention changes model prediction by intervening on the world representation of Othello-GPTT Note that the set of ground-truth legal moves also changed by the intervention. In this case, both pre-intervention and post-intervention predictions have 0 errors. Figure shows systematic results over 1, 000 cases. A random subset of 100 cases be found in Supplementary Materials.
D INTER-PROBE INTERACTION
In Figure 8 we show the same case as in Appendix C. of the world states probed from layers of Othello-GPT starting the 5th, which is the layer we start to intervene, Lg. can observe that after intervention is successfully done the 5th layer at C4, the flipped disc is corrected in the immediately following layer, as seen the pre-intervention probing results at the 6-th layer. However, when the same intervention is done later layers again, the model is more convinced that C4 should be black and stops to correct it. A total of 100 cases can found in Supplementary Materials, corresponding to examples in Appendix C
14

---

Published as a conference paper at ICLR 2023
Pre-itienartion Pradiction Hestmap
Poot-inteevartion Prediction Healmap
3.00
0.00
0.00
0.50 0.00 3.00
3.00
0.00
CD
0.00
0.11
0.00
06
0:11
8.00
0.00 0.30 2.00
3.00
3.11
0.00
0.10
0.00
0.00
0.00
0.00
1110
5.00
0.50
6.00
0.00
0.30
5.00
0.00
2.11
0.00
0.00
0.00
0.00
0.00
0.00
0.00
6111
8.80
0.60
6.00
0.00
0.30
5.00
0.00
2.12
0.00
W
0.00
0.00 0.00
0.00
0.00
0.00
M13
0.00
0.00
8:11
0.00
0.20
0.00
9:11
2.12
0.00
14
0.80
2.42 0.00 0.00
00
0.13
0.00
0.60
9-
0.50
2.00
2.11
0.00 2.00
0.00
3.00
0.00
50
0.00
0.00 8.12
0.00 00
0.00 0.00
0.60
0.00
8.00
0.00
0.00
2.00
0.00
3.00
0.00
= 0.00
0.00
0.00 0.00
0.00
0.00
0.00
0.00
6
7
8
Figure 7: Heatmaps of the probability allocated to each tile on the board with nonlinear intervention done on Othello-GPT trained synthetic dataset. To the left is the pre-intervention heatmap with pre- intervention legal moves underscored; to the right is post-intervention heatmap, with post- interentervnnnoon legal moves underscored.
Pre-intervemion Phobs Heaut! at 5th Layer
Post-mtervention Phobe Hesuf! the Sth Layer
Preiiteenentoon Probe it the Layer
Post- nost-nteeventto Probe Hesuft the Layer
0
X
X
Pre interveeeettee nntoo Probe Resuit the 7th Layer
Post interventioon Probe Result at the 7th Layer
Pre rne-ineenventon Probe Resuit at the 8th Layer
Post-inaamention Frobe Result the 8th Layer
X
X
9
X
X
X
X
x
X
C
9
x
x
Figure 8: The world states predicted by probes innt--ootnntteeenttion on internal representations at different layers. Color encodes the confidence of the plotted top-1 prediction for each tile.
E DISCUSSION OF LATENT SALIENCY MAPS
In the latent saliency maps created by attribution via intervention in Figure 4 when multiple discs are flipped, only the first one are found contributing, which is slighted misaligned to human understanding of the Othello rule. However, this is expected from the algorithm we are using because other flipped discs, in the opposite states, still make the current prediction legal.
The attribution via intervention method succeeds at visualizing the AND-logiC in the Othello rule to flip one straight line: there should be opponent discs in between and same-color disc at the other end. However, when the prediction flips more than one straight lines, intervening on one them does not nullify the prediction. This is because the Othello rule is an OR-logic move is legal when either one the eight straight lines can be flipped. How to extract such knowledge with unlimited amount of intervention experiments at hand? We leave it for future research.
top of the AND-logic:
15

---

Published as a conference paper at ICLR 2023
Fi on Natural Benchmark
on Unnatural Benchmark
0.95
Aullintereetton Booding 08?
5
3
5
Number of Intervened Layers
Number of Intervened Layers
Figure 9: Same as Figure 3 except for reporting Pi instead of Error.
Divergence Natural Benchmark
KL Divergenice Unnatural Benchmark
hull Rosckne:0.22
0.6
Intervention Saxeline
0.4
0.2
0.00
0.0
Number Intervened Layers
Number Intervened Layers
Figure 10: Same as above but for DEL
F ALTERNATIVE METRICS FOR THE INTERVENTION EXPERIMENT
Following the same multi-label classification framework that calculates the error in Figure 3 we report the Fi score in Figure 9] We measure intervention success as a multi-class classification problem, rather than top-1 prediction problem like in sestiio 3] considering the fact that the later metric is almost saturated for the Othello-GPT trained on synthetic dataset and the intervention in benchmark only cause set difference of 2.12 tiles on average.
Another line of thought to measure the alignment between model prediction and the intervened world representation is by measuring the KL divergence of the predicted next-move probability distribution from a discrete uniform prior distribution over all legal next-moves. Note that the baseline F1 and DEL are the averages of the F1 and D over all 1000 intervention cases, rather than treating the whole benchmark as one single multi-label classification problem or distribution distance calculation.
Error (Figure 3), Fi (Figure 9), and KI. divergence (Figure 10p show similar trend, confirming that the proposed intervention technique is useful and validating the causality from world representations model predictions.
G ABLATIONS ON INTERVENTION HYPER-PARAMETTE
Experiments find this optimization process is robust to different configurations of optimizer, learning rate a, and number of steps.
The complete world state, including states of all the 64 board tiles, is encoded in single internal representation æ, while during intervention experiment, we only wish to change one of them. A natural question is: will the intervention operation flip tiles but we can mitigate that by considering the cross entropy losses of other tiles as regularization term, weighted by hyper-parameter parameter That is to say, the loss main paper can be expanded as:
not intending? It is possible
(CE(Po(E), B')=
Σ B
B'(() +
Σ (ce(r(()),B(ss)
SEB=B'
We sweep B at the best L, = 5 on the natural benchmark and report average number of errors in Table 3) We can observe it does not clearly help.
Here we further discuss another syper-parameted ablated in Figure 3 the starting layer for intervention, Lag. If we intervene with more than 5 layers, shallow layers which have not developed reliable world
16

---

Published as a conference paper at ICLR 2023
/
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
error 0.22
0.31
0.77
0.19
0.51
0.65
0.42
0.45
0.46
0.55
Table 3: Average number of errors under different fiss and L, = 5 on the natural benchmark.
representations (according to Figure 6) are touched, making intervention hazardous. On the other hand, if we intervene only the deepest layers, though the world representation can be intervened successfully (see Appendix D), the model does not have enough computation to adapt to the newer world representation and make predictions corresponding it.
H STANDARD DEVIATION OF PROBE ACCURACY
In order to exclude the possibility that the differences in probe accuracies in Table and Table 2 are caused by randomness, we run the same experiment for 100 times with different random seeds and report the standard deviations of them Table and Table based which conclude that the probe accuracies are robust to randomness and there are significant differences between the probe accuracies of linear and nonlinear probes, and also between probes on randomized Othello-GPT and Othello-GPT's trained on different datasets.
x²
x²
x²
x.
x²
Randomized
0.04
0,04
0.05
0,06
0.06
0.06
0.09
0,06
Championship
0.05
0.04
0.06
0.07
0.06
0.06
0.06
0.06
Synthetic 0.06
0.06
0.07
0.06
0.05
0.04
0.06
0.08
Table 4: Standard deviations of error rates (%) of linear probes randomized Othello-GPT and
Othello-GPT' trained on different datasets across different layers (x* represents internal representa- tions after the i-th layer).
x²
x²
x
x:
2.5
x
x²
28
Randomized
0.05
0.06
0.07
0.04
0.04
0.03
0.04
0.03
Championship
0.05
0.09
0.08
0.06
0.05
0.05
0.10
0.06
Synthetic 0.07
0.09
0.04
0.06
0.06
0.09
0.05
0.05
Table 5: Standard deviations of error rates (%) of nonlinear probes randomized Othello-GPT and Othello-GPTs trained on different datasets across different layers (x* represents internal representa- tions after the i-th layer).
17


Reviews:
[{"review_id": "Z54tWg66yn", "reviewer": "Reviewer_1z26", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors study if a GPT-based language model, trained on a set of board game move data, is capable of learning a \"world model\" of the game. To investigate this:\n1. Given the representation generated by the LM, representing a game state, the authors try to use a probe (either a linear one or a non-linear one) to learn to predict the state of a tile (black/white/empty). They show that the non-linear probe can significantly outperform a baseline setting where board representations generated by random network is used. This suggests the game board state representation might be non-linear. \n2. Then the authors conduct an interventional experiment. In which, given a game board $B$ and the probe's prediction (of a tile) conditioned on $B$'s representation, they intervene the representation so that the probe provides a flipped prediction (e.g., white --> black). In such case, if the LM (intervened) output desired possible legal moves, it would suggest that the learned representations had a causal effect on the model. Experiment results show that the intervened LM can indeed produce the desired prediction, rather than ones made from the original game board state $B$.\n3. Extending the interventional experiment, the authors develop a visualization method, namely latent saliency maps. In which, for any tile on the board, the latent saliency map can show how much the network's prediction would change if an intervention had been applied to each of the other tiles. \n", "summary_of_the_review": "In general, I like this topic, and I rate the current version of the paper around (slightly above) borderline. I look forward to reading the authors responses to my questions so I can better understand the work and thus have a more precise evaluation.", "strength_and_weaknesses": "### Strength\n\n* **Nice task setting**: I like how the authors use board games as toy tasks for studying sequential data. Natural language, due to many confounding factors and properties, is not easy to be used for studies like in this work (e.g., flipping state of a tile). I'd love to see more work along this line, that can potentially help us to better model language, and the world that language describes.\n* **Towards understanding LMs**: At the same time, this work propose concrete methods towards making LMs more interpretable and transparent. \n* **Nice visualization method**: I like the latent saliency map method, which quite effectively represent dependencies among tiles. This can potentially be used in analyzing 1) word or concept dependencies in language generation; and 2) event or action dependencies in an offline RL setting (e.g., using decision transformer to model sequences of transitions, which essentially is an auto-regressive generation problem).\n* **Writing**: I enjoy reading this paper, it's clear in most part of it.  \n\n\n\n### Weaknesses\n\n* As the authors acknowledge, it remains unclear how methods proposed here can generalize to broader context, i.e., natural language, which is large language models designed for. I agree that this work is an \"pilot study\" towards that goal, but it would make the work much stronger if the authors can discuss more about, e.g., how board game trajectories connect with language, what are some common property, what are missing, and what are some potential paths towards that goal etc. As I mentioned among the strength, board games are good toy tasks studying sequential data, which gives researchers more control. It's just less clear in the sense of, how do we jump from this to language. \n* Please refer to the questions and concerns below. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XYek0YuwJ1", "reviewer": "Reviewer_uxFu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "EDIT: Increased original score from reject to borderline accept after reading the responses and other reviews. I have also increased my empirical novelty/significance score from a 2 to a 3.\n\nInspired by previous work on language models (LMs) learning chess, this paper trains a GPT model to predict legal moves in the board game Othello, which they call Othello-GPT. They then aim to go beyond the work on chess to understand the internal representations of the model, which they call the \"emergent world model\". To accomplish this, they design simple probes to understand representations of the game board in the model. They then present an intervention technique to alter the internal representation of the board, and show evidence that this alters the predictions of a probe trained on the model's world state representations. Finally, they show a visualization that they call a latent saliency map that uses the intervention to visualize the contributions of different board positions to the system's predictions.", "summary_of_the_review": "Overall, I do not think this paper is suitable for ICLR. The technical approaches themselves do not seem very novel, although my familiarity with this literature is not deep, and the Related Work discussion provided is quite brief. The main novelty of this work is in the application of these probing, intervention, and feature attribution techniques to the game Othello, which would likely only be of interest to a narrow audience. Finally, the issues with clarity and organization of the paper make this work appear rushed to submission.", "strength_and_weaknesses": "Strengths\n- Present new evidence that GPT variant can learn Othello board state, extending previous work on chess\n- Presents some evidence that their intervention technique to alter the \"emergent world state\" is effective\n- Using the intervention technique, they produce useful visualizations of \"saliency\" which describe how much a certain board position influences the prediction of Othello-GPT\n\nWeaknesses\n- There are several issues with clarity in the paper that made it difficult to understand on the first read.\n\t- I find the last paragraph of section 4 hard to understand, although I do think I grasped the method after reading the rest of the paper. It would help if the authors updated Figure 2 to show the actual metric/value that is used to determine if a representation is causal or not.\n\t- Similarly the description of the counterfactual case is not very clear. Please use notation to clarify the actual metric/value that is used to determine whether the internal representation is of the board \"rather than just a sequence\"\n\t- In the Related Work section, they authors refer to their work as having \"a focus on the geometry of internal representations\", but no geometry-related work is shown in the main results at all. This is only presented in the Appendix A, and this is not referred to anywhere in the main text.\n\t- In Tables 1 & 2, I believe \"randomized\" corresponds to probes trained on random weight GPT, rather than a random dataset. Table 1's caption makes it appear as if it is simply another dataset.\n- The authors do not present any measures of variance for the probe error rates in Tables 1 & 2 (e.g. standard deviation or confidence interval over random seeds). This makes it difficult to interpret the difference between the probes trained with the actual Othello-GPT model and the ones trained with the random weight model.\n- The authors also show that nonlinear probes perform better than linear probes, but do not address why this might be the case, despite the fact that they state this is a major contribution of the paper. What are we to take away from this result, other than \"the probe may be recovering a nontrivial representation of board state\"? The authors claim that the intervention experiments \"validat[e] this hypothesis\", but they do not discuss this hypothesis again. It appears as if this claim may be supported by Appendix A but they again do not refer to it at all.\n- Finally, the authors fail to make any additional argument as to why these results would be compelling to a wider audience beyond those interested in games / Othello. In the conclusion the authors write that \"the tools described in this paper—nonlinear probes, layerwise interventions, and latent saliency maps—may yet prove useful in natural language settings\". However it's unclear to me how these specific tools would contribute anything above and beyond current work in NLP using probes, interventions and feature attribution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L6NBRwP_7q", "reviewer": "Reviewer_VKpz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies whether LMs learn good internal representations that encode the actual process of generating the data, or simply memorizing surface features. \nThe paper investigates this problem in a game setting, i.e., Othello. \nIt presents a range of experiment results implying that, although the LM has no knowledge about the game rules, its learned representations encode information about the game board state. ", "summary_of_the_review": "This empirical paper carries out carefully designed and carefully executed experiments and have very interesting findings. \nIt has a lot of useful implications for future work. \n\nThis is the best submission that I have reviewed so far in 2022. ", "strength_and_weaknesses": "Strength \n\nThe experiments are well motivated and well designed. The claims are well supported and convincing. The findings are significant. \n\nThe paper is extremely well written and crystal clear. I personally find it enjoyable to read. \n\nWeakness \n\nThe training is a little strange to me. Why only predict y_T but not each y_t given its history? Just like how LM is pretrained. \n\nThere are some typos and grammatical errors. \nE.g., parenthesis not matched for \"(an example of which can be seen in Figure 2.\", \"as well as the change in next-step prediction in\". \nE.g., spacing is often weird, esp. before \"Figure\"s. \n\nMore visualizations may be helpful, e.g., when talking about the \"counterfactual case\". ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DXCp6WauL-Q", "reviewer": "Reviewer_LWko", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Using Othello game as the testbed for their research, the paper addresses a challenging question of whether transformers learn reasonable world-state representations, or simply exploit surface-level statistics of the data. First, the authors identify that the board state is recoverable from the network's representation, and then go an extra mile that this internal representation is causally connected with the network behavior.", "summary_of_the_review": "I thoroughly enjoyed reading this paper. I think that the contribution clearly defined the problem it is focused on, presents a well-defined plan of attack, and offers a crisp execution. The problem itself is highly important and sufficiently novel.\n\nWhile not perfect, I believe that in its present state, the contribution is above the threshold of acceptance, and can potentially become even better with relatively minor adjustments.\n\n** UPD, after the rebuttal period **\n\nI have read the authors' responses, which, I believe, address most of the concerns voiced by me and/or other reviewers (these concerns were predominantly minor in the first place). \n\nI initially thought very highly of the paper, now I am even more confident in my assessment. I think it's a great paper, and I deeply hope that it gets accepted!", "strength_and_weaknesses": "Strengths:\n- The paper addresses a highly relevant problem, and clearly states it.\n- The experiments are well-planned and well-executed, directly addressing the research problem posed.\n- The paper is clearly written, and is a pleasure to read.\n\nWeaknesses:\n- The choice of the metric used in the part 4.2 (EVIDENCE FOR A CAUSAL ROLE FOR THE REPRESENTATION) is not fully justified, and the calculation process is described too briefly, which may cause confusion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%