Learning Chess Blindfolded
Authors: Shubham Toshniwal
Sam Wiseman
Karen Livescu
Kevin Gimpel
Venue:  (2021)
Keywords: Chess
Transformers
Language Modeling
World State

Abstract: Transformer language models have made tremendous strides in natural language understanding. However, the complexity of natural language makes it challenging to ascertain how accurately these models are tracking the world state underlying the text. Motivated by this issue, we consider the task of language modeling for the game of chess.  Unlike natural language, chess notations describe a simple, constrained, and deterministic domain. Moreover, we observe that chess notation itself allows for directly probing the world state, without requiring any additional probing-related machinery.  Additionally, we have access to a vast number of chess games coupled with the exact state at every move, allowing us to measure the impact of various ways of including grounding during language model training. Overall, we find that with enough training data, transformer language models can learn to track pieces and predict legal moves when trained solely from move sequences. However, in adverse circumstances (small training sets or prediction following long move histories), providing access to board state information during training can yield consistent improvements.

Under review as a conference paper at ICLR 2021
LEARNING CHESS BLINDFOLDED
Anonymous authors Paper under double-blind review
ABSTRACT
Transformer language models have made tremendous strides in natural language understanding, However, the complexity of natural language makes it challenging ascertain how accurately these models are tracking the world state underlying the text. Motivated by this issue, we consider the task of language modeling for the game of chess. Unlike natural language, chess notations describe a simple, constrained, and deterministic domain. Moreover, we observe that chess notation itself allows for directly probing the world state, without requiring any additional probing- reelted machinery. Additionally, we have access to a vast number of chess games coupled with the exact state at every move, allowing us to measure the impact of various ways of including grounding during language model training. Overall, we find that with enough training data, transformer language models can learn to track pieces and predict legal moves when trained solely from move sequences. However, in adverse circumstances (small training sets or prediction following long move histories), providing access to board state information during training yield consistent improvements.
INTRODUCTION
Recently, transformer-base language models such as GPT-3 have stretched notions of what is possible with the simple self- self-supervised objective of language modeling, becoming fixture in state of the art language technologies (Vaswani et al. [217 Devlin et al. 2019: Brown et al.) (020). However, the black box nature of these models combined with the complexity of natural language makes challenging to measure how accurately these models represent the world state underlying the text.
Motivated by the above issues, we propose training transformer language models for the game of chess. Chess provides simple, constrained, and deterministic domain where the exact world state is known. Also, chess games can be transcribed exactly and unambiguously using chess notations (Section D In fact, the form of chess notations allows us probe language models for aspects of the board state using simple prompts (Section
Due to the simplicity and precision of chess, fine-grained level than merely comparing it to the ground truth. For example, even if the next move prediction doesn't match the ground truth move, we can still evaluate whether the move is legal given the board state, and if it is illegal, we can determine the reason why (Appendix D) Moreover, since the state can be exactly modeled, we can evaluate models using counterfactual queries as well. The proposed evaluation sets and metrics are described in Section 5.3
evaluate language model predictions more
side benefit of working with chess is that we have access to nearly unlimited data that is coupled with the exact board state at every turn. This board state is a form of grounding for the move sequence and allows us to compare training on move sequences alone to training with access to varying amounts of explicit state. Thus, modeling chess using language models may have implications for the debate surrounding the ability of language models to capture meaning if only exposed to text (Bender & Koller 2020). To test the impact of chess board grounding on learnability and data efficiency, we can train language models with varying degree of access to the board state (Section 4).
Finally, while chess represents controlled domain, it is by means trivial for a language model. To illustrate the challenges of language modeling for chess, consider the left board shown in Figure [b where white is next to move. In order to generate valid next move, the language model needs to (a) infer that is white's 's turn, (b) represent the locations of all pieces, both white and black, (c) select one of the white pieces which can be legally moved, and finally (d) make legal move with the selected

---

Under review as a conference paper at ICLR 2021
h
8
MM
A
4
4
1
6
g5
A
A
A
A
A
a2
d2
a1
h
(a) Square naming
(b) Board state hefore (left) and after (right) the bishop at f1 is moved to b5. UCT notation represents the move f1 f1b5.
Figure 1: Chess Notation
piece. Thus, language model has to learn to track the board state, learn generate moves according the rules of chess, and on top of that learn chess strategies to predict the actual move.
We find that when given enough training data, transformers can learn to both track piece locations and predict legal moves at high levels of accuracy. However, when testing predictive ability for long move histories or when only given small training sets or when the model has access to limited history (Appendix C.T. predictive ability suffers. These challenging settings can provide an interesting testbed for future development of language models, and morcover because of the probing properties, errors can be diagnosed in great detail. these more challenging settings, we show that providing parts of the board state (during training time only) can lead significant improvements in accuracy.
Our results also provide some key insights on transformer language models: (i) They are robust to various ways of incorporating explicit supervision about the board state when given enough training data. (ii) In particular, they are robust to changes in input distribution where additional tokens, related board state, added to input sequence only during training (Section 4.1) In contrast to LSTMs, transformers achieve this robustness even with smaller training sets (Appendix FF). (iii) The model performance strongly relies on access to the whole sequence history as the performance drops on limiting this access to a fixed-size window of previous tokens (Appendix C.1
To summarize, our contributions are to:
Propose chess as testhed for evaluating world state tracking capabilities of language models.
Show that by selecting (and tweaking) the appropriate chess notation, we can probe language model for aspects of the world state using simple prompts (Section 3
Propose suite of probing tasks evaluate language models for chess world state tracking
(Section (.3) These probing tasks go beyond simple exact match, and use a more fine-grained
evaluation, and allow for automated error analysis (Appendix [D)-
Show that given enough training data, transformer language models can learn track piece locations and predict legal moves at high levels of accuracy.
Evaluate the effect of grounding by training and evaluating spectrum of transformer language models with varying degrees of access the world state. We find that grounding helps in challenging settings of our proposed probing tasks.
Provide insights transformer language models such their robustness to incorporating the world state in various ways, their dependence on access to the whole history, etc.
BACKGROUND
Chess Preliminaries. Figure shows how squares are indicated in chess notations via combination of lettered columns and numbered rows. Chess notations use this square naming convention to denote the movement pieces. As our notation, we choose Universal Chess Interface (UCI) notation, which
combines the starting square and the destination square to represent a move The move in Figure 1b is
Source ttttps: ht////wwikee http:pppp.ppp.......pp..o....t.a
Forr more details see https::/en..wkkipeeee
veraaeerraa Saae____hhhh___ Interface
2

---

Under review as a conference paper at ICLR 2021
represented as f1b5 in UCI where fl indicates the starting square and 1o5 denotes the ending square. While another notation, SAN, is the standard choice for gameplay, we prefer UCI (see Appendix A for reasons for choosing UCI over SAN).
2.1 RELATED WORK
Simulated Worlds and Grounding. There have been several prior efforts in relating simulated worlds to natural language. The bAbl framework simulates a world modeled via templates to generate question-answering (QA) tasks (Weston et al. 2015a). The recent TextWorld framework facilitates generating, training, and evaluating interactive text-based games (Côté et al. 2018). bAbl and TextWorld are similar to our work in the sense that the true world state is, by construction, available. The key difference with bAbl is that the it provides explicit world-state supervision in the form of training data for QA. With TextWorld we differ in: (1) their objective (reward maximization vs. maximizing the probability of the next observation), (2) how they are ultimately evaluated (final reward vs. world state tracking), and (3) whether we can directly probe the model's knowledge of the entire state. The world models of Ha & Schmidhuber (2018) also maximize the probability of the next observation, but differ along the other two dimensions. Similarly the work by Hermann et al. (2017) and Hill al. (2017) on developing and using 3D world simulations for learning grounded language has only partial overlap with the objective function, and differ along the other two aspects.
Our work related to work on grounding in that we are interested in comparing model performance when it does not have access to grounding information to when it does (Bruni et al.) [2014] Kiros et al. 2014 Ororbia et al., 2019). However, unlike the work of [Ororbia et al. (2019), for instance, the goal is not to improve performance of language models using access to more of the world state, but to assess how much of this state has been recovered by the model from just learning the I.M task.
Cloze Tasks for Natural I. Language Models. There has been a plethora of prior work on developing
and using cloze tasks for evaluating natural language models (Hermann et al. [2015] Hill et al.) 2016).
These cloze tasks can range from testing general text understanding (Paperno et al. (0166) to targeting
particular aspects of natural language, such as commonsense/pragmaaiie (Mostafazadeh al. 2016 Ettinger 2020), narrative understanding (Mostafazadeh et al. (017). factual knowledge (Petroni et al. 2019), etc. Creating these tasks often requires human curation, and the evaluation is typically limited to exact match. In contrast, we propose cloze tasks/prompts targeting the world state, which can be precisely automated for chess, and can evaluated at fine-grained level.
Probing. One of the goals of this work is to probe the language model's board state tracking capahility. A typical solution used by prior work is to train probing model on top of pretrained
model (Ettinger et af.] 2016 Alain & Bengio 2017; 7, Adi et al., 2017] Tenney et al. 2019 Hewitt Liang 2019). This setup is time-consuming as it requires training probing models for all tasks. Moreover, the complexity of the probing model can also affect the conclusions (Pimentel et al. 2020 In our case, we show that by appropriate choice of notation, probing for board state can be accomplished via simple prompts (Section 3).
Deep Learning for Chess. Deep networks have been used in prior work to predict/mimic the next move given the true game state David et al. (2016); Oshri & Khandwala (2015). Using just self-play and the rules of chess, AlphaZero achieves superhuman performance starting from random play (Silver let al. 2018). The focus of these work is the quality of game play given the true board state while testbed for evaluating the tranformer LMs world state tracking capabilities. Recently there have been several work focusing on transformer language models for chess (Presser & Branwen 2020 Cheng. 2020 Noever al. 2020). These work similar to ours in the sense that input is limited to the move sequence and not the true board state, but the focus of these work is again the quality of game play rather than how well is the model aware of the underlying board state.
we just use chess
Automated cloze tasks without any human filtering can end up with instances which even humans can't 't answer (Hill al. 2016)
3

---

Under review as a conference paper at ICLR 2021
Table 1: Token sequences corresponding to the move sequence e2e4 e7e5 glf3 for different notations during training and inference. Notice that regardless of the RAP probability used during training. at inference time the token sequences have no piece types.
Notation
Training
Inference
UCI UCI + RAP 15 UCI + RAP 100
e2, e7, e5, gl, f3
e2, e7, e5, gl, f3 e2, e7, g1, f3 e2, e7, e5, gl, f3
e7, gl, f3 P. e4,
3 LANGUAGE MODEL PROMPTS AS BOARD STATE PROBES
Chess notations provide a simple alternate solution of using language model prompts as board state probes. For example, the prompt "e2e4 e 7e5 g f3 b8c6 d2d4 h7h6 f1" (the underlined move sequence leads to the left board state in Figure Tbp can be used for next-token prediction with language model trained on UCI notation. The generated token can be interpreted as the ending square predicted for the bishop at fl. This prediction can then be used to determine the level of board state awareness of the model. A prediction of gl may indicate that the model does not recognize that the piece type fl is a bishop, as such move is not possible for a bishop. If the model predicts g2, may mean that the model is not aware that another piece is currently located at g2.
UCI notation is sufficient for assessing ending square prediction. However, does not allow to test starting square prediction directly. That is, we would like to give language model the prompt "e2e4 e7e5 g f3 b8c6 d2d4 h7h6 N", where N represents knight, and expect to generate a valid starting position for knight of the correct color. To allow testing the model with such prompts, propose randomly including piece types in moves during training with some fixed probability p. We refer to this strategy as "randomly annotated piece type" (RAP) and use the nomenclature "UCI + RAP p" to indicate that with ps probability, piece type is part of the move notation during training. Note that for p 0, the notation reduces to UCL. When testing with these starting square prediction prompts, we only include piece type for the prompt, not for any moves in the history. Thus, using RAP during training allows to probe, at test time, where the model thinks each piece is, given any game history's 's prefix; by simply providing the desired piece type (c.g., N) the model outputs the predicted starting square for piece of that type.
LANGUAGE MODELS AND VARIANTS
We use the GPT2-small architecture for our base language model (Vaswani et al. 2017 Radford et al. 2019). GPll-small a 12-layer transformer model with 12 attention heads and an embedding size of 768 dimensions. The context size of the model is limited to 512, which is sufficient to cover the longest game in our training set. Note that we only borrow the model architecture; the models themselves are trained from scratch. We use simple regular expression based tokenizer, which considers board square symbol such as b1 as a single token. This gives us a vocabulary of tokens which includes the 64 squares, piece type symbols, and some other special symbols (see Table in the Appendix)
We next discuss variants of the base language model that incorporate the board state various ways.
4.1 RANDOMLY ANNOTATED PIECE TYPE (RAP)
One way of introducing an aspect of board state into move sequences is by randomly annotating piece types (RAP) in training sequences, introduced in Section 3 Table T iillsstrates how the use of RAP changes the token sequence during training but not during inference. We hypothesize that adding piece types during training can aid the model in learning track the board state by providing additional supervision. Similar ideas have been used when training memory networks for tasks involving entity tracking in natural language (Weston et al.) 2015b Hill al. 2016).
In initial experiments we used a delimiter token to indicate move boundary. However, removing it did not degrade performance, and made training much faster due reduced sequence length.
4

---

Under review as a conference paper at ICLR 2021
4.2 INCORPORATING BOARD STATE
One of the criticisms sometimes leveled against language models for natural language is that they can never understand the "meaning" of language, particularly given that they do not any extra-linguistic information, such as visual grounding (Bender & Koller] of chess means that chess offers nearly-unlimited data with perfect grounding. The ground truth board state can be used to learn a grounded language model, and can also be used in oracle settings which use this board state during both training and inference. We next describe details of the board state representation followed by its incorporation in the base language model.
The simple and deterministic dynamics
4.2.1 BOARD STATE REPRESENTATION
Following Oshri & Khandwala (213)), we represent a chess board state as an 8 x 8 image with six channels (see Figure 3 in the Appendix). The six channels correspond the six unique piece types in chess, namely, pawn, rook, bishop, queen, king, and knight. The channel for a given piece type has a value of +1 locations of its white instances, and 11 at locations of black instances, with 0 indicating empty positions. Apart from the piece positions, we add a seventh channel to represent additional information about the board state. In particular, we use one bit to indicate which player moves next, one bit to indicate if the current player's king is in check, and another four bits to represent castling rights of the two players These six bits are represented as another channel by padding with zeros to make it an 8 x 8 grid. Thus, we have 7 x 8 x 8 tensor for the raw board representation.
The raw board representation is fed to a 2-layer convolutional neural network (CNN). Both CNN layers have 128 filters of kernel size 2 x 2 with ReLU activations. The output of the CNN is passed through a linear layer which outputs a final 2o--immnnsionll board state representation.
4.2.2 VIEW TRAINING
We use the board state representation during training to learn grounded language model. Let (ht) represent the transformer language model's s hidden states at timestep t, where the hy are simply the embeddings for the input symbols plus the position embeddings. Then, the language model's 's state representation L; is learned convex combination of the transformer hidden states:
12
Σ a h 0m
αx softmax(wo, W w122
=
where w E 13 learned parameter vector.
Given minibatch of tokenized chess games, let K be the total number of moves in this minibatch. Let (L represent the language model state representations at the end of each move the minibatch, and let (st) represent the corresponding board state representations. We want the two representations to be close to each other when they correspond to the same state, and far apart otherwise. To promote this behavior, we use a "quadruplet" loss (Chen al., 2017):
L(, 8;) = max(0, m + d(L,, s,) - d(L., s()) + max(0, m + d(L;, 8:) d(l', s:))
where represents the margin by which we want the positive pairs to be closer than the negative pairs, d(...) represents the distance function (in our case, cosine distance), 1 is a language model state which corresponds to board state other than 8,, and s² is a board state which corresponds to a language model state other than 4, For sampling a negative state given the other view's 's state, we randomly draw 10 states from the minibatch and pick the closest the other view's 's state. For margin, we use m = 0.6 as it gets the lowest validation perplexity among (0.2, 0.4, 0.6, 0.8). Finally, the quadruplet loss is simply added to the language modeling loss without any scaling.
4.2.3 ORACLE BASELINE
Just for comparative purposes, for our final baseline we assume that the language model has access to both the piece type with which moves are made (e.g., Bflb5 instead of f1b5) and the board state
Details on castling rights: https: //www//wwwwww/////////////.ww.w/ wog.assprrrmmmmmmmm igg. wrgg/Cast win_giightha.oooo 'hhe board state used is incomplete. We skip details such en passont, threefold repetition, sipetition.fffyymmre rule counter, etc. they come into play only in rare cases.
5

---

Under review as a conference paper at ICLR 2021
Table 2: Examples of each probing task, as well as the corresponding exact move (ExM) and legal move (LgM) correct answers, are shown below. All examples assume the language model was fed the prefix e2e4 e7e5 gif3 b8c6 d2d4 h7h6 (see Figure [Tb), and that the actual next move was f1b5. While there is only one valid prompt token for both End-Acti Antull and Start-Actual tasks, there are many valid prompt tokens for the other tasks, and we show just one possibility for each. Start-tasks (bottom sub-table) assume the model was trained on games described in UCI+RAP notation.
Task
Prompt Token Correct Answers (ExM)
Correct Answers (LgM)
End-Actual fl
{b5}
{e2, d3, c4, b5 , a6 {d2, gl, h4, g5, e5}
End-Other
f3
N/A
Start-Actual
B
[f1}
[1l, c1} [E3, b1}
Start-Other
N/A
at the end of every move during both training and inference. Access to piecetype can limit the set of moves to consider, and the access to board state relieve the language model from the burden of board state tracking. To incorporate the board state, we simply sum the board state with the hidden state of all the layers of the transformer at the end of each move 7 This baseline serves an approximate upper bound to all other model variants described above.
5
EXPERIMENTAL SETUP
5.1 DATA
We use the Millionbase dataset which is freely available and has close to 2.9 million quality chess gamex 8
After filtering out duplicate games, games with fewer than 10 moves, and games with more than 150 moves (for the complete game to fit into one transformer window), we are left with around 2.5 million games. From this filtered set we randomly select 100K games for training, 15K games each for dev and test, and another 50K games to create prompt-based evaluation sets described in Section 5.3 The dev and test set are used for calculating perplexities on them. The dev set perplexity used for choosing the hyperparameters for models which incorporate the board state. From the 100K training set, we create subsets of size 15K and 30K which we refer to as "Train-S" and "Train-M", while the full training set referred to as "Train-L." For detailed statistics, see Table P in Appendix. All the data processing steps requiring chess knowledge, including parsing chess databases, are carried out using python-chess (Fickas 2012).
5.2 TRAINING DETAILS
Models are trained for 10 epochs with batch size of 60. Validation is performed at the end of every epoch and training is halted whenever the validation loss starts increasing. For optimization we use Adam (Kingma & Ba 2014] with learning rate of x 10-4 and L2 weight decay of 0.01. The learning rate is warmed up linearly over the first 10% of training followed by a linear decay. To accelerate training, we use mixed precision training (Micikevicius et al., 2018). All experiments are carried out using the PyTorch Lightning framework which is built on top of Py Torch (Falcon 2019 Paszke et al. 2019). We use the GPT-2 implementation from the transformers library (Wolf et al. 2019).
For the UCI- + RAP p models, we try E {0.01, 0.05, O. 15, 0.85, 1.00), and find p= = 0.15 get the best perplexity on the dev set. Larger values of p may lead to greater mismatch between training and inference, while small values like p 0.01 likely do not provide enough training signal.
0.05 and
5.3 BOARD STATE PROBING TASKS
In this section describe how we probe the language model's board state understanding. In each probing task we feed the model a prefix of a game followed by a single prompt token, and the model is
"Other strategies for fusing the two views Download link available at at/https: //tppel////eeellee
Ororbia et al.
2019
Ziegler et al. 2019 are left for future work.
rebs
13/ $2013. htmml

---

Under review as a conference paper at ICLR 2021
evaluated based on the highest probability next-token under the model given this context. We show an example of each probing task in T (which we further describe below), assuming the model has been fed the move sequence prefix e2e4 e755 glf3 b8c6 d2d4 h7h6, which is visualized as the left board in Figure Tb The actual next move played in the game is f1 f1b5, which takes the white bishop at square f1 to square b5, as shown in the right board of Figure
To create these evaluation sets, we use the 50K unique games reserved for this task (Section 5.1) We only consider prompts for non-pawn pieces since the dynamics of pawns are fairly limited. We ensure that the game prefixes selected are never seen in the training data. To see the effect of game prefix length (measured in number of moves) on these predictions, we create two classes of this task: (i) Short, where 5 < 1 I 50, and (ii) Long, where 51 < I & 100. For both classes we create evaluation sets with 1000 instances. Next we describe the various probing tasks in detail.
5.3.1 ENDING SQUARE TASKS
In this set of tasks, the model is given game prefix and prompted with the starting square of the next move (f1 in the example of Table The model's next-token prediction represents its prediction for the ending square of this move, which tests the model's ability to track the board state and follow the rules of chess, as well as strategic awareness 3) We consider two task variants:
1. End-Actual: Given a move sequence prefix, the model is prompted with the starting square of the actual piece moved next in the game.
2. End-Other: Given a move sequence prefix, the model is prompted with the starting square of any piece on the board that can be legally moved according to rules of chess.
We evaluate End-Actual predictions in terms of both exact move (ExM) accuracy (whether the model predicted the true ending square, b5 in running example) and legal move (LgM) accuracy (whether the model predicted a legal ending square for the piece starting at the square in the prompt). ExM accuracy evaluation is similar the typical evaluation of language models on natural language data, while LgM is less stringent and focuses on testing just the model's understanding of chess rules and the board state, Note that for End-Other, only LgM accuracy is available. See Table 2 for examples.
5.3.2 STARTING SQUARE TASKS
In this category of task, the model is again given game prefix, but prompted with just the piece rype of the next move, such as for bishop the example in Table The model's pext-token prediction thus represents its prediction for where the prompted piece type currently is the board. This task tests the model's s ability to track pieces [T] Note that only models which have seen piece types during training, i.c. "UCI+ RAP" models, actually be tested this task. Also, piece types are used in the game prefix (except for the oracle baseline). We again have two variants of this task:
1. Start-Actual: Given a move sequence prefix, the model is prompted with the piece type of the actual piece moved next in the game.
2. Start-Other: Given move sequence prefix, the model is prompted with the piece type of any piece on the board that can be legally moved according rules of chess.
We again evaluate Atart-Actual predictions both in terms of ExM accuracy (whether the model predicts the starting square of the piece actually moved next in the game), well as in terms of LgM accuracy (whether the model predicts the starting square of a piece with the correct piece type). For Start-Other, only LgM accuracy is available; see Table for examples.
6 RESULTS
To save space, perplexity results are presented in Table of Appendix. 3 results when predicting starting squares and Table ssows results when predicting ending squares. We will divide our discussion of the results into those that pertain board state tracking and those that pertain to modeling chess strategy.
"ttattegic capabilities of chess language model are strongly tied to the quality of training games.
In certain cases, this task also tests understanding of chess rules. For example, in Figure 1b only the rook h1 1 be moved.
7

---

Under review as a conference paper at ICLR 2021
Table 3: Accuracies (%) for predicting starting squares (" ("Start-Actual" and "Staat-Other" tasks)
Training Set
Model
Starting Square (Short)
Starting Square (Long)
Actual
Other
Actual
Other
ExM
LgM
LgM
ExM
LgM
LgM
UCI + RAP 5
71.3
88.8
83.2
58.6
69.5
65.4
Train-S
UCI+ + RAP 15
80.4
96.7
93.8
72.6
81.6
79.4
Oracle Baseline
86.3
99.1
97.2
89.3
99.0
98.5
UCI + RAP 5
78.8
95.1
92.4
70,9
80.9
78.4
Train-M
UCI RAP 15
84.5
98.7
97.1
83.0
93.3
92.3
Oracle Baseline
87.1
99.7
98.1
90.2
99.6
99.4
UCI RAP 5
86.8
99.7
97.9
88.6
98.0
98.3
Train-I.
UCI + RAP 15 87.4
99.7
98.2
89.6
99.3
98.8
Oracle Baseline
90.2
99.9
98.9
91.7
99.6
99.5
6.1 BOARD STATE TRACKING
There several observations to note. First, transformers can learn to identify where pieces are located. This is shown by the "I.gM" accuracies in Table 3 UCI + RAP 15 can predict a valid starting position of piece 99.7% accuracy for short histories and 99.3% accuracy for long histories. When asked to identify the location of a piece other than the one selected to be moved next, these accuracies drop only slightly, 98.2% and 98.8%, respectively. However, this capability requires the large training set. For the medium training set and long histories, accuracy drops to 93.3%, and with the small training set, it drops further 81.6%.
RAP 5 reaches the same starting position accuracy as RAP 15 for short histories, but for long histories there is a 1.3% gap. At smaller training sizes, the superiority of RAP 15 becomes much clearer, showing the benefit of providing piece type information more frequently during training.
The difference between the location of the piece the exact move (ExM) and the location of either piece of the given type (LgM) is substantial, at more than 10% absolute. However, this difference relates to chess strategy rather than board state tracking. When querying a piece type other than the one moved, the accuracies are still high for RAP 15, at 98.2% for short histories and 98.8% for longer histories, though slightly lower than when querying the actual piece type. This trend suggests that piece location tracking is slightly better for the piece type that is actually moved.
Second, transformers can learn to predict legal moves. This shown by the "LgM" accuracies in Table for which UCI RAP 15 exceeds 99% for short histories. We do observe drop to 95.1 % for long histories, which is likely because board state tracking becomes more difficult with longer histories. When decreasing training data, the short history LgM accuracy drops by 2% in each case, but the long history accuracy drops by 5%. It may be the case that training on an even larger training set can improve long-history accuracy further.
Overall, when enough training data is available, randomly adding piece types (with RAP 15) is similar to using the complete board state via multi-view training. With small training sets, the multi-view model struggles to use board states effectively, but with the large training set, the results are slightly better than RAP 15. Transformer language models are robust to various ways of incorporating explicit supervision about the board state when given enough training data.
We find consistent gains in accuracy when randomly adding piece types during training. UCI RAP 5 improves over UCI, which corresponds to RAP O, and UCI RAP 15 improves further in most settings, with the largest improvements for Train-S. With Train-I., UCI able to predict legal ending square in 98.6% of cases when using short histories. This strong result indicates that transformer without any form board state supervision can still learn to track board state well enough to predict ending squares if given enough data. A detailed error analysis of illegal moves for the Train-L setting is presented in Appendix D
Given the strong results with large training sets, future work focus challenging settings such as: (a) low data regime, (b) long(er) game histories, (c) limited access to previous tokens in self-attention (Appendix C.1 (d) predicting more board state variables simultaneously, etc.
8

---

Under review as a conference paper at ICLR 2021
Table 4: Accuracies (%) for predicting ending squares ("End-Actual" and "End-Other" tasks)
Training Set
Model
Ending Square (Short)
Ending Square (Long)
Actual
Other
Actual
Other
ExM
LgM
LgM
ExM
LgM
LgM
UCI
51.9
90.8
84.3
28.3
69.5
64.8
UCI + RAP
53.4
94.4
86.5
32.5
78.7
73.6
Train-S
UCI RAP 15
57.4
94.8
89.2
36.1
84.6
76.4
UCI+ + Multi-view Training
51.4
90.2
85.1
27.7
71.0
65.8
Oracle Baseline
59.0
95.7
89.9
39.4
89,4
84.3
UCI
56.8
95.1
90.7
35.2
82.8
76.9
UCI RAP 5
59.8
97.0
91.0
39.6
88.9
81.3
Train-M
UCI + RAP 15
61.1
96.9
90.5
43.0
90.2
83.7
UCI+ + Multi-view Training
56.8
95.9
91.7
35.7
83.3
78.6
Oracle Baseline
62.1
97.8
93.1
46.0
92.4
87.4
UCI
66.7
98.6
94.9
47.1
94.5
89.9
UCI RAP 5
69.0
98.9
94.8
50.3
95.6
91.7
Train-L
UCI + RAP 15
68.6
99.1
95.2
48.4
95.1
91.6
UCI + Multi-view Training
69.7
99.2
95.8
49.3
95.9
92.3
Oracle Baseline
69.1
99.1
95.9
53.8
97.4
94.0
Random Legal Move
26.2
21.8
6.2 CHESS STRATEGY MODELING
Overall, our results show that predicting the actual moves made by human players is much more difficult than identifying valid piece positions and legal moves.
We can view the exact move accuracy in starting square prediction (Table the ability to determine which piece to move next given the piece rype. The Oracle Baseline ExM results show the difficulty of this task even when the full board state is provided inference time. With the large training set, see gaps of 2-4% between the RAP models and the Oracle Baseline, while in terms of LgM, which does not involve strategy, the gaps are much smaller. In general, the gaps increase with less training data and longer histories.
When predicting ending squares (Table 4 the relatively low ExM accuracies of the Oracle Baseline show the difficulty of the task. The strongest models without board state during inference-RAA 5, RAP 15, and multi-view-are able to match or exceed the Oracle Baseline when having access the large training set and for short histories. This result shows that our models are not limited by their ability track the board state over short histories, but rather by the inherent difficulty in predicting the next move performed by human player. However, for long histories, the Oracle Baseline is consistently more accurate than our models.
7 CONCLUSION
We propose the task of language modeling for chess to evaluate how well language models can capture the world state. We show that with appropriate choice of chess notation, a language model can be probed for different aspects of the board state via simple prompts. The simple and precise dynamics of chess allow for (a) training models with varying amount of explicit state, and (b) evaluating model predictions fine-grained level. Results indicate that transformer language models able to track the board state when given enough data, but in adverse circumstances, providing access to board state information during training can yield consistent improvement.
Wider Implications for Natural Language Processing. Our results shed light on the following interesting properties of transformers: (a) they are robust to RAP-like changes in input distribution, and (b) they require access long context (Appendix C.TI. and large training sets. Future work can use the first finding to introduce the world state, or more specifically the output of linguistic analyzers such as coreference, via RAP-like tokens during pre-training and fine-tuning of transformers. RAP-like tokens can also be used for booougging/diannsinnn the model's understanding, similar to the starting square prediction tasks. The second finding can be another motivation to search for new architectures that adept at understanding long text with access limited history (Rae et al. 22020), and that require small training sets. The proposed framework allows for probing and understanding new architectures that address these challenges.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In ICLR, 2017.
G. Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In ICLR Workshop, 2017.
Emily M. Bender and Alexander Koller. Climbing towards NLU: On Meaning, Form, and Understand- ing in the Age of Data. In ACL, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, 2020.
Elia Bruni, Nam Khanh Tran, and Marco Baroni. Multimodal Distributional Semantics. JAIR, 49(1), 2014.
Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond Triplet Loss: A Deep Quadruplet Network for Person Re-identification In CVPR, 2017.
Ricson Cheng. Transformers Play Chess, 2020. URL htpp:://gttub..cm//ccccom
transformers playcchsssaay
Mare-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. TextWorld: A Learning Environment for Text-based Games. CoRR, abs/1806 1532, 2018.
1532,
Eli David, Nathan S. Netanyahu, and Lior Wolf. DeepChess: End-to-End Deep Neural Network for Automatic Learning in Chess. In International Conference on Artificial Neural Networks (ICANN), 2016.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, 2019.
Allyson Ettinger. What BERT is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models. TACL, 8(0), 2020.
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composition by means of simple classification tasks. In 1st Workshop on Evaluating Vector-Space Representations for NLP, 2016.
WA Falcon. PyTorch Lightning, 2019. URL htl.ttppp//tttuuuuccm///yycceeet pytorch--ightning
Niklas Fickas. python-chess: a pure Python chess library, 2012. URL htttps://pytoon-oottt readthedocs www.e///neeessse winex.iiix. hrml
David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In NeurIPS, 2018.
Karl Moritz Hermann, Tomás Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching Machines to Read and Comprehend. In NeurlPS, 2015.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded Language Learning in Simulated 3D) World. CoRR, Abs/1700.00551 2017.
John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In EMNLP- IJCNLP, 2019.
10

---

Under review as a conference paper at ICLR 2021
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. The Goldilocks Principle: Reading Children's 's Books with Explicit Memory Representations. In ICLR, 2016.
Felix Hill, Karl Moritz Hermann, Phil Blunsom, and Stephen Clark. Understanding Grounded Language Learning Agents. CoRR, abs/1710.098677, 2017. /tt/tttpp:/arxxi..rr///bb
1710.09800
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9, 1997.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. CoRR, abs/1411.2539, 2014.
Tal Linzen, Emmanuel Dupoux, and Y. Goldberg. Assessing the Ability of LSTMs Learn Syntax- Sensitive Dependencies. TACL, 4, 2016.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In ICLR, 2018.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In NAACL, 2016.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. LSDSem 2017 Shared Task: The Story Cloze Test. In 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, 2017.
David Noever, Matt Ciolino, and Josh Kalin. The chess transformer: Mastering play using generative language models, 2020.
Hiroshi Noji and Hiroya Takamura. An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models. In ACI, 2020.
Alexander Ororbia, Ankur Mali, Matthew Kelly, and David Reitter. Like Baby: Visually Situated Neural Language Acquisition. In ACL, 2019.
Barak Oshri and Nishith Khandwala. Predicting Moves in Chess using Convolutional Neural Networks. 2015.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In ACL, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Ed- ward Yang, Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py Porch:: An Imperative Style, High- Perf-reerfomancc Deep Learning Library. In NeurlPS, 2019.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In EMNLP--JCNIPP, 2019.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information- Theoretic Probing for Linguistic Structure. In ACL, 2020.
Shawn Presser and Gwemn Branwen. A Very Unlikely Chess Game, 2020. URL https:
7/s 1 ateetattttttm www.ex.ccccxccoooo
coteddxx.omm/000 com http://200011////aaerrrroo
-hh.r--hsseeeeeceeeeeeeeee- game/
11

---

Under review as a conference paper at ICLR 2021
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive Transformers for Long-Range Sequence Modelling. In ICLR, 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 2018.
lan Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? Probing for sentence structure in contextualized word representations. In ICLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In NeurlPS, 2017.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AT-Complete Question Answering: Set of Prerequisite Toy Tasks, 2015a.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory Networks. In ICLR, 2015b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-arr natural language processing. ArXiv, abs/1910.03771,, 2019.
Zachary M. Ziegler, Luke Melas Kyriazz, Sebastian Gehrmann, and Alexander Rush. Encoder- Agnostic Adaptation for Conditional Language Generation. CoRR, abs//998.06999 2019.
12

---

Under review as a conference paper at ICLR 2021
A SAN NOTATION
Standard Algebraic Notation (SAN) combines the piece type moved and the destination square to denote move For example, the move in Figure 1b represented as Bb5 SAN where represents the piece type bishop and b5 represents the destination square.
Standard Algebraic Notation (SAN) Ambiguity SAN notation doesn't use the starting square of the piece in its move representation. This limits the ability to prompt a SAN-based language model with specific piece type instances. For example, given the prompt "e4 e5 Nf3 Nc6 d4 h6 B" (the underlined move sequence leads to the left board state in Figure lb). it's not clear whether the token B refers the bishop at fl or cl. Due to this limitation on the specificity of probing queries, we do not use SAN for our experiments.
B MODEL VOCABULARY
Table 5: Model Vocabulary
Type
Examples
Count
Square names
e4,
64
Piece type
P. K. B,
6
Promoted Pawn Piece type
r. BOS, EOS, PAD
Special symbols
3
Total
77
Table 5 shows the vocabulary used in our experiments. We don't use any delimiter token to denote the move boundary. Tokens of promoted pawn piece type used when pawn gets promoted. For example, e 7 denotes the move where a pawn from c7 moves to e8 and becomes queen.
C OTHER MODEL CONFIGURATIONS
In this section, explore the effect of varying the transformer architecture along two dimensions: (a) limiting the attention window, and (b) increasing the model size. These experiments are designed to help ascertain what challenges remain tracking the state of chess game with a language model.
C.I LIMITED HISTORY
Chess is Markovian, and the exact board state can be represented in less than 1000 bits. Thus, the high-dimenssonal multi-layered transformer state representation can in theory represent the board state in the hidden state of even a single timestep. However, the transformer LM has no chess domain knowledge, and thus will likely learn very different state representation. In this ablation, inspect how much is any particular time step's 's representation important to representing the game's state. Specifically, is important for the model to attend to all the previous tokens or the model can learn compressed state representation if forced to do so. To this end, we consider the case where the model can only attend the current token and the previous w tokens. Note that the baseline model has access to the entire history.
want to
Table 6) presents results comparing the baseline model with models which have access only the past 10 or 50 hidden states. In comparison to the baseline, UCI (w = 10) suffers significant drop almost all evaluations except for accuracy over short histories with the Train-L training set. UCI (w = 50) fares better in comparison to UCI (w = 10) and matches the baseline on LgM accuracies for short histories given enough training data. However, the same not true for evaluations over long histories, where the gap betwen UCI and UCI (w 50) seems to widen with increase in training
For more details see httpr//c....at..o.aoaaooaaaaaata (ttp_otttttiooocccccsa_a__ (chess)
13

---

Under review as a conference paper at ICLR 2021
Table Accuracies (%) for predicting ending squares ("End-Actual" and "End-Other" tasks) with
varying attention window sizes.
Training Set Model
Ending Square (Short)
Ending Square (Long)
Actual
Other
Actual
Other
ExM
LgM
LgM
ExM
LgM
LgM
UCI
51.9
90.8
84.3
28.3
69.5
64.8
Train-S
UCI (w = 50)
49.0
88.5
81.7
25.7
68.1
61.7
UCI (w 10)
43.5
79.7
69.9
22.4
62.9
50.6
UCI
56.8
95.1
90.7
35.2
82.8
76.9
Train-M
UCI (w 50)
53.6
94.6
88.6
32.9
79.7
74.4
UCI (w 10)
50.9
86.1
77.3
27.7
68.9
57.6
UCT
66.7
98.6
94.9
47.1
94.5
89.9
Train-L
UCI (w 50)
64.4
98.7
94.4
42.1
89.9
85.7
UCI (w = 10)
60.0
95.5
87.9
36.5
81.0
73.1
data. These results clearly demonstrate the dependence of the transformer LM's performance access to unrestricted history, which is qualitatively different from an ideal state representation for chess.
C.2 LARGER MODELS
Table Accuracies (%) for predicting ending squares for different model sizes. GPT2-small {12 layers, 12 heads, 768 embedding size}; GPT2-intermeddin {16 layers, 12 heads, 768 embedding size}; and GPT2-medium (24 layers, 16 heads, 1024 embedding size).
Training Set
Model
Ending Square (Short)
Ending Square (Long)
Actual
Other
Actual
Other
ExM
LgM
LgM
ExM
LgM
LgM
GPT2-small
51.9
90.8
84.3
28.3
69.5
64.8
T
e
50.8
90.0
85.3
26.4
68.2
65.3
GPT2-medium
50.6
88.6
83.9
24.1
65.8
60.7
GPT2-small
56.8
95.1
90.7
35.2
82.8
76.9
Train-M
GPT2-intermediate
60.2
96.1
90.6
35.1
82.5
77.3
GPT2-medium
57.7
94.6
89.4
34.3
79.7
73.5
GPT2-small
66.7
98.6
94.9
47.1
94.5
89.9
Train-L
GPe--intermediate
67.5
98.8
95.2
48.3
95.8
91.7
GPT2-medium 67.0
98.6
94.7
49.0
95.6
91.0
Table presents results with transformer models of sizes varying from GPT2-small GPT2-medium. We also introduce new configuration, referred to as GPT2-intermediiat, which serves as an interme- diate between GPT2-small and on all evaluations. and outperforms GPP22-intermedie category. With increase in the training set size, GPT2-intermediate and GPT2-medium are able to match/outperform GPT2-small on most evaluations. In particular for Train-L,, GPT2-intermediite outperforms GPT2-small small on all evaluations.
For Train-S, GPT2- small outperforms GPT2-medium
on all but LgM accuracies for the "Other"
These results are along the expected lines of larger training sets alleviating the overfitting problem with larger models (Kaplan et al. 2020). Based on prior work and extrapolating from the current results, expect GPT2-medium to outperform GPT2-intermediate for training sets sufficiently bigger than Train-L. Note that we stick with the default GPT2 configuration for all our experiments. Tuning the regularization hyperparameters such dropout, improve results for bigger models.
14

---

Under review as a conference paper at ICLR 2021
D ILLEGAL MOVE ANALYSIS
A
2
A
(a) Syntax: Predicted move not possible for queen.
(b) Path Obstruction: The pawn at c6 is blocking the bishop.
(c) Pseudo Legal: The black king remains in check.
Figure 2: Visualization of the three categories of illegal moves.
Table 8: Error Analysis of Illegal Moves for models trained on Train-L.
Model
Ending Square (Long)
Actual Path Obst.
Other Path Obst.
Syntax
Pseudo Leg.
Syntax
Pseudo Leg.
UCI
3
21
31
34
59
UCI RAP 5
18
25
26
51
UCI+ RAP 15
2
23
24
31
46
UCI + Multi-view Training
0
14
27
17
55
Oracle Baseline
0
15
11
31
27
Illegal moves can be exhaustively categorized into three types:
The predicted move can't be made by the piece type present the starting square regardless of the board state. This error indicates that the model either fails at tracking the piece type present at the starting square or lacks the spatial understanding of the board or both. These errors rare; figure 2a presents
Syntax:
such case. In this particular case, many of the previous moves involved the b6 square, which might have led to this error.
The move can't 't be executed because there are other pieces present in the path. This eπror indicates the model's failure tracking the board state. In figure 2b the pawn at с 6 blocks the bishop's move from e4 to b7.
Path Obstruction:
Pseudo Legal: This fairly tricky category where the move illegal because the king ends up in check remains in check if the predicted move is executed. We hypothesize that training explicitly on negative examples might be helpful/required in preventing these
errors (Linzen et al. 2016 Noji & Takamura 2020).
Table [ presents results on analysis of illegal moves predicted by the model trained on the full training TrainL for the ending square task for long histories. We observe that of the gains for the oracle baseline are coming from the pseudo legal error category. The oracle baseline benefits from the input representation explicitly encoding the information that the king is in check. While the models are pretty closely matched on the syntax error category, in the path obstruction category the clear winner is the multi-vice view training setting, which suggests that the multi view approach is better at modeling the spatial arrangement of the pieces.
15

---

Under review as a conference paper at ICLR 2021
E
MISCELLANY
channels
A
A
MIA
Figure Part of the raw board representation used input to the board state model.
Table Statistics of language modeling data
Split # of games (in 10*)
Total # of moves (in 10%)
Avg. # of moves per game
Train-S
15
1.1
73.6
Train-M
30
2.2
73.5
Train-L
100
7.3
73.5
Dev
15
1.2
78.5
Test
15
1.2
79.2
Table 10: Canonical Dev and Test perplexity. By canonical mean that one move, say f1b5, counts as one token.
Training Set
Model
Dev set
Test set
UCI
3.21
3.20
UCI + RAP
3.07
3.07
Train-S
UCI + RAP 15
3.07
3.08
UCI + Multi-view Training
3.22
3.22
Oracle Baseline
2.71
2.71
UCI
2.83
2.83
UCI + RAP
2.76
2.75
Train-M
UCI RAP 15
2.79
2.79
UCI + Multi-view Training
2,80
2,80
Oracle Baseline
2.46
2.46
UCI
2.24
2.24
UCI + RAP 5
2.76
2.75
Train-L
UCI + RAP
2.44
2,44
UCI + Multi-view Training
2.29
2.29
Oracle Baseline
2.13
2.13
16

---

Under review as a conference paper at ICLR 2021
F
RESULTS FOR LSTM LANGUAGE MODELS
Table 11: Accuracies (%) for predicting ending squares for LSTM language models
Training Set
Model
Ending Square (Short)
Ending Square (Long)
Actual
Other
Actual
Other
LSTM-LM
ExM
LgM
LgM
ExM
LgM
LgM
UCI
43.1
77.7
68.1
23.8
60.0
52.5
Train-S
UCI RAP 5
42.0
77.1
67.1
23.5
59.8
53.2
UCI RAP 15
42.7
77.0
69.0
22.9
59.3
52.2
UCI
48.5
82.7
76.9
29.7
65.0
58.7
Train-M
UCI RAP
50.1
83.8
77.3
27.5
64.0
59.0
UCI RAP 15
51.1
84.0
77.3
28.8
66.2
60.8
UCI
59.9
94.2
88.5
36.1
81.6
74.8
Train-L
UCI RAP 5
59.4
93.5
87.7
38.1
83.3
75.7
UCI+ RAP 15
61.5
94.3
87.7
39.5
84.7
78.4
In this section we present results for LSTM language models (Hochreiter & Schmidhuber 1997). In particular
LSTM model with layers, 1024-dimensional hidden units, 768-dimensional input embedding size, and dropout of 0.1 applied to the output of the embedding layer and intermediate LSTM layers| | The motivation behind these experiments is to compare LSTMs and transformers in their ability to utilize the board state signal available via RAP during training. We focus on the RAP setting because with RAP there's 's distribution shift between training and inference, and want compare how the two model classes handle this shift.
Table presents results for LSTM language models, the corresponding numbers for transformers presented in Table D in the main text. For Train-S, the vanilla UCI model outperforms both the RAP settings on all but a couple of Other LgM evaluations. For Train-M, the addition of RAP improves performance in all evaluations, except ExM accuracies for long histories. And finally for Lr both UCI + RAP 15 and UCI RAP outperform UCI over long histories. For short histories, UCI outperforms or matches both UCI RAP 5 and UCI + RAP 15 on LgM evaluations, and only UCI + RAP 15 has an edge over UCI ExM accuracy.
These results are in sharp contrast the significant performance gains reported earlier for transformers with the addition of RAP, especially for smaller training sets. Thus transformers are better at utilizing RAP than LSTMs, and that transformers distribution. Our reasoning is that unlike LSTMs/RNNs, transformers have only a weak dependence on positions via position embeddings, and thus better handle RAP-like changes where tokens can be randomly inserted/deleted throughout the length of the input sequence.
more robust than LSTMs to RAP-like changes in input
Hyypepprrameterss were chosen via grid search with dev perplexity the evaluation metric
17


Reviews:
[{"review_id": "n_C9O0bQdV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers an intriguing problem: can language models, when trained on\npurely textual representations of Chess games, learn the underlying dynamics of\nthe game? The authors argue that this could be a preliminary step towards\ntackling the symbol grounding critique of methods like transformers. When\ntransformers are utilized in natural language settings, it's challenging to\ndetermine whether the models are operating at a pure syntactic level, or whether\nthere is some rudimentary level of \"understanding\", given that the models are\nonly exposed to text. In contrast, in Chess, one can train in a purely textual\nfashion (or with some limited symbol grounding information), and probe how well\nthe result models the state of the underlying Chess position.\n\nThe authors use the GPT2-small based transformer architecture and train from scratch on a dataset of\nhigh-quality Chess games between humans, represented in UCI notation -- a\ntextual listing of the moves made by each player. In some experiments, this\npurely textual input is supplemented with explicit board state information as\nwell, to test the impact of this additional signal. The authors evaluate the\nsystem on two types of inference tasks: a trained model's ability to\nsuccessfully locate a piece on the board, and the model's ability to determine\nwhere to move a chosen piece to next. In each case, there are two\nevaluation metrics: an \"exactness\" metric (i.e., whether the model picked the\nsame piece/move as the human did in the corresponding game) and a \"legality\"\nmetric (i.e., whether the model picked a permissible move). The former metric is\nmore stringent, as it's also measuring the strategic awareness of the model. The\nauthors demonstrate through their experiments that transformers are successful\nat both inference tasks with very high accuracy, particularly when evaluated\nusing the legality metric, using shorter sequences of moves, and larger\ndatasets.\n\nStrengths of the paper:\n  + This is a creative application of transformers to a non-traditional textual\n    inference task. It may inspire others to devise other interesting applications.\n    The results are intriguing and expand our understanding of what may be\n    possible with transformer architectures.\n  + The authors' approach is a fairly straightforward application of off-the-\n    shelf techniques -- and I mean that in a good way. There are no unnecessary\n    complications or ad hoc additions to the system design.\n  + The paper is very clearly written, well-organized, and easy to follow.\n\nAreas for improvement/questions for the authors:\n  - I'm a little confused as to why performance peaks with p=0.05/0.15 in the\n    UCI+RAP models. If my understanding is correct, then RAP with\n    p=1.0 would include piece annotations with *every* move during training\n    time and at inference. So shouldn't this make it easier to track pieces (as\n    the authors themselves note in 4.1)? The reason for this is given as greater\n    \"mismatch between training and inference\" -- I'm not sure what this means.\n  - On a related note, I'm also confused by the performance of the Oracle Baseline\n    in Table 3. In some instances, this is outperformed by trained models with\n    *less* information. But my understanding was that this oracle would serve\n    as an upper bound on model performance. So how are we outperforming the\n    upper bound?\n\nOn balance, I think the strengths of the paper outweigh my concerns, and I\nrecommend ACCEPTANCE.\n\nCouple of minor issues:\n  - At the bottom of Page 6, the text references results in Table 3, which\n    should be Table 2.\n  - Section C in the Appendix is currently empty.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q1nqxtbXL8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper is an interesting exploratory study analyzing the ability of language models to track the state of a chessboard. The authors adopt a clever chess notation which allows them to probe the language model's state tracking ability by looking at its next word prediction (akin to probes in [1]). Quite remarkably, language models finetuned on chess data store a very accurate state representation, and predict legal moves over 90% of the times even without a visual representation of the board.\n\n-----------------------------------\n\nStrengths of the Paper:\n\n1. A clever probing method to analyze language model's state tracking abilities. \n\n2. Well-designed experiments and very interesting results in a mostly unstudied area.\n\n3. Wel written paper with several baselines and ablations.\n\n-----------------------------------\n\nWeaknesses of the Paper / Possible additional analysis:\n\nWhile the work in itself is very interesting and clean, I would have loved to see more analysis studying the model. This a very rich testbed where a lot of interesting experiments can be done! For instance,\n\n(1) Does model performance / chess quality improve with larger language models like GPT2-md or GPT2-l?  \n(2) What are the kinds of errors these language models make (in terms of legal moves)? Do these errors disappear when you check top-k tokens?  \n(3) What are the kinds of moves the model is good at (when considering argmax predictions)? Is it learning any strategy at all? You can measure this quite well automatically using the chess engine scores which indicate who is winning, and comparing the change in scores when the actual move is played vs the language model's predicted move  \n(4) Finally can insights from (2) and (3) be transferred to other real-world applications? Is there a correlation between the probing literature on natural language processing tasks and the results you find?  \n(5) It will be cool to check other kinds of visual state fusion strategies like pseudo-self attention [2]\n\n-----------------------------------\n\nOther Feedback:\n\n1. on the bottom of page 6, did you mean Table 2?\n2. A couple of baselines will be useful in Table 2 and 3. The first could be upperbound EM baselines using engines like Stockfish or AlphaGo (since nearly everyone is worse at Chess than them, I expect the EM score to be lower than 100%). The second could be a random lowerbound to EM, where a random move from the set of LM is chosen. Finally it will be good to see EM performance of GPT-2 considering only the set of LM.\n3. LM is an overloaded acronym which can cause confusion to the reader (language model vs legal move).\n\n-----------------------------------\n\nOverall Recommendation:\n\nThis is an exciting and rich testbed with a lot of interesting questions to answer. The authors have conducted well-thought experiments and reported interesting results. I'm leaning accept, but I encourage the authors to keep working on this setup (perhaps using some of the suggestions discussed above) and try to check if any of the insights here can be transferred to better understanding of language models on natural language.\n\n-----------------------------------\n\nReferences:\n\n[1] - https://www.mitpressjournals.org/doi/pdfplus/10.1162/tacl_a_00115  \n[2] - https://arxiv.org/pdf/1908.06938.pdf\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LQmJrTEqC5a", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper explores the abilities of transformer-based models to do grounded state tracking via chess. They train a GPT-2 on chess games, showing that it can learn the rules of the game by being able to state-track and predict valid next actions.\n\nPros:\n1. A very thorough set of experiments are given to explore how transformers can be used to track states in a game such as chess and results show that with enough data, transformers can predict the locations of pieces across a decent amount of history as well as predict legal actions.\n2. The paper is well-written in terms of general writing clarity and I was able to follow *what* was happening throughout.\n\nCons:\n1. I am a bit lost as to the motivation and positioning behind the paper, i.e. I was unsure as to *why* things were happening as they were. The authors say that they are using transformers to see how transformers can learn grounded language when world states are available but I do not see this work positioned with respect to other work on grounded language nor work on state tracking generally found in model-based RL.\n(i) An example of the former for grounded language learning (given that they cite Bender and Koeller) would be instances such as vision and language navigation (Anderson et al. https://openaccess.thecvf.com/content_cvpr_2018/papers/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.pdf), the Nethack Learning Environment for game grounding (Kuttler et al. https://arxiv.org/abs/2006.13760), or text games (Cote et al. https://arxiv.org/abs/1806.11532 and Hausknecht et al. https://arxiv.org/abs/1909.05398). I am not sure how state tracking in chess implies that transformers can do grounded language learning.\n(i) In terms of just the state tracking parts, there has already been much work in agents that learn the rules of the world as they play through them. This can happen with World Models (Ha and Schmidthuber https://arxiv.org/abs/1803.10122) or in cases like Alpha-Zero (Silver et al. open access version https://kstatic.googleusercontent.com/files/2f51b2a749a284c2e2dfa13911da965f4855092a179469aedd15fbe4efe8f8cbf9c515ef83ac03a6515fa990e6f85fd827dcd477845e806f23a17845072dc7bd) which learn the rules of Chess from scratch via self play. How does using a transformer compare to these works?\n2. Given the above and the fact that all of the pieces of the methodology of this work are taken from others (the architecture, training, state representation, etc.) - the main contribution is the experimental design and the results themselves. In this case, I would have liked to see more analysis regarding exactly what properties of the transformer they think is responsible for helping the model to learn and also a potential qualitative analysis of what the failure cases are.\n\nOverall, the paper has some interesting ideas, experiments, and results but is not connected to the motivation/is not positioned well with respect to closely related work.\n\nPost author response:\nSee comment below for further score justification.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-EaJH0Qn9dR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\t\n### Topic\nThis paper explores learning chess from raw notation as a benchmark for the ability of language models to track world state. Chess is an interesting benchmark, as a set of moves can be unambiguously linked to a world state, there are large amounts of data available and the model can easily be probed for its board tracking abilities. The contributions of this paper are twofold: (i) introducing blindfolded chess as a benchmark for grounded language learning and world state tracking, as well as a suite of probing tasks to evaluate models; (ii) empirical evidence that transformer language models can learn both the rules of the game and to track board state.\n\n### Pros:\n-\tBlindfolded chess is an interesting benchmark for grounded language learning and as a testbed for models to track world state. It is unambiguous, data-rich, has a limited vocabulary and models trained on it can easily be probed. This adds to prior papers on learning chess with transformers that have mainly focused on the performance of such models.\n-\tThe use of SAN + RAP trick is an interesting one to be able to probe the current location of pieces.\n-\tThe oracle model is interesting, as it demonstrates the gap between a model that must track world state and one that has access to it. The multi-view model is also interesting as an example of how additional supervision may help the model.\n-\tThe analysis across different data sizes and game length is interesting.\n\n## Cons:\n-\tThe paper feels rushed at times: for instance, there are no references in text to Appendix D, despite it being an interesting demonstration of the analysis that can be done in this environment. \n-\tThe comparison to related work on world state tracking and grounded language learning could be substantially improved. The authors only very briefly mention previous work: the papers on the TextWorld environment (Cote et al, 2018), seem relevant, although the framework is that of interactive environments and RL. Other papers on grounded language learning also seem relevant, such as Alexander G. Ororbia II et al, 2019. More so than the lack of references, an issue with this paper is that few connections are made to wider research issues in grounded language learning / implicit world state tracking. Instead, the text tends to simply state experimental results. This makes the contribution of this paper very narrow.\n-\tIt is not clear why the multi-view trained models underperform on the low data settings compared to the models without the extra supervision. If there is more explicit supervision of the model, should it not be better at tracking world state?\n-\tIt would be good to make clear if this dataset and probing tasks will be released, and the code made available. This would be a welcome step in standardizing this new domain, as prior approaches have all used different settings/datasets/notations/evaluations.\n-\tDespite this being a GPT-small architecture and the authors using a small subset of the available training data, the accuracies are high for legal move predictions on Train L. Those are the ones that are directly related to tracking the world state and knowing which piece movements are allowed. This limits the use of this task as a future benchmark for world state tracking in the data-rich setting. It might be possible to make the task more challenging by asking the language model to predict the entire board state (deconvolution) or by focusing on hard subsets of the task (e.g: pseudo-legal infractions on long histories).\n-\tMany challenges of natural language are not present in this environment, such as coreference, limiting the applicability of results on this task to e.g bAbl. The use of a composite vocabulary (e.g: “e4” instead of “e”, “4”) also limits the compositional challenges of the dataset. However, it is true that using a vocabulary where “e4” is two tokens would make probing more difficult.\n\nMinor comments:\n-\tIncomplete board state: The board state as represented is actually not complete, as it ignores whether a pawn can be taken en passant ( https://en.wikipedia.org/wiki/En_passant ). En passant is rare enough that this might not matter, but it does mean the Oracle/Multi view models do not perfectly capture the state. Note that this seems to also be an issue in Oshri and Khandwala (2015) from whom this representation is derived.\n-\tThe ending square task does not allow us to probe whether the model captures the full range of possible moves for a piece, or whether it selects a subset of those. E.g: the model might reach 100% accuracy on this task without ever moving a rock/bishop/queen by more than one square (less likely that there are pieces in between).\n-\tIt is not clear what you do with games of length 100-150. These seem included in the training set but excluded from the probes. Are those included in the perplexity results of Table 6.\n-\tTypos:\no\tTable 4 breakdown in appendix B\n\n### Recommendation:\n\nI lean reject for this paper.\nThe core idea is interesting, but the paper fails to make connections to wider issues in world state tracking and grounded language learning, making it overly narrow. Both the missing references and the missing links to wider concepts in this litterature are a symptom of that. With a few caveats, the experiments are sound, but the analysis could be improved to go further than simply stating the results. The overall wording and presentation of the paper must also be improved.\n\nQuestions:\n- Are you planning to release the code and data to facilitate work on this topic?\n\n\n\n## Author response update\n\nIn light of the author response, I have decided to increase the score to 5. I have also decreased my confidence to 3.\nThe main reasons for this score increase are the release of code and data as well as thoughtful clarifications on the experimental setup. This is good experimental work. I also think Appendix C.1 is a good first step towards drawing wider scientific conclusions from this work.\n The main reason not to increase the score further is that I believe the contribution still is quite narrow.  \n\nI chose to decrease confidence in my evaluation since it is now based more on the narrowness of the contribution, which is harder to assess, than on the experimental validity of this work.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and very active participation in the author response period. World state tracking is an important problem that encompasses existing problems like coreference resolution. I agree with R2 and R3 that proposing a novel environment in which we can investigate to what extend Transformers can tackle world state tracking should be interesting to the community. The majority of the reviewers agree that this paper presents an interesting benchmark [R2,R3,R4] with good thorough experimental work [R1,R2,R4]. However, R1 is confused about the positioning of the work and R4 finds the work narrow. R2, despite positive review, agrees with this assessment. I agree with this assessment as well and, after discussion with the program chairs, came to the decision that this paper is not ready for publication in its current state. I strongly encourage the authors to incorporate R1's and R4's feedback, in particular with respect to positioning this environment in comparison to TextWorld, and resubmit to the next venue.
Decision: Reject