Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper considers an intriguing problem: can language models, when trained on
purely textual representations of Chess games, learn the underlying dynamics of
the game? The authors argue that this could be a preliminary step towards
tackling the symbol grounding critique of methods like transformers. When
transformers are utilized in natural language settings, it's challenging to
determine whether the models are operating at a pure syntactic level, or whether
there is some rudimentary level of "understanding", given that the models are
only exposed to text. In contrast, in Chess, one can train in a purely textual
fashion (or with some limited symbol grounding information), and probe how well
the result models the state of the underlying Chess position.

The authors use the GPT2-small based transformer architecture and train from scratch on a dataset of
high-quality Chess games between humans, represented in UCI notation -- a
textual listing of the moves made by each player. In some experiments, this
purely textual input is supplemented with explicit board state information as
well, to test the impact of this additional signal. The authors evaluate the
system on two types of inference tasks: a trained model's ability to
successfully locate a piece on the board, and the model's ability to determine
where to move a chosen piece to next. In each case, there are two
evaluation metrics: an "exactness" metric (i.e., whether the model picked the
same piece/move as the human did in the corresponding game) and a "legality"
metric (i.e., whether the model picked a permissible move). The former metric is
more stringent, as it's also measuring the strategic awareness of the model. The
authors demonstrate through their experiments that transformers are successful
at both inference tasks with very high accuracy, particularly when evaluated
using the legality metric, using shorter sequences of moves, and larger
datasets.

Strengths of the paper:
  + This is a creative application of transformers to a non-traditional textual
    inference task. It may inspire others to devise other interesting applications.
    The results are intriguing and expand our understanding of what may be
    possible with transformer architectures.
  + The authors' approach is a fairly straightforward application of off-the-
    shelf techniques -- and I mean that in a good way. There are no unnecessary
    complications or ad hoc additions to the system design.
  + The paper is very clearly written, well-organized, and easy to follow.

Areas for improvement/questions for the authors:
  - I'm a little confused as to why performance peaks with p=0.05/0.15 in the
    UCI+RAP models. If my understanding is correct, then RAP with
    p=1.0 would include piece annotations with *every* move during training
    time and at inference. So shouldn't this make it easier to track pieces (as
    the authors themselves note in 4.1)? The reason for this is given as greater
    "mismatch between training and inference" -- I'm not sure what this means.
  - On a related note, I'm also confused by the performance of the Oracle Baseline
    in Table 3. In some instances, this is outperformed by trained models with
    *less* information. But my understanding was that this oracle would serve
    as an upper bound on model performance. So how are we outperforming the
    upper bound?

On balance, I think the strengths of the paper outweigh my concerns, and I
recommend ACCEPTANCE.

Couple of minor issues:
  - At the bottom of Page 6, the text references results in Table 3, which
    should be Table 2.
  - Section C in the Appendix is currently empty.

review_title: An interesting application of transformers to Chess playing

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper explores the abilities of transformer-based models to do grounded state tracking via chess. They train a GPT-2 on chess games, showing that it can learn the rules of the game by being able to state-track and predict valid next actions.

Pros:
1. A very thorough set of experiments are given to explore how transformers can be used to track states in a game such as chess and results show that with enough data, transformers can predict the locations of pieces across a decent amount of history as well as predict legal actions.
2. The paper is well-written in terms of general writing clarity and I was able to follow *what* was happening throughout.

Cons:
1. I am a bit lost as to the motivation and positioning behind the paper, i.e. I was unsure as to *why* things were happening as they were. The authors say that they are using transformers to see how transformers can learn grounded language when world states are available but I do not see this work positioned with respect to other work on grounded language nor work on state tracking generally found in model-based RL.
(i) An example of the former for grounded language learning (given that they cite Bender and Koeller) would be instances such as vision and language navigation (Anderson et al. https://openaccess.thecvf.com/content_cvpr_2018/papers/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.pdf), the Nethack Learning Environment for game grounding (Kuttler et al. https://arxiv.org/abs/2006.13760), or text games (Cote et al. https://arxiv.org/abs/1806.11532 and Hausknecht et al. https://arxiv.org/abs/1909.05398). I am not sure how state tracking in chess implies that transformers can do grounded language learning.
(i) In terms of just the state tracking parts, there has already been much work in agents that learn the rules of the world as they play through them. This can happen with World Models (Ha and Schmidthuber https://arxiv.org/abs/1803.10122) or in cases like Alpha-Zero (Silver et al. open access version https://kstatic.googleusercontent.com/files/2f51b2a749a284c2e2dfa13911da965f4855092a179469aedd15fbe4efe8f8cbf9c515ef83ac03a6515fa990e6f85fd827dcd477845e806f23a17845072dc7bd) which learn the rules of Chess from scratch via self play. How does using a transformer compare to these works?
2. Given the above and the fact that all of the pieces of the methodology of this work are taken from others (the architecture, training, state representation, etc.) - the main contribution is the experimental design and the results themselves. In this case, I would have liked to see more analysis regarding exactly what properties of the transformer they think is responsible for helping the model to learn and also a potential qualitative analysis of what the failure cases are.

Overall, the paper has some interesting ideas, experiments, and results but is not connected to the motivation/is not positioned well with respect to closely related work.

Post author response:
See comment below for further score justification.
review_title: Review 1

final_score: rating: 5: Marginally below acceptance threshold
review_content: 	
### Topic
This paper explores learning chess from raw notation as a benchmark for the ability of language models to track world state. Chess is an interesting benchmark, as a set of moves can be unambiguously linked to a world state, there are large amounts of data available and the model can easily be probed for its board tracking abilities. The contributions of this paper are twofold: (i) introducing blindfolded chess as a benchmark for grounded language learning and world state tracking, as well as a suite of probing tasks to evaluate models; (ii) empirical evidence that transformer language models can learn both the rules of the game and to track board state.

### Pros:
-	Blindfolded chess is an interesting benchmark for grounded language learning and as a testbed for models to track world state. It is unambiguous, data-rich, has a limited vocabulary and models trained on it can easily be probed. This adds to prior papers on learning chess with transformers that have mainly focused on the performance of such models.
-	The use of SAN + RAP trick is an interesting one to be able to probe the current location of pieces.
-	The oracle model is interesting, as it demonstrates the gap between a model that must track world state and one that has access to it. The multi-view model is also interesting as an example of how additional supervision may help the model.
-	The analysis across different data sizes and game length is interesting.

## Cons:
-	The paper feels rushed at times: for instance, there are no references in text to Appendix D, despite it being an interesting demonstration of the analysis that can be done in this environment. 
-	The comparison to related work on world state tracking and grounded language learning could be substantially improved. The authors only very briefly mention previous work: the papers on the TextWorld environment (Cote et al, 2018), seem relevant, although the framework is that of interactive environments and RL. Other papers on grounded language learning also seem relevant, such as Alexander G. Ororbia II et al, 2019. More so than the lack of references, an issue with this paper is that few connections are made to wider research issues in grounded language learning / implicit world state tracking. Instead, the text tends to simply state experimental results. This makes the contribution of this paper very narrow.
-	It is not clear why the multi-view trained models underperform on the low data settings compared to the models without the extra supervision. If there is more explicit supervision of the model, should it not be better at tracking world state?
-	It would be good to make clear if this dataset and probing tasks will be released, and the code made available. This would be a welcome step in standardizing this new domain, as prior approaches have all used different settings/datasets/notations/evaluations.
-	Despite this being a GPT-small architecture and the authors using a small subset of the available training data, the accuracies are high for legal move predictions on Train L. Those are the ones that are directly related to tracking the world state and knowing which piece movements are allowed. This limits the use of this task as a future benchmark for world state tracking in the data-rich setting. It might be possible to make the task more challenging by asking the language model to predict the entire board state (deconvolution) or by focusing on hard subsets of the task (e.g: pseudo-legal infractions on long histories).
-	Many challenges of natural language are not present in this environment, such as coreference, limiting the applicability of results on this task to e.g bAbl. The use of a composite vocabulary (e.g: “e4” instead of “e”, “4”) also limits the compositional challenges of the dataset. However, it is true that using a vocabulary where “e4” is two tokens would make probing more difficult.

Minor comments:
-	Incomplete board state: The board state as represented is actually not complete, as it ignores whether a pawn can be taken en passant ( https://en.wikipedia.org/wiki/En_passant ). En passant is rare enough that this might not matter, but it does mean the Oracle/Multi view models do not perfectly capture the state. Note that this seems to also be an issue in Oshri and Khandwala (2015) from whom this representation is derived.
-	The ending square task does not allow us to probe whether the model captures the full range of possible moves for a piece, or whether it selects a subset of those. E.g: the model might reach 100% accuracy on this task without ever moving a rock/bishop/queen by more than one square (less likely that there are pieces in between).
-	It is not clear what you do with games of length 100-150. These seem included in the training set but excluded from the probes. Are those included in the perplexity results of Table 6.
-	Typos:
o	Table 4 breakdown in appendix B

### Recommendation:

I lean reject for this paper.
The core idea is interesting, but the paper fails to make connections to wider issues in world state tracking and grounded language learning, making it overly narrow. Both the missing references and the missing links to wider concepts in this litterature are a symptom of that. With a few caveats, the experiments are sound, but the analysis could be improved to go further than simply stating the results. The overall wording and presentation of the paper must also be improved.

Questions:
- Are you planning to release the code and data to facilitate work on this topic?



## Author response update

In light of the author response, I have decided to increase the score to 5. I have also decreased my confidence to 3.
The main reasons for this score increase are the release of code and data as well as thoughtful clarifications on the experimental setup. This is good experimental work. I also think Appendix C.1 is a good first step towards drawing wider scientific conclusions from this work.
 The main reason not to increase the score further is that I believe the contribution still is quite narrow.  

I chose to decrease confidence in my evaluation since it is now based more on the narrowness of the contribution, which is harder to assess, than on the experimental validity of this work.

review_title: Interesting new benchmark that would benefit from more connections to the literature. 

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper is an interesting exploratory study analyzing the ability of language models to track the state of a chessboard. The authors adopt a clever chess notation which allows them to probe the language model's state tracking ability by looking at its next word prediction (akin to probes in [1]). Quite remarkably, language models finetuned on chess data store a very accurate state representation, and predict legal moves over 90% of the times even without a visual representation of the board.

-----------------------------------

Strengths of the Paper:

1. A clever probing method to analyze language model's state tracking abilities. 

2. Well-designed experiments and very interesting results in a mostly unstudied area.

3. Wel written paper with several baselines and ablations.

-----------------------------------

Weaknesses of the Paper / Possible additional analysis:

While the work in itself is very interesting and clean, I would have loved to see more analysis studying the model. This a very rich testbed where a lot of interesting experiments can be done! For instance,

(1) Does model performance / chess quality improve with larger language models like GPT2-md or GPT2-l?  
(2) What are the kinds of errors these language models make (in terms of legal moves)? Do these errors disappear when you check top-k tokens?  
(3) What are the kinds of moves the model is good at (when considering argmax predictions)? Is it learning any strategy at all? You can measure this quite well automatically using the chess engine scores which indicate who is winning, and comparing the change in scores when the actual move is played vs the language model's predicted move  
(4) Finally can insights from (2) and (3) be transferred to other real-world applications? Is there a correlation between the probing literature on natural language processing tasks and the results you find?  
(5) It will be cool to check other kinds of visual state fusion strategies like pseudo-self attention [2]

-----------------------------------

Other Feedback:

1. on the bottom of page 6, did you mean Table 2?
2. A couple of baselines will be useful in Table 2 and 3. The first could be upperbound EM baselines using engines like Stockfish or AlphaGo (since nearly everyone is worse at Chess than them, I expect the EM score to be lower than 100%). The second could be a random lowerbound to EM, where a random move from the set of LM is chosen. Finally it will be good to see EM performance of GPT-2 considering only the set of LM.
3. LM is an overloaded acronym which can cause confusion to the reader (language model vs legal move).

-----------------------------------

Overall Recommendation:

This is an exciting and rich testbed with a lot of interesting questions to answer. The authors have conducted well-thought experiments and reported interesting results. I'm leaning accept, but I encourage the authors to keep working on this setup (perhaps using some of the suggestions discussed above) and try to check if any of the insights here can be transferred to better understanding of language models on natural language.

-----------------------------------

References:

[1] - https://www.mitpressjournals.org/doi/pdfplus/10.1162/tacl_a_00115  
[2] - https://arxiv.org/pdf/1908.06938.pdf

review_title: Interesting testbed and surprising results!

Meta-review: I thank the authors for their submission and very active participation in the author response period. World state tracking is an important problem that encompasses existing problems like coreference resolution. I agree with R2 and R3 that proposing a novel environment in which we can investigate to what extend Transformers can tackle world state tracking should be interesting to the community. The majority of the reviewers agree that this paper presents an interesting benchmark [R2,R3,R4] with good thorough experimental work [R1,R2,R4]. However, R1 is confused about the positioning of the work and R4 finds the work narrow. R2, despite positive review, agrees with this assessment. I agree with this assessment as well and, after discussion with the program chairs, came to the decision that this paper is not ready for publication in its current state. I strongly encourage the authors to incorporate R1's and R4's feedback, in particular with respect to positioning this environment in comparison to TextWorld, and resubmit to the next venue.
Decision: Reject