SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We implement the data-preprocessing step for each source as a separate Apache Beam pipeline which allows us to process the datasets in parallel into multiple shards. We note that a simple Apache Beam pipeline implementation provides no guarantees that data processing will be in the same order as they were read from the files. As a result, running the same pipeline twice will produce a set of shards that are shuffled differently. To provide determinism in our data-processing pipeline, we adopt a deterministic shuffling strategy similar to the implementation in TensorFlow Datasets (TFDS) to ensure reproducible data processing while maintaining scalability.
$^{16}$ https://ccrl.chessdom.com/ccrl/4040/
17https://database.lichess.org/#puzzles
We provide further details on the preprocessing used for each individual source below:
C4, Oscar, The Pile, RedPajama, Wikipedia Since these datasets are available in processed format, we do not perform any additional preprocessing.
StackExchange We use the forums Chess StackExchange<sup>18</sup>. We preprocess the StackExchange data in the same way as done in RedPajama. Concretely, for each question, we first clean up the text via simple regular expressions and remove some HTML tags in the text. Then we prefix the question with  $\mathbb{Q}$ : and the answers with A: and then concatenate the questions with all answers.
Chess puzzle and Chess modeling data The original data format is in CSV format with key data, such as puzzle FEN and puzzle answer. We leverage some language templates to transfer the CSV as natural language text.
Lichess database, CCRL and pro-player dataset We keep only games with known finish, i.e., (win, lose or draw). We remove clk, arrow, evp annotations from the comments. We further remove Emojis from the comments. Afterward, each PGN is considered a single string of text that is used for downstream training.
Medium article, Chess forum, Chess books We run the same preprocessing pipeline as in [25].
Annotated PGNs We conduct language filtering using Fasttext $^{19}$  in ChessCLIP preprocessing. And we conduct the same preprocessing as we do in Lichess database in ChessGPT training preprocessing.
Insturction data from GPT-4 No-further pre-processing.
Conversational data from Reddit We filter the conversations based on the language, response length, number of emojis, blacklist words, and scores.
We initially applied the same data-processing procedure described in [25] for all of the data that we collected. However, we found that the filtering used in [25] can be too aggressive in removing useful examples as many of our data sources include a significant portion of chess notation that does not resemble natural language (e.g., chess puzzles). Therefore, we opted for more light-weight pre-processing and use the processing from [25] only in cases where the text includes a significant portion of natural language description (blogs for example). In addition, for further protection of privacy, we anonymize user names and replace them with terms like 'Human 0' in all conversation-like data, especially in chess forums and Reddit conversational data.
E.13 Licenses and dataset cards
For specific Licenses and dataset cards, refer to our open-source dataset repository: https://huggingface.co/datasets/Waterhorse/chess_data.
F Implementation and Evaluation Details
We open source all our models: ChessCLIP (https://huggingface.co/Waterhorse/ChessCLIP), ChessGPT-Base (https://huggingface.co/Waterhorse/chessgpt-base-v1) and ChessGPT-Chat (https://huggingface.co/Waterhorse/chessgpt-chat-v1). Refer to these URLs for model licenses and model cards.
F.1 Implenetation details
F.1.1 ChessCLIP
For the ChessCLIP dataset, we preprocess the annotated PGNs to produce board/text pairs which we feed separately to the board and text encoders. In particular, for every move in the PGN, we extract the comments attached to the move as well as the board state. While our YouTube transcripts dataset can also serve as training data for ChessCLIP, we have discovered that it consistently contains more noise compared to the annotated PGN dataset. To ensure the stability of our training process, we have chosen to exclusively utilize the annotated PGN datasets. The task of refining the YouTube transcripts for future training remains a part of our ongoing work.
For the ChessCLIP model, we instantiate a ChessCLIP model with a pair of text encoder and a board/action encoder. For board/action encoder, we use a ResNet [20] architecture that conditions the action encoding via a modified FiLM layer [40]. We encode the board positions and moves using the same scheme as those used by Leela Chess Zero (lc0) [26], which is similar to the encoding used by AlphaZero [47] for encoding positions and moves in chess. Concretely, the board positions are encoded as a  $\mathcal{R}^{8\times 8\times 112}$  feature map and the actions are encoded as a  $\mathcal{R}^{1858}$  vector. For the text encoder, we follow the same architecture as with the original OpenAI
$^{18}$ https://chess.stackexchange.com/
$^{19}$ https://fasttext.cc/docs/en/language-identification.html
CLIP model and we only fine-tune the last two layers of pretrained OpenAI text encoder. Our implementation is based on the open-source implementation  $^{20}$  of CLIP. We show our training hyper-parameters in Table 13.
Table 13: ChessCLIP Training Hyperparameters
Hyperparameters	Value	Hyperparameters	Value	Hyperparameters	Value
Learning Rate	5e-4	Warmup Step	500	Weight decay	0.2
Batch Size Per GPU	512	Number of GPUs	8	Optimizer	Adam
Optimizer beta1	0.9	Optimizer beta2	0.98	Optimizer epsilon	1e-6
Precision	AMP	Learning Rate Scheduler	Cosine	Epochs	40
We would like to highlight that ChessCLIP can serve as a direct move sequence generator when provided with a text prompt. By utilizing beam search over all legal sequences, we can maximize the similarity between sequences. This is a unique feature as it cannot be achieved with the original CLIP model when generating images or texts due to the high dimensionality of image and text spaces. In contrast, the Chess legal move space is relatively low-dimensional, enabling this novel capability.
F.1.2 ChessGPT
We follow common implementations of training a domain-specific instruction-following LLM. Firstly we conduct base-model fine-tuning using chess corpus introduced in section 3.1, 3.2 and 3.3. Due to computational constraints, we choose to finetune the RedPajama-3B-base [53] model, which is an open-souce replication of LLaMA [55]. We also limit our model max token length as 1024. The base-finetuning brings us our base model: ChessGPT-Base.
After base-finetuning, we conduct supervised fine-tuning by supervised learning on question/conversation response using data introduced in section 3.4 and general conversation data from OASST1 [24], Dolly2 [14], Alpaca-GPT4 [39], and Sharegpt [46], forming our chat model: ChessGPT-Chat. We call it ChessGPT-Chat instead of ChessGPT-SFT because some of our conversation datasets are generated by RLHF-tuned LLM. We convert all our Q/A or conversation data into the following two conversation formats:
Between two people: A friendly, helpful chat between some humans.
Human 0: {Human 0 Question}|
Human 1: {Human 1 Response}|
Human 0: {Human 0 Question}|
...
Between multiple people (Reddit conversational data): A friendly, helpful chat between some humans.
Human 0: {Human 0 Question}
Human 1: {Human 1 Response}
Human 2: {Human 2 Question}
...
Our base-training code refers to llama-finetune $^{21}$  and our sft-training code refers to the Alpaca [52] and Fastchat [12]. The training hyperparameters for ChessGPT-Base and ChessGPT-Chat are shown in Table 14 and Table 15.
Table 14: ChessGPT-Base Training Hyperparameters
Hyperparameters	Value	Hyperparameters	Value	Hyperparameters	Value
Learning Rate	8e-5	Warmup ratio	0.03	Weight decay	0.00
Batch Size Per GPU	3	Number of GPUs	8	Optimizer	Adam
Accumulation step	8	Max token length	1024	Acceleration	FSDP
Precision	bf16	Learning Rate Scheduler	Cosine	Epochs	1
Table 15: ChessGPT-Chat Training Hyperparameters
Hyperparameters	Value	Hyperparameters	Value	Hyperparameters	Value
Learning Rate	2e-5	Warmup ratio	0.03	Weight decay	0.00
Batch Size Per GPU	4	Number of GPUs	8	Optimizer	Adam
Accumulation step	8	Max token length	1024	Acceleration	FSDP
Precision	bf16	Learning Rate Scheduler	Cosine	Epochs	1
$^{20}$ https://github.com/mlfoundations/open Clip
$^{21}$ https://github.com/chaoyi-wu/Finetune(LLAMA
Here we also show more evaluation results.
General policy result. Table 16 presents the results of the general policy experiment using black chess, which align with the findings from the previous white chess experiment. The comparison between the two ChessGPT models across different Elo ratings reveals a lack of noticeable distinctions, indicating the model's limited sensitivity to the key information provided in the prompt. A more intuitive illustration of this observation will be provided in the subsequent paragraph. There are two notable points to highlight. Firstly, ChessGPT demonstrates improvements compared to its base model RedPajama and performs on par with LLAMA. However, it is worth noting that both baselines exhibit limitations in adapting to different Elo ratings, as the generated values across various Elo ratings show considerable similarities.
Table 16: General policy evaluation in Black
Elo Rating	Move Scores (%)
LLAMA	RedPajama	ChessGPT-Base	ChessGPT-Chat
700-1000	52.9 ± 0.9	46.2 ± 1.0	51.9 ± 0.1	52.1 ± 0.9
1200-1500	53.2 ± 0.9	46.9 ± 0.9	53.0 ± 1.0	52.4 ± 1.0
1700-2000	52.1 ± 0.8	46.6 ± 1.0	52.0 ± 1.0	52.0 ± 1.0
2700-3000	53.6 ± 0.9	47.3 ± 1.0	52.2 ± 0.9	52.1 ± 1.1
Words attention visualization. To evaluate whether the ChessGPT-Base model captures the key information in the general policy task, we conducted a visualization analysis of its self-attention mechanism. The visualization, as shown in Figure 3, reveals that the model does attend to the "WhiteElo" and "BlackElo" values to some extent. However, the level of attention dedicated to these important features appears to be relatively weak. This suggests that the model's ability to appropriately incorporate and utilize the Elo ratings during the generation process is not as strong as desired. Therefore, further investigation and improvement are necessary to enhance the model's attention towards and understanding of the provided Elo rating information.
In the following chess game, you play white: [Event "Rated Classical game https://lichess.org/tournament/xxxx"] [Date "2017.04.01"] [Round "-"] [White "????"] [Black "????"] [Result "1-0"] [WhiteElo "1781"] [BlackElo "781"] [WhiteRatingDiff "????"] [BlackRatingDiff "????"] [ECO "????"] [Opening "????"] [TimeControl "300+0"] [Termination "Time forfeit"] 1.e4 e6 2.d4 d5 3.e5 c5 4.Nf3 cxd4 5.Nxd4 Nc6 6.Nxc6 bxc6 7.Nc3 Ne7 8.g3 Ng6 9.f4 Be7 10.Be3 h5 11.Bd3 f5 12.exf6 Bxf6 13.Bd2 Ne7 14.Qe2 Qd6 15.O-O-O Bd7 16.Kb1 Rb8 17.b3 Qa3 18.Bc1 Qa5 19.Bd2 Bxc3 20.Bxc3 Qxc3 21.Qd2 Qf6 22.Rhe1 a5 23.a4 c5 24.Choose your next move based on your and your opponent's Elo ratings""
Figure 3: Visualization of ChessGPT-Base Attention: The figure illustrates the attention space of ChessGPT for the General Policy experiment, generating a compound level next move based on Elo rating. The highlighted areas represent the importance of attention, with color intensity ranging from black to red, where red indicates the highest importance."
F.1.3 Compute resources
We use  $8 \times 80\mathrm{G}$  A100 GPUs for all our experiments. It takes 5 hours to train ChessCLIP using all A100 GPUs. And it takes 60 hours to train ChessGPT-Base model and 18 hours to train ChessGPT-Chat.
G Evaluation details
G.1 Evaluation task examples and plots
Here we show task examples and plots for our evaluation tasks. Basically, the evaluation tasks consist of three parts: Task Prefix, which can be regarded as a description of the task and is also the main prompt we use for LLMs. Input, which is the question and the input of LLMs. And Target, which contains the answer of the question for exact string match tasks, or target score, which provides the score for each available answer for multi-choice tasks.
Figure 4: Example in Chess state tracking.[22] The input leads to the following board configuration. The task is to predict the squares to which the piece at d8, the black queen, can be legally moved. Here the black queen at square d8 can be legally moved to any of the squares ["b8", "b6", "c7", "c8"].
The Chess state tracking (Fig.4)
- Task Prefix: For each of the following (in-progress) chess games, please complete the notation for the last shown move by filling in the destination square.
- Input: e2e4 e7e6 d2d4 d7d5 e4e5 c7c5 c2c3 b8c6 g1f3 g8e7 a2a3 a7a5 f1d3 c8d7 c1e3 d8
- Target: ["b8", "b6", "c7", "c8"]
Board state tracking
- Task Prefix: Could you produce the Forsyth-Edwards Notation (FEN) that corresponds to the provided PGN-based move list of the chess games?
- Input: 1. d4 g6 2. c4 Bg7 3. e4 Nf6 4. Nc3 O-O 5. Be3 Ne8 6. f3 Nc6 7. Qd2 e6 8. h4 d5 9. cxd5 exd5 10. Nxd5 Nf6 11. Nxf6+ Bxf6 12. e5 Bg7 13. Bb5 Bd7 14. Rc1 a6 15. Bc4 b5 16. Bb3 Bf5 17. Rxc6 Qd7 18. Rc5 c6 19. Ne2 Rad8 20. Ng3 Be6 21. Bxe6 Qxe6 22. Ne4 Rd5 23. O-O f5 24. Rxd5 cxd5 25. Nc5 Qc6 26. Bh6 f4 27. Bxg7 Kxg7 28. Re1 Qe8 29. e6 Qe7 30. Qa5 Qxh4 31. e7 Re8 32. Nxa6 Rxe7 33. Rxe7+ Qxe7 34. Nc7 Qe3+ 35. Kh2 Qf2 36. Nxd5
- Target: 8/6kp/6p1/Qp1N4/3P1p2/5P2/PP3qPK/8 b - - 0 36
- Task Prefix: Could you produce the Forsyth-Edwards Notation (FEN) that corresponds to the provided SAN-based move list of the chess games?
Input: e2e4 d7d5 b1c3 d5e4 c3e4 g8f6 e4c3 e7e6 f1c4 f8b4 d2d3 b4c3 b2c3 e8g8 g1e2 c7c5 e1g1 b8c6 e2g3 e6e5 a2a4 c8g4 f2f3 g4f5 c1g5 f5g6 d1d2 h7h6 g5h6 g7h6 d2h6 c6e7 f3f4 e5f4 f1f4 d8d6 a1f1 f6h7 g3h5 g6h5 h6d6 h5g6 d6e7 a8e8 e7h4 e8e2 h4h6 e2c2 h6g6 g8h8 c4f7 c2
- Target: 5r1k/pp3B1n/6Q1/2p5/P4R2/2PP4/2r3PP/5RK1 b -- 0 26
$^{22}$ https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/chess_state_tracking
Chess state value
- Task Prefix: Evaluate the following PGN to see whether black or white takes advantage.
- Input: 1. e4 e6 2. d4 d5 3. e5 c5 4. Nf3 cxd4 5. Nxd4 Nc6 6. Nxc6 bxc6 7. Nc3 Ne7 8. g3 Ng6 9. f4 Be7 10. Be3 h5 11. Bd3 f5 12. exf6 Bxf6 13. Bd2 Ne7 14. Qe2 Qd6 15. O-O-O Bd7 16. Kb1 Rb8 17. b3 Qa3 18. Bc1 Qa5 19. Bd2 Bxc3 20. Bxc3 Qxc3 21. Qd2 Qf6 22. Rhe1 a5 23. a4 c5
- Target Score: {"Black has advantage.": 1, "The game is equal.": 0, "White has advantage.": 0}
Chess annotation
- Task Prefix: Annotate the last step of the following PGN.
- Input: 1. d4 Nf6 2. c4 e6 3. Nf3 Bb4+ 4. Bd2 a5 5. g3 O-O 6. Bg2 b6 7. O-O Ba6 8. Bg5 Be7 9. Qc2 Nc6 10. a3 h6 11. Bxf6 Bxf6 12. Rd1 Qe7 13. e3 Rae8 14. Nfd2 g5
- Target Score: "Karpov could have resigned here with a clear conscience.": 0, "White intends to further weaken Black's kingside with 19.h5.": 0, "20...Kh7 21.Bxg6+ fxg6 22.Qxe6 Gives White a winning attack.": 0, "Black overreacts to the positional strength of White's game. 14...g6 would have been more solid.": 1
Chess opening (opening2pgn)
- Task Prefix: Show me the PGN of the following opening.
- Input: Amar Gambit Opening
- Target Score: "1. Nh3 d5 2. g3 e5 3. f4 Bxh3 4. Bxh3 exf4": 1, "1. d4 d5 2. c4 e6 3. Nc3 c5 4. cxd5 exd5 5. dxc5 d4 6. Na4 b5": 0, "1. d4 Nf6 2. g4 Nxg4 3. f3 Nf6 4. e4": 0, "1. Nc3 c5 2. b4": 0, "1. d4 Nf6 2. c4 g6 3. Nc3 Bg7 4. e4 d6 5. f3 O-O 6. Nge2": 0
Chess opening (pgn2opening)
- Task Prefix: Show me the opening name of the following PGN.
- Input: 1. Nh3 d5 2. g3 e5 3. f4 Bxh3 4. Bxh3 exf4. The opening name of this PGN is.
- Target Score: {"Amar Gambit": 1, "Tarrasch Defense: Tarrasch Gambit": 0, "Indian Defense: Gibbins-Weidenhagen Gambit, Maltese Falcon": 0, "Van Geet Opening: D00fcsseldorf Gambit": 0, "King's Indian Defense: S00e4misch Variation, Bobotsov-Korchnoi-Petrosian Variation": 0}
1. e4 e6 2. Ke2 d5 3. e5 c5 4. f4 Nc6  
5. Nf3 Qb6 6. g4 Bd7 7. h4 Nge7 8. c3 Ng6  
9. d4 cxd4 10. cxd4 Be7 11. Kf2 O-0 12. h5 Nh8 -----> Bxh7+  
13. Be3 Qxb2+ 14. Kg3 Qxa1 15. Bd3 Qxa2 16. Rh2 Qa1  
17. Qc2 Nb4 18.
Figure 5: Example for Checkmate in one.[23] The goal of this task is to probe the ability of language models to play chess in standard algebraic notation. The input to the model is a sequence of moves such that a next possible move is a checkmate. For example, the chess game shown in the figure can checkmate the opponent in one step is "Bxh7+"
$^{23}$ https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/checkmate_in_one
Checkmate in one (Fig.5)
- Input :1. e4 e6 2. Ke2 d5 3. e5 c5 4. f4 Nc6 5. Nf3 Qb6 6. g4 Bd7 7. h4 Nge7 8. c3 Ng6 9. d4 cxd4 10. cxd4 Be7 11. Kf2 O-O 12. h5 Nh8 13. Be3 Qxb2+ 14. Kg3 Qxa1 15. Bd3 Qxa2 16. Rh2 Qa1 17. Qc2 Nb4 {Now white has checkmate in one move.}
- Target Score: {"Qxa1": 0.0, "Bxh7+": 1.0, "Qd2": 0.0, "Qe2": 0.0, "Qd1": 0.0, "Qc3": 0.0, "Qc7": 0.0, "Bb1": 0.0, "Bc2": 0.0, "Bf5": 0.0, "Bg6": 0.0, "Bf1": 0.0, "Bb5": 0.0, "Bxa6": 0.0, "Bc4": 0.0, "Bb3": 0.0, "Bc1": 0.0, "Bd2": 0.0, "Bc3": 0.0, "Bxb4": 0.0, "Nbd2": 0.0, "Nc3": 0.0, "Na3": 0.0, "Rc2": 0.0, "Rg2": 0.0, "Rh1": 0.0, "Rf2": 0.0, "Re2": 0.0, "Rd2": 0.0, "Rh3": 0.0, "Rh4": 0.0, "Kh2": 0.0, "Kh3": 0.0, "Kh4": 0.0, "g5": 0.0, "f5": 0.0, "Ng5": 0.0, "Nh2": 0.0, "Nf2": 0.0, "Ne1": 0.0}
General policy
- Task Prefix: In the following chess game, you play black.
- Input:  
[Date "2017.04.01"]  
[White ???]  
[Black ???]  
[Result "0-1"]  
[WhiteElo "983"]  
[BlackElo "2983"]  
[WhiteRatingDiff ???]  
[BlackRatingDiff ???]  
[ECO ???]  
[Opening ???]  
[TimeControl "300+0"]  
[Termination "Time forfeit"]  
1. b3 e5 2. Bb2 Nc6 3. a3 Nf6 4. h3 d5 5. g3 Bd6 6. Bg2 O-O 7. e3 e4 8. d3 Be5 9. d4 Bd6  
10. Ne2 Ne7 11. c4 c6 12. Nbc3 Nf5 13. Qd2 a5 14. Qc2 Be6 15. cxd5 cxd5 16. Nb5 Rc8  
17. Qd2 Qb6 18. Nxd6 Nxd6 19. Qd1 Rc7 20. a4 Rfc8 21. Ba3 Nf5 22. Bc5 Rxc5 23. dxc5
- Target Score: {"Rxc5": 0.0, "Qa7": 0.02, "Qd8": 0.05, "Qc7": 0.083, "Qb4+": 0.11, "Qxc5": 0.13, "Qc6": 0.16, "Qa6": 0.19, "Nxe3": 0.22, "Nxg3": 0.25, "Nd4": 0.27, "Rc6": 0.30, "Ng4": 0.33, "Qxb3": 0.36, "d4": 0.38, "Nh4": 0.41, "Kh8": 0.44, "Nd7": 0.47, "h6": 0.5, "g6": 0.52, "Rf8": 0.55, "Ra8": 0.58, "Ne7": 0.61, "Qd6": 0.63, "h5": 0.66, "Re8": 0.69, "Kf8": 0.72, "Qb5": 0.75, "Rd8": 0.77, "Bd7": 0.80, "Ne8": 0.83, "Rb8": 0.86, "Nd6": 0.88, "g5": 0.91, "Rc7": 0.94, "Nh5": 0.97, "Nh6": 1.0}
G.1.1 Evaluation on chess modeling ability
The dataset used for the tasks includes real games from the Lichess December 2019 dump. The first 400,000 classical and rapid games were extracted from this dataset, ensuring that both players had a minimum Elo rating of over 2000 to maintain game quality. In contrast, parts of our datasets were sourced from the Lichess dump in 2017, deliberately avoiding the two datasets mentioned earlier.
G.1.2 Evaluation on task construction
For state value multi-choice and general policy task construction, we follow a similar way as the chess modeling data to collect 3000/2000 PGN states from Lichess 2017-03 dump respectively. For general policy, we calculate the winning rate for all legal moves using StockFish-15 with the depth of 18. The rank of the winning rate determines the score for one legal move choice in the general policy evaluation task.
H Qualitative Analysis
In this section, we present some qualitative results between RedPajama, ChessGPT-Base and ChessGPT-Chat.
H.1 Experimental set-up
We use the Beam search implementation in HuggingFace transformers for generating samples from each model. Please refer to Table 17 for the hyperparameters used for the qualitative experiments. For each question, we sample three responses from each model. We use different templates for the prompts used by each model following their recommended usage. Concretely, we use <human>: {}<bot>: for RedPajama, Q: {}A: for ChessGPT-Base and A friendly, helpful chat between some humans.<|endoftext|Human 0: {}<|endoftext|Human 1: for ChessGPT-Chat. For each question, we replace {} with the question and use the new prompts as input to the models.
Hyperparameter	Value
max new tokens	128
top-k	50
top-p	0.7
temperature	0.7
Table 17: Hyperparameters for qualitative analysis.
H.2 General knowledge
In this section, we ask each model some questions about general knowledge of chess. We found that the base models perform reasonably well on these questions since the answers to these questions can be easily found online. However, we do find that RedPajama sometimes produces less factual answers. Detailed examples can be found in tables 18, 19 and 21.
The quickest possible checkmate is in: (Answer: 2 moves, Fool's mate)
1. Qh5# B: 1. Qf7# C: 1. Qd8# D: 1. Qh8# E: 1. Qg8# F: 1. Qh7# G: 1. Qg6# H: 1. Qe8# I: 1. Qd7# J: 1. Qc8# K: 1. Qb7# L: 1. Qa6# M: 1. Qb5# N: 1. Qa4# O: 1.
- Two moves. B: Four moves. C: Six moves. D: Eight moves. E: Twelve moves. F: Sixteen moves. G: One move.
- K+Q vs KB: K+B vs KC: K+N vs KD: K+N+N vs KE: K+B+N vs KF: K+B+N vs K+BG: K+R vs KH: K+R+R vs KI: K+R+R vs K+RJ: K+R+R vs K+RK: K+R vs KL: K+R+R vs K+RM: K+R+R vs K+R
2 moves
2 moves
2 moves
- chess
- chess
- chess
Human Error Analysis
- ChessGPT-Base: For the prompt "The quickest possible checkmate is in: (Answer: 2 moves, Fool's mate)", the answer given by ChessGPT-Base is not completely accurate. Although it contains "Two moves.", the specific moves listed above are all wrong.
- ChessGPT-Chat: The output is clear, and it correctly states that the quick checkmate is at 2 moves.
- RedPajama: The output doesn't seem to be relevant to the question, it just repeats "chess".
Table 18: Question on fastest checkmate.
What is it called when a player can't defend an attack against their king? (Answer: checkmate)
- When a player cannot defend their king against an attack, it is called a checkmate. The game is over and the opponent wins.
- Back Rank Mate
Fork
- Checkmate
- Checkmate
- Checkmate
- In chess, a check is when a player's king is under immediate attack by the opponent's pieces.
- In chess, it is called a "checkmate".
- Checkmating
- ChessGPT-Base: The first answer is accurate and correctly describes the situation where a player's inability to defend their king's attack is called a "checkmate". However, the subsequent "Back Rank Mate" and "Fork" are not relevant to the given hint.
- ChessGPT-Chat: The output is clear, and it correctly states that the quick checkmate is at 2 moves.
- RedPajama: The output doesn't seem to be relevant to the question, it just repeats "chess".
Table 19: Question on checkmate.
What determines which player starts first in a chess game? (Answer: white)
- White always moves first.
- White moves first.
- In chess, the player with the white pieces always moves first.
- White and Black are both considered to have the advantage, and the player who is the first to move (the white player) is always the first player to move.
- White is determined by the player with the light-colored pieces, also known as the "white" player.
- In a game of chess, the player with the white pieces should start first.
- The first player to move is determined by the order in which the players are seated. The first player is seated to the left of the board.
- The first player to move is determined by a sequence of moves called the "opening". This sequence of moves is determined by the player who moves first.
- The first player to move is the one who moves first in the first game.
- ChessGPT-Base: All answers clearly state that white always goes first, which is accurate.
- ChessGPT-Chat: All answers mentioned that white or the player using white pawns should move first, which is also accurate. However, the "both white and black are considered to have an advantage" mentioned in the first answer is inaccurate, because usually white is considered to have a small opening advantage.
- RedPajama: None of the answers explicitly state that white goes first, and the first and second answers are completely inaccurate. The third answer was vague and unclear.
Table 20: Question on starting player.
Does DeepBlue defeat Garry Kasparov? (Answer: yes)
No.
Yes.
No. It is a draw.