Title: LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
Primary area: datasets and benchmarks
Keywords: Large Language Models (LLMs), Instruction Following, Reasoning Evaluation, Multi-Agent Systems, Chess AI

Abstract: We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extended agentic interaction in the domain of chess. We rank over 50 open and closed source models by playing against a random opponent using a range of behavioral metrics, including win and loss rates, move quality, move legality, hallucinated actions, and game duration. For a subset of models, we derive an Elo estimate by playing against a chess engine with a variably configured skill level. Despite the simplicity of the instruction-following task and the weakness of the opponent, many state-of-the-art models struggle to complete games or achieve consistent wins. Similar to other benchmarks on complex reasoning tasks, our experiments reveal a clear separation between reasoning and non-reasoning models. However, unlike existing static benchmarks, the stochastic and dynamic nature of LLM CHESS uniquely reduces overfitting and memorization while preventing benchmark saturation. To support future work on evaluating reasoning and instruction-following in LLMs, we release our experimental framework, a public leaderboard, and a dataset of associated games.

TL;DR: We introduce LLM CHESS, a dynamic benchmarking framework that evaluates reasoning and instruction-following in language models through chess gameplay against weak opponents.