Title: Generating Creative Chess Puzzles
Venue: NeurIPS 2025
Accepted: yes
Keywords: chess puzzle

creativity

generative model
Abstract: While Generative AI rapidly advances in various domains, generating truly creative, aesthetic, and counter-intuitive outputs remains a challenge. This paper presents an approach to tackle these difficulties in the domain of chess puzzles. We start by benchmarking Generative AI architectures, and then introduce an RL framework with novel rewards based on chess engine search statistics to overcome some of those shortcomings. The rewards are designed to enhance a puzzle's uniqueness, counter-intuitiveness, diversity, and realism. Our RL approach dramatically increases counter-intuitive puzzle generation by 10x, from 0.22\% (supervised) to 2.5\%, surpassing existing dataset rates (2.1\%) and the best Lichess-trained model (0.4\%). Our puzzles meet novelty and diversity benchmarks, retain aesthetic themes, and are rated by human experts as more creative, enjoyable, and counter-intuitive than composed book puzzles, even approaching classic compositions. Our final outcome is a curated booklet of these novel AI-generated puzzles, which is acknowledged for creativity by three world-renowned experts.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a methodology for the generation of creative chess puzzles, combining the definition of quality metrics and an RL pipeline. There are extensive method descriptions and I like the human evaluation study.\n\nBut overall I find this paper has several problems:\n* It does not clearly specify what the contributions are and does not relate the approach to existing work (no related work section in the main paper)\n* It uses a lot of jargon, and stays vague or imprecise on several key technical details. \n* It is very applied to chess puzzle generation, and does not seem to introduce any technical novelty, except perhaps the definition of the counter-intuitiveness measure, which is again very domain-specific.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposes to solve chess puzzle but these are not really defined anywhere in the paper. Definitions themselves use a lot of technical jargon: eg ”excluding puzzles with multiple, solutions, non-standard elements, special stipulations, retrogrades, missing information, shortest mates, or mathematical aspects.” Do all chess puzzle have the same objective (check mate)? \n\nMetrics definitions\n\nThe section dedicated to give definitions for the different metrics does not completely achieve its aim.\n* Uniqueness: this definition is rather unclear, are we talking about the uniqueness of solutions or the uniqueness of moves? Is a unique solution a solution where each move is unique? And does the proposed criterion (Eq 1) really translates in qualifying uniqueness of a solution? How does the Stockfish engine work, and can we rely on it? If the second best move has a chance of leading to win >0, then it means there might be another solution right there?\n* Creativity: this paragraph doesn’t offer any precise metric, just some discussion about the difficulty of evaluating creativity.\n* Novelty: \n  * What are FEN strings? Why is it a good choice? What is the principal variation measure? \n  * about the entropy of the board representation: what is the model used? how is it trained?\n  * I’m not sure the Levenshtein distances capture any kind of semantic distances between board positions — pieces could be differently placed but the abstract configurations may be similar, and inversely they could be placed very similarly but the abstract configurations might be very different\n* Counter-intuitiveness: This is interesting, but here again a bit vague: this paragraph starts to define it, but stops before giving a final definition\n* Aesthetic: here again a lot of discussion but no precise definition\n\nGiven how central these terms are, it would perhaps be good to have a concise, descriptive descriptions of these features right from the introduction. This is done to some extent but using very vague terms: eg “Counter-intuitive solutions are those that, at first glance, seem are terrible or out-of-mind, but are, in fact, brilliantly effective.” I think having a more technical, practical definition would help here. The abstract also appears to use vague terms and technical jargon.\n\nOptimization\n* Wouldn’t the use of a reverse KL divergence loss limit the creativity of the policy by forcing it to only generate puzzle within the distribution of the base model?\n* I’m not sure I understand, is the policy pretrained with the lichens dataset with supervised finetuning, or are these 100k examples only added to the buffer at the beginning of the PPO phase? Does PPO even have a replay buffer? It’s an online policy, so not sure this would work so well to add completely offline data? This doesn’t seem to significantly help either (Fig 4), so not sure why it is introduced. \n* Is there any reason why we should apply RL here instead of doing supervised fine-tuning on positive examples: diverse set of unique and counter-intuitive solutions?\n* It would be interesting to see the ratio of the number of solutions accepted to the replay buffer as a function of time? If diversity crashes we’d have fewer and fewer novel-enough solutions to enter the replay buffer\n\nCounter-intuitiveness measure:\n* This seems like the main contribution of the paper, and it takes only a limited part of the paper and is not very clearly laid out.\n* The fact that it captures well human-judgement on counter-intuitiveness is good but does not prevent the measure from being hacked by the policy during RL training. I would like to see an analysis of highly counter-intuitive models and some evidence that the measure is not being completely hacked by the policy in some strange way — eg by adding elements that would confuse the short-budget search (eg more pieces = higher branching factor = slower search) more than the high-budget search.\n\nNovelty:\n* “Inspecting Table 3 indicates that the datasets sampled from the generative models exhibit both novelty (when compared to the Lichess data) and internal diversity (when compared to other samples from the same model)” —> I find it hard to interpret these measures in an absolute sense, what’s the threshold that allows us to say these are definitely “novel” vs “not novel” here? Using the Lichess dataset as a baseline needs to be a bit justified, how do we know this dataset is diverse? If it’s a collection of boards people have played, maybe people don’t play such diverse games? The examples in the booklet help with that, thanks for including these. \n\nAesthetic:\n* From Fig 3 it’s hard to say if anything special is going on. There is some aesthetic distribution in the training data, and some in the fine-tuned model and some in the RL-trained model. There is not obviously more in one or the other, it seems more or less random. Of course, it wasn’t optimized for. Is there a reason why not?\n\nConclusion:\n* It's not clear what the contribution and novelty of this paper are compared to previous work\n* The main contribution seems to be the definition of evaluation metrics / reward functions, which have a rather limited scope --- maybe this work belongs to a more specialized venue?\n* The text is not always clear, it's dense and uses a lot of jargon, while missing key details. A lot is sent back to a very long appendix.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper trains generative models to synthesize chess puzzles. It first analyzes various kinds of generative models using dataset of chess puzzles and then fine-tunes the autoregressive generative model using RL for improved performance. The results show that the final RL-tuned model surpasses the existing puzzle dataset in various metrics (e.g. counter-intuitiveness). Another tangible outcome of the project is a manually curated booklet of 50 AI-generated puzzles.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- This project represents a useful, creative and unique application of generative AI tools. To my knowledge, while there have been numerous works which develop chess-playing systems, generative approaches for creating puzzles have not been explored. \n\n- The paper takes a pretty sensible approach — training a generative model and finetuning with RL. The results demonstrate clear benefits from the second stage of RL-based training.\n\n- The final booklet of puzzles could be a valuable outcome on its own as the quality is comparable to puzzles found in typical chess books.\n\nWeaknesses:\n\n- The technical contributions of this work are somewhat limited. Beyond the interesting choice of the setup tackled, the approach is to simply train typical generative models on an existing dataset and finetune the auto-regressive one using RL (mirroring standard practices in training LLMs e.g. for code generation).\n\n- While the setup tackled in this work is unique, the final system that can generate puzzles is somewhat limited in scope. In particular, unlike chess engines which can start from custom positions, the puzzle generation system lacks the flexibility to help coach a player for specific scenarios. For example, this system can only be queried to generate a puzzle, but cannot be instructed to generate one of a specific kind “e.g. generate an opening position puzzle reflecting a defensive scenario”.\n\n- The performance of the final system, while comparable to puzzles in books, is not ‘superhuman’, and in fact requires manual effort in curation etc (for example the booklet required significant manual effort). Moreover, the uniqueness of the puzzles from the RL-based system is limited. As such, it is unclear whether random samples from the learned model are of the desired quality.\n\nQUESTIONS:\nThe results in Table 2 are rather surprising. One would expect a well-trained generative model to mirror the statistics of the training data, but the models here significantly underperform the original dataset in all metrics. Is there any reason why the generative models are so far off compared to their training data?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of generating creative, aesthetic, and novel chess puzzles using generative models. The paper first defines numerical metrics for measuring these unique aspects of chess puzzles, then benchmarks multiple generative schemes, trained on a collection of chess game datasets. The author then adapts the autoregressive transformer with RLHF-PPO based fine-tuning using reward functions designed to maximize realism, novelty, and counter-intuitiveness in the output. From multiple numerical experiments, ablation studies, and human evaluation, the author showcases the effectiveness of GenAI and RL techniques for chess puzzle generation and provides a booklet of generated puzzles as an end product.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n- Well-written paper with tight structure and clear logic flow\n- clear quantification of various measurement with articulated intuitions (such as quantification of puzzle uniqueness, novelty and counter intuitiveness) and solves unique challenge for chess puzzle generation\n- meticulously designed reward with experiment demonstrate appropriate of design choices\n- demonstration of interesting property such as emergent of aesthetics from AR and RL backbones\n\n### Weakness\n\n- the work mainly used well established generative technique (AR, diffusion, RLHF-PPO), therefore the contribution largely lies in engineering choices specifically for chess puzzle generation.\n\nQUESTIONS:\n- Graph on line 122 need minor fixing\n- 3.3 and 3.4 might need to be introduced in chronological order in section 3 overview paragraph\n\nLIMITATIONS:\n- Despite successfully generating puzzles that are evaluated favorably in both numerical experiment and human evaluation, the RL model still requires significant design effort to ensure good convergence.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a method for generating creative chess puzzles using a transformer model enhanced with reinforcement learning. By defining and quantifying creativity through uniqueness, novelty, counter-intuitiveness, and aesthetics, the authors design reward functions based on chess engine search statistics. Their RL approach significantly increases the generation of counter-intuitive puzzles while preserving aesthetic qualities. Human evaluations show that the generated puzzles rival or surpass human-composed ones in creativity and engagement.

STRENGTHS AND WEAKNESSES:
S1) This paper demonstrates high originality, introducing novel problem formulations and a metric that quantifies creativity in chess puzzles. It also has a good motivation as AI's role in creative domain is an important area.

W1) While the approach is compelling, the study is heavily domain specific, relying on chess engines, Lichess data, and hand crafted puzzle heuristics, which raises questions about the generalizability of the framework.

QUESTIONS:
Q1) Typo in Section 2 "(latent diffusion [60, LDMs] )"?

Q2) In Table 1, the reported precision scores (e.g., 0.747504, 0.731198, 0.740115) have 6 decimal places, which is unnecessarily precise. Consider using only 4 dp?

Q3) A discussion on potential transfer to other creative domains or a proof of concept in a second domain would strengthen the paper's broader impact?

Q4) The Golden Set used to tune the counter-intuitiveness reward contains fewer than 100 samples. Would it  limit the robustness and generalizability of the learned scoring function? Will expanding this dataset or incorporating crowd-sourced annotations could improve the reliability of the reward design?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02