Title: Enigmata:  Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reasoning Model

Large Language Model

Reinforcement Learning

Logical Reasoning
Abstract: Large Language Models (LLMs), such as OpenAI’s o1 and DeepSeek’s R1, excel at advanced reasoning tasks like math and coding via Reinforcement Learning with Verifiable Rewards (RLVR), but still struggle with puzzles solvable by humans without domain knowledge. We introduce ENIGMATA, the first comprehensive suite tailored for improving LLMs with puzzle reasoning skills. It includes 36 tasks across 7 categories, each with: 1) a generator that produces unlimited examples with controllable difficulty, and 2) a rule-based verifier for automatic evaluation. This generator-verifier design supports scalable, multi-task RL training, fine-grained analysis, and seamless RLVR integration. We further propose ENIGMATA-Eval, a rigorous benchmark, and develop optimized multi-task RLVR strategies. Our trained model, Qwen2.5-32B-ENIGMATA, consistently surpasses o3-mini-high and o1 on the puzzle reasoning benchmarks like ENIGMATA-Eval, ARC-AGI (32.8%), and ARC-AGI 2 (0.6%). It also generalizes well to out-of-domain puzzle benchmarks and mathematical reasoning, with little multi-tasking trade-off. When trained on larger models like Seed1.5-Thinking (20B activated parameters and 200B total parameters), puzzle data from ENIGMATA further boosts SoTA performance on advanced math and STEM reasoning tasks such as AIME (2024-2025), BeyondAIME and GPQA (Diamond), showing nice generalization benefits of ENIGMATA. This work offers a unified, controllable framework for advancing logical reasoning in LLMs. Project page: https://seed-enigmata.github.io.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ENIGMATA, designed to enhance Large Language Models' logical reasoning capabilities, particularly for solving puzzles. The ENIGMATA suite consists of three main components: ENIGMATA-Data, a dataset of 36 diverse puzzle tasks across seven categories, each equipped with an automated generator for scalable, difficulty-controlled examples and a rule-based verifier for automatic evaluation; ENIGMATA-Eval, a benchmark derived from this data; and ENIGMATA-Model, a training recipe combining rejection fine-tuning and multi-task Reinforcement Learning with Verifiable Rewards (RLVR). The authors demonstrate that their trained model, Qwen2.5-32B-ENIGMATA, achieves good performance on ENIGMATA-Eval, ARC-AGI, and out-of-domain puzzle benchmarks, while maintaining strong mathematical reasoning abilities. The work aims to provide a unified, controllable framework for advancing logical reasoning in LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\nENIGMATA-Data is featuring 36 distinct puzzle tasks across 7 broad categories. \n\nThe proposed ENIGMATA-Model recipe, which combines rejection fine-tuning (RFT) to establish reasoning patterns and multi-task Reinforcement Learning (RL) for general reasoning skills, is good. \n\nThe trained Qwen2.5-32B-ENIGMATA model demonstrates good performance.\n\nThe model seems to exhibit good generalization to out-of-domain puzzle benchmarks (KOR-Bench) and maintain good performance in mathematical reasoning (AIME 24).\n\n\n\n[Weaknesses]\nThe paper admits that 6 of the 36 tasks draw from \"fixed pools\" and some instances \"rely on manually collected and annotated data rather than auto-generation\".  Would this inconsistency undermine the claim of universal scalability and fully automated data generation/verification across the entire ENIGMATA suite?\n\nDespite implementing multi-stage RL to mitigate task conflicts, the paper notes \"more volatile response length under Multi-Stage RL suggests instability in generation\". This indicates that the proposed training strategies might not fully resolve the inherent challenges of simultaneously optimizing for diverse reasoning skills.\n\nThe analysis explicitly states that \"code utilization actually hindered performance on puzzle tasks\" and that \"current models fail to effectively use code without executing it for complex reasoning tasks\" This probably reveals a technical limitation.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new dataset of puzzle tasks for the evaluation of Large Reasoning Models (LRM), along with an accompanying benchmark and training procedure for fine-tuning a model on the task. The dataset consists of 36 puzzle tasks across 7 high-level categories, and most tasks also feature a generator with a scalable difficulty parameter. The authors demonstrate that current state-of-the-art LRMs do not solve the ENIGMATA benchmark and also show that their fine-tuned open-weight model performs comparably to the best closed-weight models on the benchmark. The paper concludes with a deeper analysis of the fine-tuned model, including various ablation studies.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a strong paper. The authors introduce a broad dataset of puzzle tasks and effectively demonstrate that it represents a reasonable challenge for current models. I’m also impressed with the thoroughness of the training procedure for ENIGMATA-Model and the accompanying experiments, which help immensely in understanding the effects of the various decisions / hyperparameters. My sense is that the benchmark could easily prove useful to the larger research community. However, I must also admit that I’m not the most knowledgeable about the various reasoning benchmarks that already exist -- it may be worth the authors providing additional explanation of why the ENIGMATA benchmark is a useful test suite in the context of other (potentially more challenging) evaluations like ARC-AGI 2.\n\nI would also strongly encourage the authors to include some indication of the statistical significance of their experiments. In the checklist, the authors indicate that this information is present in Section 5.3 -- however, this section only concerns the ablation studies and does not mention anything about variance or statistical significance. The tables and figures also appear to be missing error bars / indications of variance. I don’t feel that this critique sinks the paper, given the scale of the other contributions, but I again would urge the authors to address it.\n\nI also encourage the authors to cite this relevant paper on the automatic generation of programming puzzles:\n\n[1] Pourcel, Julien, et al. \"ACES: generating diverse programming puzzles with autotelic language models and semantic descriptors.\" Neurips (2024).\n\nQUESTIONS:\n- How would you propose addressing the low performance of models in the Graph and Sequential categories? Is this something that can be solved with more data?\n\n- What is the main benefit of using the ENIGMATA benchmark over other similar reasoning benchmarks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problems of reinforcement learning from verifiable feedback in the context of logical puzzles. The paper introduces a collection of logical puzzles, called ENIGMATA, for training and evaluating LLMs. ENIGMATA contains 36 logical puzzles across 7 categories in total, where each of the puzzles supports generation of problems with controlled difficulty and rule-based evaluators for reliable evaluation.\n\nUsing ENIGMATA, the paper trains Qwen-32B models which achieves strong performance on logical puzzles, and also sees some generalization to math domains. The paper also provides some analysis such as impacts of training data size and the impacts of SFT training.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe collected data suite, ENIGMATA, contains over 30 puzzles from diverse types. The tasks have some nice properties of being able to generate new examples, configurable difficulty and reliable evaluation.  This collection of data suites can be useful for the research community in the field of RLVR for reasoning.\n\nThe authors also conduct experiments of running RL over the logical puzzles, which shows good in-domain results. The experiments contain multiple settings, which shows some insights on when SFT is necessary.\n\nThe paper is mostly well-written and easy to follow.\n\n## Weakness\n\nThe experiment only contains one model of one family and one scale. It is unsure how the results could generalize across different model families.\n\nThe paper claims AIME as OOD evaluation, though the paper uses some math data for training as well (ln 161). I feel to some extent, this is not completely OOD.\n\nThe paper presents a collection of gams and empirical study on using them for training model, I feel the paper could benefit from more analysis on 1) generalization across games and 2) generalization more realistic settings, like what kind of games could enable generalization on what types of OOD games.\n\nWhile the paper compiles a set of tasks, most of them can be found in existing resources. The paper does not add new things for testing LLMs.\n\nQUESTIONS:\nSee weakness for questions.\n\nLastly I have one practical concern (not affecting the scores), the practical value of this  work maybe overshadowed by some more comprehensive collections of these kinds of tasks, such as reasoning gym (containing over 100 tasks). The paper might consider some analysis to set it apart from similar work.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Enigmata, a comprehensive and scalable benchmark suite aimed at enhancing puzzle reasoning in LLMs. It features 36 tasks across 7 categories, each equipped with a deterministic rule-based verifier, enabling large-scale multi-task training and integration with Reinforcement Learning with Verifiable Rewards (RLVR). The authors also propose Enigmata-Eval as a rigorous evaluation benchmark and develop multi-task RLVR strategies. Their trained model, Qwen2.5-32B-Enigmata, outperforms leading models like o1 and o3-mini-high on several reasoning benchmarks about puzzles and maths.\n\nSTRENGTHS AND WEAKNESSES:\nThe model demonstrates impressive performance across a comprehensive suite of puzzle and math benchmarks, even performing on par with the proprietary LLMs. The writing is clear and accessible, and the method and analysis is presented in a simple yet effective manner. However, the paper may lack key experiments as follows:\n\n**Transferability of Puzzle Training Data Alone**: It’s encouraging to see that combining puzzle and math tasks in the proposed training framework yields strong performance across both domains. However, it remains unclear how effective puzzle training data alone is at supporting generalization, especially given that many puzzles involve complex, long-horizon reasoning and mathematical computation. A deeper investigation into the standalone impact of puzzle data on reasoning capabilities would be valuable. Also, identifying which specific puzzle domains contribute most to transferable skills could guide future efforts in data selection for promoting general reasoning abilities.\n\n**Cross-Domain Transferability in Puzzle Games**: It would also be insightful to explore whether training on one puzzle domain facilitates generalization to others. Since many puzzles share underlying cognitive skills (e.g., constraint satisfaction, pattern recognition), cross-domain transfer is a promising area of inquiry. A data efficiency study, such as identifying the most representative subsets of puzzles for skill acquisition, could further illuminate how to build compact yet powerful training curricula.\n\n**Transfer Across Difficulty Levels**: The dataset spans tasks of varying difficulty: easy, medium, hard. An interesting question is whether models trained only on easy tasks can generalize to harder ones, which was observed in other domains such as SimpleRL (Zeng et al., 2025). Studying this easy-to-hard generalization would provide insight into how task difficulty impacts skill development and transfer.\n\nQUESTIONS:\n**Choice of VC-PPO**: The paper adopts VC-PPO, but it’s unclear why this particular variant was chosen over alternatives such as VC-GRPO or VC-DAPO. A more comprehensive comparison with other common RL algorithms would help clarify whether VC-PPO is especially well-suited for this setting or whether other methods might perform similarly or better.\n\nLIMITATIONS:\nSee Weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}