Title: SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

LLMs Reasoning

Reinforcement Learning

Curriculum Learning
Abstract: How to design reinforcement learning (RL) tasks that effectively unleash the reasoning capability of large language models (LLMs) remains an open question. Existing RL tasks (e.g., math, programming, and constructing reasoning tasks) suffer from three key limitations: (1) Scalability. They rely heavily on human annotation or expensive LLM synthesis to generate sufficient training data. (2) Verifiability. LLMs' outputs are hard to verify automatically and reliably. (3) Controllable Difficulty. Most tasks lack fine-grained difficulty control, making it hard to train LLMs to develop reasoning ability from easy to hard.

To address these limitations, we propose Saturn, a SAT-based RL framework that uses Boolean Satisfiability (SAT) problems to train and evaluate LLMs reasoning. Saturn enables scalable task construction, rule-based verification, and precise difficulty control. Saturn designs a curriculum learning pipeline that continuously improves LLMs' reasoning capability by constructing SAT tasks of increasing difficulty and training LLMs from easy to hard. To ensure stable training, we design a principled mechanism to control difficulty transitions.

We introduce Saturn-2.6k, a dataset of 2,660 SAT problems with varying difficulty. It supports the evaluation of how LLM reasoning changes with problem difficulty. We apply Saturn to DeepSeek-R1-Distill-Qwen and obtain Saturn-1.5B and Saturn-7B. We achieve several notable results:
(1) On SAT problems, Saturn-1.5B and Saturn-7B achieve average pass@3 improvements of +14.0 and +28.1, respectively.
(2) On math and programming tasks, Saturn-1.5B and Saturn-7B improve average scores by +4.9 and +1.8 on benchmarks (e.g., AIME, LiveCodeBench).
(3) Compared to the state-of-the-art (SOTA) approach in constructing RL tasks, Saturn achieves further improvements of +8.8\%.
We release the source code, data, and models to support future research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a reinforcement learning loop that trains large language models (LLMs) to solve SAT queries. The reward scheme is simple; the SAT query construction scheme, and especially the task difficulty estimation, is well-motivated and not obvious (in a good way). After training, DeepSeek-R1-Distill-Qwen-1.5B and 7B improve performance on SAT queries and, more surprisingly, on other tasks, like math and programming. The authors suggest that this generalization is due to improved self-verification capabilities of LLMs + SATURN.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper is clear, well-written, and well typeset.\n- The task difficulty estimation is well-motivated and works empirically. The task construction method follows nicely from the difficulty estimation metric.\n- The evaluation is solid.\n- SATURN's ability to generalize beyond SAT is interesting and significant.\n\n## Weaknesses\n- Logic-RL is quite similar, although, of course, there is value in SATURN's data generation methodology.\n- While the task difficulty estimation is well-motivated and empirically effective, the paper would benefit from an argument for why this metric is the best choice. Why does something simpler not work just as well? Are all parts of the estimation method necessary? A comparison to alternatives or an ablation study for the metric specifically may be appropriate, given that this is the paper's strongest contribution.\n- The syntax constraint (e.g., \"\\boxed{}\") is never really explained. It was not until the appendix that it became clear that the LLM must output a satisfying model and not just SAT/UNSAT.\n- It is not obvious outside of the appendix that all queries are constructed to be satisfiable. The LLM is acting as a model finder, not as a SAT solver. This makes sense—you can avoid calling a SAT solver on all your generated queries to get the ground truth—but it should be explicit.\n\nQUESTIONS:\n- Which aspects of the difficulty estimation metric are most crucial?\n- (Optional) Is the author's metric good at estimating difficulty for CDCL SAT solvers?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SATURN, a curriculum-based reinforcement learning (RL) framework that uses SAT problems to train and evaluate large language models (LLMs) on reasoning tasks.  A dataset named SATURN-2.6k is introduced to benchmark performance over varying difficulty levels. The authors apply their approach to LLMs like DeepSeek-R1-Distill-Qwen and show improvements on SAT solving tasks and modest improvements on math and programming benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper introduces SATURN, a curriculum-based reinforcement learning framework that leverages SAT problems to train and evaluate LLMs on complex reasoning tasks.\n2. The authors provide a benchmark dataset, SATURN-2.6k, which enables evaluation across varying difficulty levels.\n3. The models fine-tuned with SATURN show improvements on SAT tasks and some generalization to math and code domains.\n\n**Weaknesses**\n1. **Overly simple SAT datasets**:\n- The SAT problems used in training (e.g., 3-SAT with clause-to-variable ratios far from the phase transition) are syntactically and semantically simple. \n- Many tasks can likely be solved by pattern matching or shallow heuristics rather than logical reasoning, especially with SAT instances constructed to be trivially satisfiable. \n- Despite the theoretical hardness of SAT, the practical instances used here do not reflect the complexity needed to truly test or build reasoning capabilities.\n\n2. **Whether SAT is suitable for reasoning skills**:\n- The paper claims that solving SAT via RL builds general reasoning skills. However, SAT solving is fundamentally search-based and benefits from symbolic constraint satisfaction strategies — not the kind of causal reasoning or abstraction often required in math or programming.\n\n- The assumption that SAT instances naturally induce verification and self-reflection is speculative and lacks empirical justification or ablation analysis.\n\n3. **Lack of RL Analysis**:\nThe contribution is based on the use of RL, but the role of RL is under-analyzed. There is no comparison to standard supervised fine-tuning on the same tasks, and it remains unclear whether RL offers any unique benefit over direct optimization on pass@k.\n\n4. **Modest improvements** : Improvements on downstream tasks (math, code) are modest and possibly attributable to performance variance introduced by training or domain transfer from additional training data, not necessarily from enhanced reasoning from SAT tasks.\n\nQUESTIONS:\n1. How do you ensure that the SAT tasks used in SATURN actually require multi-step logical reasoning rather than simple token pattern recognition or template-based decoding?\n\n2. Can you provide a comparative study of SATURN vs. supervised fine-tuning on the same SAT dataset to justify the use of reinforcement learning?\n\n3. Is there any evidence that training on richer formal logic problems (e.g., QBF, theorem proving) would provide stronger reasoning benefits than SAT? If so, why was SAT selected despite its limited expressiveness?\n\n4. How does the reasoning learned from SAT generalize to tasks involving semantic understanding or abstraction, which SAT does not cover?\n\n5. Have you tested model behavior on harder SAT instances near the phase transition (e.g., α ≈ 4.2 for 3-SAT) with much more variables where symbolic reasoning becomes nontrivial?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SATURN, a novel method for improving the reasoning abilities of large language models (LLMs) by training them on a curriculum of Boolean satisfiability (SAT) problems. The core idea is to leverage the structured, symbolic nature of SAT problems to enhance the underlying reasoning skills of LLMs. The authors propose a reinforcement learning framework where the LLM is trained on a progressively more difficult sequence of SAT problems. The difficulty of these problems is determined by a novel metric that estimates the size of the search space. A key contribution is the demonstration that training on this curriculum of SAT problems improves not only the model's ability to solve SAT problems but also its performance on a range of other reasoning tasks, including mathematical reasoning, code generation, and logical puzzles. The authors show that SATURN-trained models outperform baseline models of similar size and even some much larger models on several benchmarks. The paper also provides an in-depth analysis of the model's behavior, showing that the training process enhances the LLM's ability to perform self-verification and to generate more structured and logical reasoning step\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Clarity: The paper is well-written and easy to follow. The problem statement is clear, the SATURN framework is explained in detail, and the research questions are well-defined. The authors have done an excellent job of presenting their work in a way that is accessible to a broad audience, even those who may not be experts in SAT solvers or reinforcement learning.\n\n- Originality: The paper presents a novel and significant contribution by using SAT problems as a training ground for improving the general reasoning abilities of LLMs. This is a departure from the more common approach of training on vast amounts of text data, and it opens up a new avenue for research into how to best instill abstract reasoning skills in LLMs. \n\n- Quality: The experimental design is sound, and the authors have conducted a thorough set of experiments to validate their claims. The inclusion of extensive ablation studies in the appendix strengthens the paper by showing the impact of different components of the SATURN framework. The authors have also released their code, data, and models, which is a significant contribution to the community and allows for the reproduction and extension of their work.\n\n##  Weaknesses\n\n- Transferability: A key weakness is the limited transferability of the reasoning skills learned from SAT problems to more knowledge-intensive domains. While the paper shows improvements on a range of reasoning tasks, the improvements are most significant on tasks that are structurally similar to SAT problems. This suggests that the model may be learning to solve a particular type of reasoning problem, rather than developing a more general reasoning ability.\n\n- Justification for Difficulty Metric: The paper proposes a novel metric for estimating the difficulty of SAT problems, but the justification for this particular metric is not entirely convincing. While the authors provide some intuition and empirical validation, it is not clear why this metric is superior to other possible metrics. A more thorough discussion of the design choices behind the difficulty metric would strengthen the paper.\n\nQUESTIONS:\n1. Could you provide more intuition or empirical validation for the chosen difficulty metric? Have you explored other possible metrics, and if so, how did they compare? A more in-depth justification for the chosen metric would help to clarify why it is particularly well-suited for this task.\n\n2. The paper provides some excellent case studies showing how SATURN enhances the model's ability to perform self-verification. However, it would be helpful to have a more detailed analysis of the specific reasoning mechanisms that are being improved. For example, is the model learning to better understand logical connectives, or is it learning to perform a more systematic search? A more fine-grained analysis of the model's reasoning process would provide valuable insights into how SATURN works and helps.\n\n3. The paper notes that the performance of the model plateaus after a certain point in the curriculum. Have you investigated any strategies for overcoming this plateau? For example, would it be possible to create a hybrid curriculum that integrates SAT problems with other types of reasoning tasks? This could help to further improve the model's general reasoning abilities.\n\nLIMITATIONS:\nControllable Difficulty seems to make sense. However, it should be discussed with different kinds of metrics that define the difficulty.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose SATURN, a RL framework intended to improve the reasoning capabilities of LLMs with SAT problems. Towards this end, they construct SAT problems (SATURN 2.6k dataset) of increasing difficulty to create a curriculum of reasoning problems of increasing difficulty. This is then interleaved with a LLM RL training loop (using GRPO) to improve LLM reasoning ability. They apply the framework to DeepSeek-R1-Distill-Qwen-1.5B and 7B, and demonstrate that the application of SATURN can improve the performance of the base LLMs on SAT problems. They also demonstrate that it can generalize to Math and Programming questions as well as complement other approaches to improve reasoning in LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper works on an important problem—new methods to improve reasoning capabilities in LLMs. The proposed approach of combining curriculum learning and RL is natural. The study also is ablated (e.g., testing the efficacy of mixed training vs. curriculum) and comprehensive (e.g., testing transferability of SAT training to math/programming). Lastly, I found it promising that training on SAT problems could improve reasoning in other domains.\n\nWhile I believe the methods and study are sound, I found myself wishing for more explanation as to why such a method should work and for a deeper analysis of the reasoning chains produced as a result of training with SATURN. For instance, is it because training on SAT problems causes the model to take smaller reasoning steps, enumerate more cases, or something else, that also enables it to generalize better to math/coding problems?\n\nQUESTIONS:\n1. Do the authors have an explanation as to why such a method, even though it is trained on SAT, should generalize to other settings? For instance, does it encourage the LLMs to consider more cases and/or negations that help it reason more effectively in math/coding domains?\n2. Have you tested SAT problems that are not given as 3-SAT problems? I wonder about data leakage as the models have likely picked up instances of 3-SAT problems during pre-training. However, I would imagine that moderate-size SAT problems would be less prevalent on the internet since they are too large to be instructive or solvable without being converted into 3-SAT.\n3. Why does Saturn-7B perform worse on AIME24/25 compared to DeepSeek-R1-Distill-Qwen-7B?\n4. What does the lengthening phenomenon look like for a curriculum approach vs. a mixed training approach? While I appreciate the difficulty of performing all these ablations, I am curious, since curriculum learning adds another layer of complexity to the post-training step. Does mixed-training result in shorter responses compared to a curriculum approach?\n\nLIMITATIONS:\nThe authors correctly highlight limitations in Section 4.1.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}