Title: Reviving DSP for Advanced Theorem Proving in the Era of Reasoning Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Theorem proving

Large Reasoning Models

Neuro-symbolic

Lean
Abstract: Recent advancements, such as DeepSeek-Prover-V2-671B and Kimina-Prover-Preview-72B, demonstrate a prevailing trend in leveraging reinforcement learning (RL)-based large-scale training for automated theorem proving. Surprisingly, we discover that even without any training, careful neuro-symbolic coordination of existing off-the-shelf reasoning models and tactic step provers can achieve comparable performance. This paper introduces DSP+, an improved version of the Draft, Sketch, and Prove framework, featuring a fine-grained and integrated neuro-symbolic enhancement for each phase: (1) In the draft phase, we prompt reasoning models to generate concise natural-language subgoals to benefit the sketch phase, removing thinking tokens and references to human-written proofs; (2) In the sketch phase, subgoals are autoformalized with hypotheses to benefit the proving phase, and sketch lines containing syntactic errors are masked according to predefined rules; (3) In the proving phase, we tightly integrate symbolic search methods like Aesop with step provers to establish proofs for the sketch subgoals. Experimental results show that, without any additional model training or fine-tuning, DSP+ solves 80.7%, 32.8%, and 24 out of 644 problems from miniF2F, ProofNet, and PutnamBench, respectively, while requiring fewer budgets compared to state-of-the-arts. DSP+ proves imo_2019_p1, an IMO problem in miniF2F that is not solved by any prior work. Additionally, DSP+ generates proof patterns comprehensible by human experts, facilitating the identification of formalization errors; For example, eight wrongly formalized statements in miniF2F are discovered. Our results highlight the potential of classical reasoning patterns besides the RL-based training. All components will be open-sourced.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits a technique called Draft, Sketch, Prove (DSP) on the performance of formal theorem proving with more advanced reasoning models. The authors identify that the original DSP framework has three independent phases that do not coordinate with each other. The authors propose DSP+ by tightly connecting the phases. Specifically, the draft phase leverages advanced reasoning models like DeepSeek-R1 while still making the response concise, the sketch phase checks for syntactically incorrect statements and repairs them and finally prove phase uses a more advanced BFS-prover. The results show that DSP+ attains strong performance close to SOTA DeepSeek Provers while using fewer inference tokens.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper revisits baselines that benefit from more advanced LLMs. This is crucial since baselines should be kept up to date in order to be fairly compared.\n- The method does not require additional training, which is easy to adopt for practitioners.\n\nWeaknesses\n- Although the number of inference tokens is fewer than some baselines, the framework has a sequential workflow with different models which could make wall-clock time or resource requirement larger than some baselines.\n- More weaknesses can be found in the Questions section.\n\nQUESTIONS:\n1. Is it possible to integrate the techniques of DSP+ on top of the best models for proving such as DeepSeek-Prover-V2 671B and see if new SOTA results can be established?\n2. For the ablation of components, how does \"Draft + Sketch\" prove theorems? Is it just with the common tactics?\n3. In Figure 4, it seems that the No Format draft mode is better than Concise Steps, why not use this as the default and report main results with this setting?\n4. Are authors fully committed to release all code to reproduce the main results? I believe a paper of such type needs to release code in order to benefit the community.\n5. What is the iterative repair process discussed on Line 154?\n\nLIMITATIONS:\nThe limitations section is very helpful and the authors provide various failure mode for their framework.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper designs a novel formal theorem prover by advancing the Draft, Sketch and Prove framework. The model integrates neural-symbolic tools with LLMs. In contrast to prior works, LLMs are introduced in each phase, and integrated with neuro-symbolic tree search methods via built-in implementation in Lean4. The framework is free of training or finetuning, while achieves comparable performance to frontier theorem provers based on post-hoc training of LLMs. Ablation studies validate the robustness of design, and efficiency w.r.t. inference tokens.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. As the most significant implication, the paper demonstrates that incorporating multiple existing LLMs and step provers into a delicately designed DSP framework can result in a significantly more powerful formal theorem prover. It increases BFS-Prover's accuracy on MiniF2F from 73% to 84%. Notably, the framework is free of post-hoc training of LLMs, while achieving comparable performance to the frontier large-scale whole-proof generator based on RL. The training-free feature is particularly valuable given the scarcity of formal proof samples.\n\n2. The proposed method sharply improves over previous DSP frameworks (39%-84%). The improvement is mostly attributed to the prove stage, which features a novel built-in integration of LLM-based step prover with neuro-symbolic tactic tree searcher. \n\n3. Sufficient ablation studies validate the synergy of all three component (DSP). The framework is flexible and modular, while being robust to the choice of LLMs for each phase.\n\n### Weaknesses\n1. The paper is not absolutely free of model training, since the step prover is post-trained from LLMs. However, the performance boost over  the step prover is significant enough.\n\nQUESTIONS:\n1. Since the most advanced formal prover generates whole proof, is it possible to integrate whole proof generators to the Prove phase, to replace or complement the BFS-prover?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper revisits the Draft-Sketch-Prove approach to formal theorem proving in the lean language. That is: given a lean statement we wish to prove we 1) prompt a model first to make a natural language proof draft, 2) prompt a model to extract formal goals conditional on this draft (which is roughly formatted as a partially completed lean proof with placeholder statements that need to be filled in) and 3) we use a llm + search algorithm to fill in the missing details of the previous step. This is all prior work - the current paper extends this with various improvements, most notably 1) using reasoning model for the draft step, 2) better prompting and post processing of the sketch and 3) integrating with a tree search method (aesop) rather than the simpler heuristic “sledgehammer” algorithm used in the original DSP paper.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe overall problem is important. The experimental results are strong, only beaten out by the bespoke and super strong RL fine tuned DeepSeek Prover. The authors appear to be very talented at making the LLM pipeline really shine.\n\nWeaknesses\n\nThe paper has relatively little technical novelty, but is rather a well executed application of lots of small known ideas. It presents a nice engineering effort, but this may not be appropriate for the NeurIPS main conference, and may be better suited to a more practical workshop track or so.\n\nThe presentation somewhat obscures the actual novelty in the paper. For example, the authors repeat the phrase “fine grained and integrated neuro symbolic enhancement” as a contribution, but this is rather vague. On line 64 the explanation for the good results is also rather vague.\n\nThe contributions such as certain prompt tweaks are somewhat tied to the specific LLM and not especially valuable scientifically.\n\nQUESTIONS:\nQ1) Can you please clarify the numbers on the rhs of figure 1? Why the + ? Why the 1024 vs 8192? Make the xB explicit as params in billions. Thanks!\nQ2) Can you please list in a precise and unambiguous manner what the contributions are? My guess is that it is a) the use of chain of thought, b) using a newer and better base LLM, c) some heuristics to clean the sketches, d) prompting the model to make a concise sketch, e) masking generated lines of the sketch that produce lean errors, f) explicit specification in the subgoal hypothesis of the draft step.\nQ2b) What does f) in Q2 mean?\nQ3) Might # tokens be a better x axis for Fig 3?\nQ4) Is the legend clear? I could not quite figure out exactly what was presented.\n\nMinor points\nLine 17 fewer budgets -> reword + state of the arts -> reword\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-03"}