Title: APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: formal reasoning

automated theorem proving

lean prover

ai for math
Abstract: Formal reasoning and automated theorem proving constitute a challenging subfield of machine learning, in which machines are tasked with proving mathematical theorems using formal languages like Lean. A formal verification system can check whether a formal proof is correct or not almost instantaneously, but generating a completely correct formal proof with large language models (LLMs) remains a formidable task. The usual approach in the literature is to prompt the LLM many times (up to several thousands) until one of the generated proofs passes the verification system. In this work, we present APOLLO (**A**utomated **P**r**O**of repair via**L**LM and **L**ean c**O**llaboration), a modular, model‑agnostic agentic framework that combines the strengths of the Lean compiler with an LLM’s reasoning abilities to achieve better proof‐generation results at a low token and sampling budgets. _Apollo_ directs a fully automated process in which the LLM generates proofs for theorems, a set of agents analyze the proofs, fix the syntax errors, identify the mistakes in the proofs using Lean, isolate failing sub‑lemmas, utilize automated solvers, and invoke an LLM on each remaining goal with a low top‑K budget. The repaired sub‑proofs are recombined and reverified, iterating up to a user‑controlled maximum number of attempts. On the miniF2F benchmark, we establish a new state‑of‑the‑art accuracy of 84.9% among sub 8B‑parameter models (as of August 2025) while keeping the sampling budget below one hundred. Moreover, _Apollo_ raises the state‑of‑the‑art accuracy for Goedel‑Prover‑SFT to 65.6% while cutting sample complexity from 25,600 to a few hundred. General‑purpose models (o3‑mini, o4‑mini) jump from 3–7% to over 40% accuracy. Our results demonstrate that targeted, compiler‑guided repair of LLM outputs yields dramatic gains in both efficiency and correctness, suggesting a general paradigm for scalable automated theorem proving. The codebase is available at https://github.com/aziksh-ospanov/APOLLO


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes APOLLO, a modular, agentic pipeline that incorporates signals from lean compiler with LLM prover to achieve better performance in mathematical theorem proving. The proposed system includes a set of agents to analyze the proofs, fix the syntax errors, identify the mistakes, decompose failing lemmas and utilize automated solvers to boost the success rate of the whole proofs generated by an LLM prover. With APOLLO, the authors achieve 75% accuracy on miniF2F test set and at the same time improve the LLM prover’s sample complexity. They also demonstrate its wide applicability to general-purpose reasoning models such as o3 and o4.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written, and the authors have shown a good amount of improvement with their proposed APOLLO. However, there is still space to improve:\n\n1.\tThe authors claim that they achieve state-of-the-art accuracy, but Deepseek-prover-V2-7B has an accuracy of 82%. I suggest the authors include this reference.\n\n2.\tI am not sure if the comparison of sampling budget is fair. Based on my understanding, the mean number of samples for APOLLO is computed as the number of function calls until it successfully proves the theorem or reaches a maximum recursion depth. Then to make fair comparison, I think for pass@k, the authors should also report the mean of number of samples until it FIRST samples a correct proof. I believe this number will be lower than the sample budget reported in the table.\n\n3.\tThe proposed method APOLLO acts like a search tree. As a result, it may be important to include comparisons with tree-search methods such as Monte Carlo Tree Search and check how much APOLLO improves over those tree-search methods.\n\nQUESTIONS:\nI will update the score if the authors address some/all the weakness points I listed in the last section. Additionally, I have the following questions.\n\n1.\tIn Fig. 3, the curve for APOLLO is cutoff at a relatively low sample budget. It’s important to understand how the accuracy scales with larger sample budge. Does APOLLO continue to prove more theorems or does it plateau and thus cannot further improve much more than the base model.\n\n2.\tIn Fig. 4, is the data included in the figure conditioned on the proof being correct? I think it only makes sense if the authors show the results on correct proofs only. Making the wrong proofs longer is not interesting or important.\n\n3.\tHow does the hyperparameter r transfer to sample budget? I suggest including a plot showing the relationship between r and sample budget.\n\n4.\tIs it possible to do pass@N on top of APOLLO? Is there a tradeoff between increasing r and increasing passes?\n\nLIMITATIONS:\nThe authors have included a limitations section in the appendix. I agree that not considering step-level LLM prover is a main limitation of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an inference-time workflow for theorem proving with large language models (LLMs) in Lean, which comprises a series of specialized tools that fix up common syntax errors in LLM-generated code, remove parts with errors, apply a suite of Lean's standard proof automation tools and otherwise recurse by extracting the remaining proof gaps as standalone proving problems for the LLM.\n\nCompared to the baseline of $k$ independent samples from the prover LLM, this post-processing and recursive repair significantly reduces cost / outperforms the baseline on the standard MiniF2F benchmark when comparing at the same amount of LLM calls. The method can be applied to any prover model and is both used with Lean-finetuned models like Goedel-Prover and general-purpose reasoning models such as o3-mini.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The post-processing and recursive repair appear extremely efficient at catching and fixing minor mistakes in the code, producing strong evaluation numbers on a variety of models.\n2. The different post-processing stages are clearly described and the paper is easy to understand.\n\n**Weaknesses:**\n1. The method is not compared against other inference-time methods such as putting the model into a multi-turn agentic loop where it reads in error messages and can decide freely how to solve the issue (by replacing a block, using automation tools, reworking a line etc). I understand that this is in part due to the fact that whole-proof generation models such as Goedel-Prover are not able to follow such instructions, but o3-mini and o4-mini certainly could.\n2. The comparison based on LLM calls is fair as long as GPU compute is the primary limiting factor. Given that in Lean proving setups, CPU execution is an increasingly important bottleneck, it would be important to see Lean execution runtimes, too, for getting the full picture.\n3. There are no ablations on the different parts of the post-processing pipeline. At least, it would be useful to see average numbers on how often which tool triggers with which model. I would be surprised, for instance, if the Syntax Refiner often triggers for the Lean4-specific models. Moreover: How many gaps can be solved with the Lean automation? How many recursive calls are being issued on average? (Table 1 lists the average total number of calls per problem but doesn't state from how many independent attempts these stem.)\n4. The method proposes ad-hoc fixes to rather superficial problems of current LLMs for Lean (confusion between Lean 3 and Lean 4 among general-purpose models; incomplete knowledge of all automation tactics), it is conceivable that the method will not be relevant for a very long time.\n\nQUESTIONS:\nsee weaknesses 1-3: comparison to agentic setup? CPU times? ablations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Apollo, an automated modular pipeline that enhances large language model (LLM)-based theorem proving by integrating LLM outputs with feedback from the formal verification system (Lean4). Rather than naive resampling, Apollo iteratively identifies and repairs errors in LLM-generated proofs through a sequence of specialized modules. Specifically, the pipeline includes a Syntax Refiner that corrects superficial syntax issues; a module that replaces faulty sub-proofs with Lean's proof placeholders; an Auto Solver that applies built-in tactics to resolve sub-goals; and a recursive repair mechanism that re-invokes the LLM on remaining goals using Lean's feedback. Finally, the Proof Assembler re-integrates the repaired components and verifies the complete proof. The proposed method demonstrates strong empirical performance on the miniF2F benchmark across different base models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is well-organized and easy to follow. The proposed method achieves strong performance on miniF2F benchmark with reduced sampling. The results analysis is thorough. The agentic pipeline is practical and aligns with human-like proof workflows.\n\nWeaknesses:\nComparisons lack consistency in model and token budget; more datasets are needed. The method focuses on workflow design rather than algorithm or architecture-level innovation. No comparison with standard depth-first or best-first search methods. Modules rely on hand-crafted rules, limiting generalizability. Key artifacts (README, requirements, data) are missing from the code release.\n\nQUESTIONS:\n1) For a more rigorous evaluation, I suggest fixing the base model and comparing Apollo with alternative strategies under the same sampling budget. In particular, using the number of tokens as the budget unit rather than the number of sampled completions, would more accurately reflect computational cost. Additionally, evaluating the method on a broader set of benchmarks would help demonstrate the generalizability of the approach.\n2) The core idea of identifying erroneous segments and selectively repairing them is conceptually straightforward. The contribution primarily lies in the agentic system design and modular workflow. Clarifying how Apollo differs from standard modular repair workflows or existing agentic frameworks would help assess its conceptual contribution.\n3) The paper would benefit from a direct comparison with standard depth-first or best-first search strategies. In tree search, invalid branches are pruned early, potentially reducing token usage by avoiding downstream propagation of errors. In contrast, Apollo must retrospectively correct affected steps, which could be less efficient.\n4) The descriptions of Apollo's modules suggest that they are LLM-based agents, but no prompt templates or examples are provided. Upon reviewing the source code, it appears that the modules are rule-based programs. If so, this raises concerns about scalability and generalizability: manually crafting rules for different failure modes is unlikely to scale across programming languages (e.g., from Lean to Prolog, even across different Lean versions), different LLMs (some correction rules are model-specific according to the code comments), or datasets. Clarifying this aspect and discussing its limitations would strengthen the paper.\n5) Several components necessary for reproducibility are missing from the released source code, including a README, installation requirements, and datasets or download instructions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}