Title: 3D-Prover: Diversity Driven Theorem Proving With Determinantal Point Processes
Venue: NeurIPS 2025
Accepted: yes
Keywords: theorem proving

proof search

diversity

representation learning
Abstract: A key challenge in automated formal reasoning is the intractable search space, which grows exponentially with the depth of the proof. This branching is caused by the large number of candidate proof tactics which can be applied to a given goal. Nonetheless, many of these tactics are semantically similar or lead to an execution error, wasting valuable resources in both cases. We address the problem of effectively pruning this search, using only synthetic data generated from previous proof attempts. We first demonstrate that it is possible to generate semantically aware tactic representations which capture the effect on the proving environment, likelihood of success, and execution time. We then propose a novel filtering mechanism which leverages these representations to select semantically diverse and high quality tactics, using Determinantal Point Processes. Our approach, 3D-Prover, is designed to be general, and to augment any underlying tactic generator. We demonstrate the effectiveness of 3D-Prover on the miniF2F and LeanDojo benchmarks by augmenting popular open source proving LLMs. We show that our approach leads to an increase in the overall proof rate, as well as a significant improvement in the tactic success
 rate, execution time and diversity. We make our code available at https://github.com/sean-lamont/3D-Prover.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel search algorithm for guiding LLM-based automated (interactive) theorem proving. The core idea is to train a small model to predict the \"dynamics\" of the theorem proving environment, and to then use this model as a signal from which to select a set of diverse and high-quality tactics at each timestep. In order to turn the signal into a distribution over the tactics which incorporates both notions, the authors leverage Deterministic Point Processes (DPP). The experiments are somewhat limited in scope, but suggest this method can lead to small (but meaningful, if statistically significant) improvements over existing methods. As the authors point out, while improving search is unlikely to ever yield as large of an improvement as improving the policy model, progress in this area is still meaningful as it is complementary to works seeking to improve the base model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea is, as far as I know, novel. Leveraging DPPs in particular is a fun and interesting approach to balanced diversity and quality.\n- The paper is well-written, with clear figures and methodology and no unnecessary formalizations (that is: the math that is in the paper serves its purpose).\n- The limitations of the study, including limited (but non-zero) impact on downstream performance and the limited scope of the experiments, are both acknowledged and discussed extensively in the paper.\n- I appreciate that the authors quantitatively evaluated the overhead that this search procedure introduces. In particular, a priori I expected that running the pipeline on each generated tactic would exceed the time it takes to simply execute that tactic, however this appears not to be the case as per Appendix G. This makes me much more optimistic about the method's practical relevance.\n- Work in this area is complementary to a lot of the other stuff that is happening in NLP right now, which from the point of view of interactive theorem proving largely boils down to improving the base (policy) model.\n\nWeaknesses:\n- While it is good that the authors acknowledge that a lack of computational resources casts some doubt on the signifance and generalizability of their findings, it is hard to quantify how big this effect is. As a result, future work would need to confirm that this methodology indeed works as well as these initial findings would suggest.\n\nQUESTIONS:\nSuggestions:\n- In the introduction there is a typo: \"It applications\"\n- On page, \"columns b_i\" should likely be \"rows b_i\" (or you have accidentally transposed your matrix dimensions)\n\nQuestions:\n- In table 3, you state that you report the mean += one std dev over 2 runs for pass@1. Am I understanding correctly that you literally computed the arithmetic mean and sample std dev over 2 samples? That is not a meaningful metric, if so. If your sample size is small it would be much better to report (min, max) ranges, or a confidence interval which does not rely on asymptotic normality (eg based on Hoeffding's inequality).\n- In general, do you have any supporting (statistical) evidence that suggests your findings are even statistically significant? I am sympathetic to the difficulties of doing research in this field without access to large computational resources, but as it stands I am not entirely convinced that the single-digit improvements reported downstream are significant.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-20"}

comments: SUMMARY:
The paper addresses a critical issue in the intersection of LLMs and Lean: the semantic redundancy among multiple strategies generated by LLMs, which significantly increases unnecessary computational overhead. To tackle this, the authors propose 3D-Prover, a strategy selection algorithm for automated theorem proving. Its objective is to select a subset of candidate strategies that are both high-quality and diverse, thereby improving proof efficiency and success rates.

STRENGTHS AND WEAKNESSES:
Strengths：
The content of the paper is comprehensive and the presentation is relatively clear. It addresses an important problem in the field.

Weaknesses：
The proposed method relies heavily on black-box models, such as the encoder and predictor, whose internal mechanisms are not transparent to us. Therefore, I have some doubts about the generalizability of this approach. Methods that are effective on the small miniF2F dataset may not necessarily perform well on other tasks.

QUESTIONS:
1.If the predictor ( P ) does not predict the execution time, how much would it affect the performance of the algorithm?
2.Did the encoder, predictor, and decoder in the paper reuse the internal architecture of the LLM, or were they additional components introduced by the authors?
3.My understanding is that this method has two advantages: on the one hand, it ensures the diversity of the subsets and reduces redundancy; on the other hand, it allows for the selection of higher-quality tactics.
I am not sure if my understanding is correct.

LIMITATIONS:
As mentioned by the authors in the paper.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
The paper introduces 3D-Prover, a novel approach to automated theorem proving that leverages Determinantal Point Processes (DPPs) to filter semantically diverse and high-quality tactics. Evaluated on miniF2F and LeanDojo, 3D-Prover enhances models like ReProver and InternLM2.5-Step-Prover, boosting proof success rates, tactic quality, speed, and diversity.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The use of DPPs for filtering tactics in theorem proving is innovative and addresses the challenge of exponential search space growth.
2. The paper provides extensive empirical results, including ablations and comparisons with baselines, demonstrating consistent improvements across multiple benchmarks.
3. The method has a solid mathematical foundation.

Weaknesses:
1. The experimental results are not sufficiently pronounced, and the latest SOTA models (e.g., BFS-Prover) were not employed in the experiments.
2. There are certain computational cost considerations that the authors need to further address.

QUESTIONS:
1. Could you provide an analysis of the computational cost associated with the proposed method?

2. Table 11 shows that 3D-Prover tends to generate longer proofs. Does this raise concerns about potential redundant proofs?

3. The ablation study appears insufficiently evaluated. Tables 5, 6, and 7 only conduct experiments on two specified hyperparameter settings. Are more hyperparameter ablation studies needed?

4. Can neural network embeddings effectively characterize proof states? To my knowledge, they are primarily used for shallow semantics.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces 3D-Prover, a novel approach to enhance automated theorem proving by addressing the challenge of intractable search spaces in proof trees. It proposes a filtering mechanism using Determinantal Point Processes (DPPs) to select semantically diverse and high-quality tactics, leveraging tactic representations that capture their effect on the proving environment, likelihood of success, and execution time. These representations are learned from synthetic data generated from previous proof attempts. The method is designed to augment existing tactic generators and is evaluated on the miniF2F and LeanDojo benchmarks, demonstrating improvements in proof success rate, tactic success rate, execution time, and tactic diversity when applied to models like ReProver and InternLM2.5-Step-Prover. The paper also explores learning environment dynamics to predict tactic outcomes, contributing to more efficient proof search.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths:\n\n1. The paper presents a novel application of Determinantal Point Processes (DPPs) to theorem proving, a domain where such probabilistic models for diversity-driven filtering are underexplored. The idea of learning tactic representations that capture semantic effects on the proving environment (rather than syntactic similarity) is innovative and addresses a critical gap in automated theorem proving (Section 1.1, Page 2).\n2. The experimental evaluation is robust, with clear results on standard benchmarks (miniF2F and LeanDojo) using two different models (ReProver and InternLM2.5-Step-Prover). The paper provides detailed metrics, including Pass@1, tactic success rate, execution time, and diversity (Tables 2, 3, 5, 6, 7, 11), supported by statistical significance measures (standard error). The use of a transition model to predict tactic outcomes (status, time, output) is well-executed and validated (Table 1, Page 5).\n3. The paper is well-structured, with clear explanations of the problem, methodology (Sections 2 and 3), and evaluation. Figures (e.g., Figure 2, Page 4; Figure 3, Page 6) and Algorithm 1 (Page 6) effectively illustrate the architecture and filtering process. The use of standard metrics like BLEU, ROUGE-L, and cosine similarity enhances interpretability.\n\n#### Weaknesses:\n\n1. The evaluation is somewhat limited by the scale of experiments, as acknowledged in the limitations section (Page 9). The hyperparameter sweep is minimal (Table 10, Page 13), and the authors note that comprehensive tuning was computationally prohibitive. This raises questions about the generalizability of the results across different hyperparameter settings or larger models.\n2. While the paper is generally clear, the explanation of the DPP-based filtering mechanism could be more accessible to readers unfamiliar with DPPs. The mathematical formulation (Page 3) is dense, and a more intuitive explanation or example could improve understanding. Additionally, the truncation of text in several sections (e.g., Pages 1, 2, 3) makes it harder to fully assess some details.\n3. The improvements, while notable (e.g., 3.2–3.6% relative improvement on LeanDojo, Page 13), are modest compared to advancements in base model improvements (e.g., 36% improvement cited in Xin et al., 2024, Page 8). This suggests that 3D-Prover’s impact may be incremental rather than transformative, particularly for shallow proofs where broader search is favored (Page 7).\n\nQUESTIONS:\n1. The hyperparameter sweep is limited (Table 10, Page 13), and the authors note computational constraints. Could you provide additional results or analysis (e.g., in supplementary material) showing the sensitivity of 3D-Prover to key hyperparameters (e.g., \\(\\lambda_s\\), \\(\\lambda_t\\), \\(\\theta\\))? This could strengthen the quality score by demonstrating robustness or optimal configurations. A response showing improved performance with tuned parameters could increase the quality rating to 4.\n\n2. The DPP-based filtering mechanism is mathematically dense (Page 3). Could you include a more intuitive explanation or a concrete example (beyond Figure 3) to clarify how DPPs balance quality and diversity in tactic selection? This could improve the clarity score to 4 by making the method more accessible to a broader audience.\n\n3. The evaluation focuses on ReProver and InternLM2.5-Step-Prover. Have you tested 3D-Prover with other theorem-proving models or environments beyond Lean? Providing evidence of generalizability could elevate the significance score to 4 by showing broader applicability.\n\n4. The paper highlights improved performance for deeper proofs (Table 11, Page 13). Could you quantify the improvement in proof depth (e.g., average proof length) across benchmarks and compare it to baselines? This could strengthen the significance score by emphasizing 3D-Prover’s contribution to a known challenge in theorem proving.\n\nLIMITATIONS:\nYes. The authors addressed limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}