Title: Tree-Based Premise Selection for Lean4
Venue: NeurIPS 2025
Accepted: yes
Keywords: Premise Selection

Formal Theorem Proving

Lean Proof Assistant

Expression Trees

Structural Similarity

Weisfeiler-Lehman Kernel

Tree Edit Distance.
Abstract: Premise selection is a critical bottleneck in interactive theorem proving, particularly with large libraries. Existing methods, primarily relying on semantic embeddings, often fail to effectively leverage the rich structural information inherent in mathematical expressions. This paper proposes a novel framework for premise selection based on the structure of expression trees. The framework enhances premise selection ability by explicitly utilizing the structural information of Lean expressions and by means of the simplified tree representation obtained via common subexpression elimination. Our method employs a multi-stage filtering pipeline, incorporating structure-aware similarity measures including the Weisfeiler-Lehman kernel, tree edit distance, $\texttt{Const}$ node Jaccard similarity, and collapse-match similarity. An adaptive fusion strategy combines these metrics for refined ranking. To handle large-scale data efficiently, we incorporate cluster-based search space optimization and structural compatibility constraints. Comprehensive evaluation on a large theorem library extracted from Mathlib4 demonstrates that our method significantly outperforms existing premise retrieval tools across various metrics. Experimental analysis, including ablation studies and parameter sensitivity analysis, validates the contribution of individual components and highlights the efficacy of our structure-aware approach and multi-metric fusion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a graph-based approach to premise selection for Lean 4. The framework consists of multiple filtering stages. It employs two major steps: the WL kernel for search-space reduction and tree edit distance (TED) for fine-grained retrieval. Experimental results indicate improvements across two test sets and at different iteration levels (@k).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n**The paper is easy to follow.** Definitions appear naturally and are well explained.\n\n**The proposed method can run on consumer‑level hardware**, as it balances precision and speed without relying on heavy computations.\n\n**Experimental results support the main claims.** Improvements are shown across two test sets and various iteration levels.\n\n**The method leverages Lean expression trees directly**, rather than relying on embeddings or other generic encodings, the framework is tailored to graph representations of Lean theorems.\n\n## Weaknesses\n**Concerns regarding novelty.** The presented work does a good job of conveying the choices behind each part of the pipeline; however, all of these methods have been explored before, and some are fundamental to working with graphs. The authors combine these methods, but what is novel about this work specifically? Which part of the pipeline was or is underexplored in the literature?\n\n**Parameter selection is confusing.** The authors present Table 5 with parameter selection results but do not offer guidelines for choosing them. It would be helpful if they explained how each parameter impacts the results (e.g., sensitivity, relevance to the query) rather than just presenting numbers. Moreover, the accuracy differences across these selections are quite small, how do the authors interpret this? They use $(\\alpha, \\beta, \\gamma, \\sigma) = (0.1, 0.3, 0.1, 0.5)$ but fail to explain the rationale behind these values. The choice of parameter $\\sigma$ also seems underexplored.\n\nQUESTIONS:\nBased on the Weaknesses section, I have the following questions:\n- What is the novelty of the proposed framework?\n- How can we efficiently choose hyperparameters?\n- What does each hyperparameter affect?\n- What is the role of $\\sigma$ and how can we choose it?\n\nAnd recommendations:\n- Conduct a similar experiment to Table 5 but for $\\sigma$\n- Add one more paragraph about hyperparameter selection and their effect on the pipeline\n\nLIMITATIONS:\nThe limitations of the work are discussed in the Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of premise selection in interactive theorem proving for Lean4, where existing methods relying on semantic embeddings fail to leverage structural information effectively. The authors propose a tree-based framework that normalizes Lean expressions using Common Subexpression Elimination (CSE) and employs a two-stage filtering pipeline: coarse screening via the Weisfeiler-Lehman (WL) kernel and refined ranking via Tree Edit Distance (TEDS). The framework integrates adaptive fusion of multiple structural/semantic metrics (WL cosine, TEDS, Const Jaccard, Collapse-Match similarity) and cluster-based search optimization. Experiments on Mathlib4 demonstrate significant performance gains over baselines, with the method achieving up to 82% Recall@1 on small-scale problems and 25.8% on large-scale test sets. Key contributions include structured theorem representation, multi-metric fusion, and scalable retrieval mechanisms for large theorem libraries.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The explicit use of expression tree structures (via CSE) and WL kernel encoding addresses a critical problem in prior work, improving retrieval for structurally similar theorems .\n2. The two-stage pipeline balances speed (WL kernel for coarse filtering) and precision (TEDS for fine-grained ranking), making it feasible for large libraries (217k theorems) .\n3. Rigorous experiments (Ablation Study, Parameter Sensitivity) validate component contributions, e.g., Const Jaccard similarity shows the highest impact on performance .\n4. The method directly aids Lean4 users by reducing proof search space, with code/data release ensuring reproducibility .\n\n**Weaknesses:**\n\n1. Performance drops in complex domains (e.g., $Algebra, Recall@1=4.3%$) suggest struggles with nested type structures .\nManual Parameter Tuning: Adaptive fusion weights $(α, β, γ, δ)$ require empirical optimization, limiting generalizability .\n2. The framework focuses on syntactic structure but underleverages Lean’s dependent type system for semantic compatibility .\n3. TED’s $O(|T_1|×|T_2|)$ complexity may hinder very large tree comparisons, though CSE mitigates this partially .\n\nQUESTIONS:\n1. Can the authors explore GNN-based encodings to handle nested type structures in complex domains (e.g., Algebra)? A case study on improving $Recall@1$ from $4.3%$ to > $10%$ would strengthen the claim of generalizability.\n2. Are there plans to implement an automated mechanism for optimizing fusion weights (Eq. 2) instead of manual tuning? Demonstrating a data-driven approach (e.g., reinforcement learning) could enhance practical utility.\n3. How might integrating type-checking information (e.g., dependent type compatibility) into TED’s cost function improve retrieval for theorems with complex type signatures? A ablation study on type-aware vs. type-agnostic filtering would be valuable.\n4. For theorems with $>500$ nodes, TED’s computation time becomes prohibitive. Can the authors propose the hierarchical TED approximation or early pruning strategies? Benchmarking on extremely large trees (e.g., $>1k$ nodes) would validate scalability.\n\nLIMITATIONS:\nThe authors acknowledge limitations in complex domains and parameter sensitivity in Appendix G, but the discussion of potential negative societal impacts (e.g., misuse in formal verification of harmful algorithms) is cursory. To improve, they should:\n\n(1). Discuss risks of incorrect premise selection in safety-critical applications (e.g., autonomous systems verification).\n\n(2). Address biases in Mathlib4’s domain distribution (e.g., over-representation of Natural Number problems) and its impact on real-world deployment.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the task of premise selection and retrieval for Lean4. In particular, the authors introduce a multi-step filtering pipeline operating on a tree-like representation to better capture structural information in the expressions. The evaluation, performed on a large library derived from Mathlib4 shows significant improvements over a set of existing tools and baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Reasons to accept:**\n\n- The paper is generally well-written and easy to follow. The method is well-motivated and presented.\n- The idea of leveraging structural information in the expressions at scale is an important technical contribution, which could serve as an inspiration for future work in the field and to develop tools for premise selection. \n- The results show that the method can significantly outperform the baselines on a carefully constructed dataset from Mathlib4.\n\n**Reasons to reject:**\n\n- The presented methodology is highly specialised for a single formal language (i.e., Lean4) and evaluated on a single benchmark constructed by the authors. It is unclear how their pipeline can generalise to different formal languages (e.g. Isabelle) and different setups, and how structural information can be leveraged when the tree representation is not directly accessible. \n- While the improvements when compared to the baselines are impressive, several crucial details are missing in the experimental setup that make it very hard to assess whether such an improvement is the result of a fair comparison. The details and the motivations for choosing the baseline models are not described in the paper.\n- Most of the baselines seem to perform very poorly on the constructed dataset. However, by looking at the original datasets on which some of these baselines are tested, the performances are significantly different. For example, the LeanSearch paper reports a performance of P@10 and R@10 of 19.6 and 91.3, respectively, while in the constructed evaluation setup, this baseline only achieves 0.5 and 5.0. This casts serious doubts on the fairness of the comparison and on the deployment of the selected baseline methods. \n- It is unclear from the paper whether the authors compare their approach with deep learning based methods designed to leverage structural information. For example, to really show the effectiveness of their approach, I believe it would be important to include baselines based on Graph Neural Networks (GNNs). Moreover, it is unclear whether the baselines adopt neural models (e.g., Transformers), and whether such neural models are directly fine-tuned on their constructed dataset or applied out-of-distribution.\n\nQUESTIONS:\n- Could you provide additional details on each baseline, how you deploy them in your dataset, and why you chose to compare against each of these models?\n- Do the baselines include deep learning methods? Are these methods trained/fine-tuned on your dataset?\n- What causes the performance of the baselines to be so weak in your dataset? What explains such a significant difference in performance?\n- How does your approach compare with deep learning models designed to leverage structural information (e.g., GNNs)?\n- How can your pipeline be applied to different formal languages (e.g., Isabelle) and datasets where the tree-like representation is not directly accessible?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical problem of premise selection in LLM+Lean field, proposing a novel tree-based premise selection framework . \nThe method first applies CSE to standardize and compactly represent mathematical expressions as trees. A two-stage filtering pipeline is then introduced: the first stage uses the WL kernel for efficient, training-free coarse screening, while the second stage employs TED and a fusion of multiple structural and semantic similarity metrics for fine-grained ranking. To further enhance scalability and accuracy, the framework incorporates cluster-based search space optimization and structural compatibility constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe contributions of the paper are clearly presented, and the proposed method is practical and feasible. \n\nWeaknesses:\nCompared to traditional approaches, the method proposed in this paper is significantly more complex and cumbersome, requiring more parameters and procedural steps. However, it is difficult to perceive why, from an overall perspective, this method is fundamentally more effective or intelligent. Instead, it seems that the performance improvements are mainly achieved by introducing more parameters, more stages, and combining a greater number of metrics. The method appears to lack a unifying insight or an intuitive explanation that would make its overall advantage clear，is more of an engineering solution rather than a purely theoretical innovation.\n\nQUESTIONS:\n1. How does the method scale with extremely large theorem libraries, and what are the computational costs associated with the two-stage filtering pipeline in real-world Lean4 environments?\n\n2.To what extent can the proposed framework be generalized?\nI find the proposed method somewhat complex. Could the authors clarify why such algorithmic complexity is necessary for this task? Would it be possible to address the premise selection problem using a very large neural network instead, and how would such an approach compare to your method?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}