FormalML: A Benchmark for Evaluating Formal Subgoal Completion in Machine Learning Theory
Authors: Xiao-Wen Yang
Zihao Zhang
Jianuo Cao
Zhi Zhou
Zenan Li
Lan-Zhe Guo
Yuan Yao
Taolue Chen
Yu-Feng Li
Xiaoxing Ma
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language model
Neural Theorem Proving
Machine Learning Theory
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) have recently demonstrated remarkable progress in formal theorem proving. Yet their ability to serve as practical assistants for mathematicians, filling in missing steps within complex proofs, remains underexplored. We identify this challenge as the task of subgoal completion, where an LLM must discharge short but nontrivial proof obligations left unresolved in a human-provided sketch. To study this problem, we introduce FormalML, a Lean 4 benchmark built from foundational theories of machine learning. Using a translation tactic that converts procedural proofs into declarative form, we extract 4,937 problems spanning optimization and probability inequalities, with varying levels of difficulty. FormalML is the first subgoal completion benchmark to combine premise retrieval and complex research-level contexts. Evaluation of state-of-the-art provers highlights persistent limitations in accuracy and efficiency, underscoring the need for more capable LLM-based theorem provers for effective subgoal completion.

Published as a conference paper at ICLR 2026
FORMALML: A BENCHMARK FOR EVALUATING FORMAL SUBGOAL COMPLETION IN MACHINE LEARNING THEORY
Xiao-Wen Yangl.2; Zihao " Cao Zhi Zhou', Zenan Li², Lan-Zhe Guol.3, Yuan Yao', Taolue Chen², Yu-Feng Li Xiaoxing Mal
State Key Laboratory of Novel Software Technology, Nanjing University
School of Artificial Intelligence, Nanjing University
3 School of Intelligence Science and Technology. Nanjing University
"Department of Computer Science, ETH Zürich, Switzerland
"School of Computing and Mathematical Sciences, Birkheck, University of London, UK yangxw@lamdd amda. nju. edu. cn, zihaozhang@ssail nju. edu..cd.cc cn
ABSTRACT
Large language models (LLMs) have recently demonstrated remarkable progress in formal theorem proving. Yet their ability to serve as practical assistants for math- ematicians, filling in missing steps within complex proofs, remains underexplored. We identify this challenge the task of subgoal completion, where an LLM must discharge short but nontrivial proof obligations left unresolved in human-providee provided sketch. To study this problem, we introduce FormalML, Lean 4 benchmark built from foundational theories machine learning. Using a translation tactic that converts procedural proofs into declarative form, extract 4,937 problems spanning optimization and probability inequalities, with varying levels of difficulty. FormalML. is the first subgoal completion benchmark to combine premise utiliza- tion and complex research-level contexts. Evaluation of state-of-the-ard provers highlights persistent limitations in accuracy and efficiency, underscoring the need for more capable LLM-based theorem provers for effective subgoal completion.
Codebase
Dataset
INTRODUCTION
Recent advancements in large language models (LLMs) have showcased their impressive performance in formal theorem proving (Ren et al. [025 Wang al. 2025 Xin et al.) (025), particularly culminat- ing in silver-medal level performance at the 2025 International Mathematical Olympiad (Chen et al.) 2025). Nevertheless, their effectiveness as copilots for mathemattciias in tackling research-level problems (Yang et al.] 2024) remains far from satisfactory. For example, the recent Equational Theories Project (Tao al. (2024) reported that "LLMs did provide useful suggestions beyond what the human participants could already propose" in most of the difficult cases.
In this paper, rather than attempting to generate a complete proof for such research-level problems, we investigate an intermediate milestone, which we refer to as subgoal completion. To elaborate, human experts specify the problem in natural language, formalize the theorem, and outline the high-level proof structure, while Al systems assist by filling in the missing technical details (Jiang et al. (022). Figure illustrates this process using the proof of gradient descent convergence (Li et al.) 2024a 2025b): The expert provides the informal statement and the main reasoning steps. At the natural language level, the proof appears complete; however, once formalized, it remains incomplete, as indicated by the sorry placeholders. This gap arises because the theorem prover cannot automatically bridge each reasoning step. Subgoal completion precisely aims to bridge this gap, aligning the informal and formal proofs.
"qqual Contribution.
Corresponding Author.

---

Published as a conference paper at ICLR 2026
Theorem. I differentiable, and its gradient I' (-lipschitz, then
f- the bound for one step of the gradient aethod
gradient descent stepsize a [0. ] at any point satufies
using the Lipschitz continuity of the gradient -/
theores convex_ conver_Lipschitte Lipschitz
the V X1 : E, lissracienttt (f' xxl x))
Proot, Let's plug the gradient descent update by letting (K) We then get.
the > (0 RH ths. Ls a) (hax a M1
Cho Lipschitznithh 1.)
r00 s roo 19700 -
Intro
cale
inner It' x]
by sorry
100 100-9118001
:- by serry
1- by sorry
Figure 1: The informal and formal versions of the statement and proof regarding the sufficient decrease in gradient descent. Both the formal statement and proof can be accurately formalized from the natural language version. However, to verify each step in Lean 4, there are unproven holes (notated as sorry) that need to be further completed by existing automated theorem provers.
Despite its natural role as interface between informal and formal reasoning, subgoal completion still lacks dedicated benchmarks. Existing evaluations are built around competition-style problems and emphasize full-proof generation, whereas target lies in research-level theorems and the task of completing intermediate proofsteps. Moreover, while proof-generation benchmarks primarily measure the LLM performance by proving success rate, subgoal completion poses a different challenge: handling complex proof contexts efficiently without resorting to verbose or speculative reasoning chains. In practice, a useful prover must therefore strike a balance between accuracy and conciseness. To achieve these goals, we make the following contributions:
A Lean 4 translation tactic for subgoal extraction. We propose symbolic strategy for the dataset construction. Specifically, we implemented a new Lean tactic that extracts subgoal problems from procedural-style proofs. By adapting the length of the proof segments during extraction, we further utilize this tactic to generate new problems with varying levels of proving difficulty.
The FormalML benchmark. We narrow the scope to the foundational theory of machine learning (ML) and establish new benchmark in this domain. Our motivation is twofold. First, Al agents are becoming increasingly central to automating scientific discovery
Chomera-Paredess et al. 2024).
particularly within machine learning research (Lu et al., 2024 Yamada et 2025) Gottweis ett al., 2025). Hence, automated theorem proving plays a key role in guaranteeing the soundness of derived results. Second, automation tools that verify theoretical correctness can substantially ease the burden on human reviewers (Xu et al.) Pineau et al. 2021), especially given the rapidly growing volume of ML conference submissions. A well-designed benchmark would accelerate the development of such tools. To this end, building on two Lean 4 libraries Optlib (Li et al., (024a) and FoMI. (Sonoda al. 2025), we establish dataset of 4,937 subgoal completion problems.
Systematic evaluation of LLM-based provers MoL We evaluate state-of-the- art LLM- based provers on FormalML, highlighting fundamental limitations in accuracy, token coken-efficiency, and premise utilization. Our results show that while models such let al. (005) improve premise utilization capabilities, overall performance drops sharply on higher- difficulty problems. Moreover, chain-of-thoughh prompting, though effective in natural language reasoming, fails to improve proof completion and often reduces efficiency in this context. Addi- tionally, we find expert iteration be a effective training approach on FormalML. These insights highlight the need for further development and refinement in LLM- -bsee theorem provers to better support mathematicians in their work.
PrepSeek-Prover V2 (Ren)
2 RELATED WORK
L.M-- Theorem Proving. In recent years, the rapid development of LLMS has significantly advanced research on formal theorem proving (Li et al. (0244)) Current mainstream approaches can be broadly divided into two categories. The first category employs tree-search strategies, including
best-first search (Yang et al. 2023 Xin et al.) [025 Ying et al. 2024 Polu & Sutskever 2020). Monte Carlo tree search (Xin et al. 2024| Wang et al.) 2023: Kocsis & Szepesvári 2006 Yang et al. 2025), and others (Lin et al. 2024). Although these methods align with the Markov Decision Process (MDP) framework of theorem proving, their step-by- tactic generation and frequent interaction with the Lean environment result in low search efficiency. The second category is whole-
2

---

Published as a conference paper at ICLR 2026
proof generation (Dong & Ma 2025 Lin et al.
2025a Zhang al. 2025 Wang et al. 2025 Ren
et al. 2025 Chen et al. 2025). J. where LLMs directly generate complete proots, usually through multiple rollouts verihed in Lean identify the final proof. This approach has demonstrated notable advantages. For instance, DeepSeek-Provee Prover- (Ren et al. 2025) and Limina-Preview (Wang et al., (023) leverage natural aanuuage-aided Long-CoT reasoning to produce formal proofs, achieving breakthroughs in competition-level tasks. However, the performance of existing LLM-based theorem provers in supporting practical formal proving has not yet been fully evaluated.
Formal Theorem Proving Benchmarks. The current mainstream Lean formal theorem proving benchmarks can be categorized into three primary types based on problem characteristics. The first type focuses Olympiad- level high school mathematics competition problems, exemplified by: the MiniF2F (Zheng et al. 2022) dataset, which comprises 244 challenging problems from competitions such AMC, AIME, and IMO; and the PutnamBench benchmark (T'ouuklass al.. 2024), featuring problems derived from the Putnam Mathematical Competition. The second type targets undergraduate- level mathematical tasks, including ProofNet (Azerbayev et al., 2002) (containing analysis and algebra problems at the undergraduate level) and MinCTX (Hu et al., 2024) (theorems sourced from real- world Lean projects and textbooks). Additionally, hybrid benchmarks like FormalMATH (Yu et al., 2025) cover theorem proving across domains ranging from high school to undergraduate mathematics. Another category, exemplified by Leandojo (Yang et al.) 2023), directly constructs datasets from the Mathlib library. Notably, existing benchmarks either operate within simple contexts or do not require premises, and all involve full proof generation tasks. In contrast, our proposed FormalML benchmark specializes in subgoal completion tasks, design better aligned with practical applications where LLM-based theorem provers assist humans in practical theorem proving. A comprehensive comparative analysis of benchmarks is presented in Table I
Table I: Comparison of existing Lean 4 benchmarks.
Complex Context
Subgoal Completion
Benchmark
Problems
Type
Premise
MiniF2F (Zheng et al.
2022
244
Olympiad Olympiad
X
x
PutnamBench soukalas ct al 2024
522
X
x
ProofNet (Azerbayev et al 20241
186
Undergraduate (UG)
X
x
FornalMATH c. 2025
5. 560
Olympiad & UG
X
LeanDojo Yang al. 2025
2000
Mathlib
X
MiniCTX [Hu et al. 2024
762
UG
FormalMI. (Ours)
4, 937
UG
V
FORMALML
3.1
SUBGOAL EXTRACTION
With the growing availability of formalized libraries in machine learning theory, we aim leverage these resources for dataset construction. A key challenge, however, lies in extracting subgoals from procedural-style proof scripts (Winograd, 1975] Wiedijk 2012). Unlike declarative-style proofs (Syme) 1997), which progress by iteratively introducing and proving intermediate subgoals, mocedural-style proofs instead transform the proof state step by step. As a result, existing goal- extraction tactics such as Lean 4's extract_goo can only capture the overall goal of proof state, but not the finer-grained intermediate subgoals corresponding to each reasoning step. This limitation is particularly pressing as most proofs in current libraries are written in the procedural style. For example, in the proof of the lemma linear _gradeent (see Figure and Appendix ()) procedural scripts such rw and simp span nine lines, yet declarative statements (e.g., have) are entirely absent.
Fortunately, in Human--wittee Lean 4 proofs, we observe consistent pattern: although proofs are expressed procedurally as sequences of tactics, each proof line typically corresponds to a single reasoning step identified by the human author. In practice, human experts tend to first complete the reasoning informally and subsequently encode it in tactics. Accordingly, a single line frequently chains multiple tactics until the intended reasoning step has been fully realized, at which point new line is introduced. Consider the running example in Figure 2 the expert combines two tactics (i.e., repeat [dotProduct] and simp [mul_comm]] within a single line to ensure that the resulting proof state aligns with the underlying informal reasoning. This observation motivates us to analyze human--ritten proofs the line level.
3

---

Published as a conference paper at ICLR 2026
lenna gradier_ MasGradientAt funn x.1 (Eacgpdeangpece B. (Fin n1)
|EuclideanSpace (Fán ni),
extracted, tormat in Ne)
(b Fān ra
Matrix (Fin mm) (Fin 1n) R.
=> (b ". IA My X231 (A" Hy b1
Ix Eue Lideanspace R (Fan tn)) la ()) Japos :
ly Euc LideanSppace R (Fin end) [a_1
[inner__eg_a wwt.rrrddoooructttllceooomee repeat [dotProcuct]; zamp [nul.com)] apply mul_nonneg; Linorith [spos]
to_ to_theoree
1h
AT
vil== by
reseat nr loctProdact] at simp I at
try repeat assumption
Na
Fin
Matrix (Fin 1n) BucLideanSpace (Pin 10)
(FAD FuclideanSpace (Fin m))
repeat [dotProduct]]
a I >
[mm]__com]]
2 apos EuclidennSpace (Fin m))
EuclideanSpece (Fin mn)
Figure 2: An example of the to_ to_theorem tactic illustrates its functionality. When applied to the tactic repeat rw [dotProduct]; simp [oml_comm] it captures pre- and post-execution proof states, abstracts their transition, and synthesizes subgoal.
We implement customized Lean 4 tactic to_ theorem, which automatically encodes line-level, procedural proofsteps into subgoal, with the corresponding proofsteps serving as their proofs. As shown in Figure 2 the tactic operates by recording the proof states immediately before and after executing the procedural steps repeat rw [dotProduct]; simp [mul_comm]] It then inserts the prior state as hypotheses and the subsequent state as the goal, thereby deriving a new theorem. This theorem effectively isolates a subgoal the original proof, which can then be discharged directly by replaying the original procedural tactics.
The tactic to_ theorem extracts the subgoal from single line. We can naturally extend it to proof- segment extraction, where a sequence tactics spanning multiple lines is first converted into the declarative format, and then its subgoal is extracted. For the running example linear__e gradient in Figure 2 which totally comprises a nine-line proof, we can, in principle, extract up to nine single-line subgoals; four oveellapppping two-line subgoals; and so forth.
3.2 DATA CURATION
We categorize machine learning theories into two main types: optimization theory and probability theory. The former typically analyzes the convergence of optimization algorithms (e.g., gradient descent), while the latter addresses error bounds of machine learning models (e.g., Hoeffding's inequality). To develop FormalML, we expand and extract data based on two projects: Optibb (Li al. 2024a) and FoML (Sonoda et al. 2025). For each project, we designate the top-level theorems (e.g., the convergence of gradient descent) for subgoal extraction, while preserving lower-level lemmas (e.g.. Lipschitz continuity) as local library for premises. We utilize the to_ theorem tactic to extract the subgoals, and ultimately compile 4,937 theorems into FoLmalML.
a project started in September 2023 that formalizes wide range of optimization algorithms, including gradient descent (GD), subgradient method (SubGD), proximal gradient descent (PGD), Nesterov acceleration method (NAG), block coordinate descent (BCD), and alternating direction minimization method (ADMM). Building on Optlib, we extend and refine theorems related to these algorithms, and update some proofs ensure compatibility.
Optlib (Li et al. 2024a 2025a
FoML (Sonoda et al. 2025) is recent project initiated in March 2025, primarily focused on formal- izing the generalization enror bound using Rademacher complexity. It also includes some important probability inequalities, such as the expectation inequality. the bounded differences inequality, and McDiarmid's inequality. Furthermore, we formalize Hoeffding's lemma, the Bennett inequality, and the Bernstein incquality into the library to enhance its comprehensiveness.
Each theorem in FormalML is stored in JSON format (See Appendix
B). including:
(1) Source location metadata (file path and line number coordinates for theorem extraction).
(2) The formal Lean 4 theorem statement.
(3) Required module imports and namespace declarations.
(4) The complete tactic sequence constituting the original proof.
(5) Proof-relevant premises from the mathematical context.
4

---

Published as a conference paper at ICLR 2026
Table 2: Statistics of theorems in FormalMI. across various machine learning theories.
Optimization
SubGD
PGD
NAG
BCD
ADMM
Other
Total 2,907
Problems
211
331
388
528
433
1,016
Probability
Exp
Bennett 136
McDiarmid
Rademacher
Hoeffding
Mcasure 315
Other 170
Total 2,030
Problems
100
642
615
52
Statisties of Theorems
Library Statistics
1000
2258
Optimization
Number Problems
814
Optimization
6 Average Premises
2000
Probability
750
Pribahility
Numbers
1666
Op.. Arg. Premises
500
499
Prob. Avg. Premisos
1000
3.87
0
488
294
161
70
250
1.90
1.03
149
1.02
112
Mathlib
Local I Library
Difficulty level
Library Type
Figure 3: (Left) distribution by proving difficulty; (Right) statistics of premises.
This encapsulation ensures a complete proof context for each theorem, enabling independent evalua- tion. We release both the dataset and the evaluation code open source.
3.3 BENCHMARK CHALLENGES
At first glance, subgoal completion may appear simpler than the proof generation for complex competition problems, in which LLM-based theorem provers have demonstrated their excellent performance. However, identify that the proofstep generation task remains particularly challenging for LLM-based theorem provers, attributable to the following three key reasons.
Complex proof context. In practice, users often rely on automated theorem provers to close subgoals within lengthy and complex proofs. This requires LLMs to comprehend all accumulated hypotheses and intricate goals to generate accurate proofs. However, current provers pay little attention complex contextual reasoning.
Premise utilization. Accurately using relevant premises from both local contexts and global libraries essential, as many proofs rely on interdependent lemmas (Hu et al. 2024). A common workflow involves first performing premise selection via retrieval from large library form small candidate set, which is then fed into an LLM for premise utilization. This paper focuses on the latter stage, as the ability of model to choose and properly use premises from candidate sets is critical for practical subgoal completion. However, current LLM-based provers still perform inadequately in this regard (Yang et al. 2023). and few studies have systematically explored this.
Overthinking and efficiency. Unlike competition problems that demand long chains of reasoning, subgoal completion is often repetitive and involves relatively straightforward reasoning. LLMs trained for complex competitions may overthink such tasks, exploring unnecessary inference paths that hinder performance (Chiang & Lee| 2024 (Sui et al. 2025). Moreover, efficiency is crucial: it is impractical expend substantial computational resources on proving subgoals that require only brief, direct proofs. Balancing reasoning depth with computational cost is thus key challenge for practical subgoal completion.
3.4 BENCHMARK STATISTICS
Summary. We establish FormalML by systematically expanding and extracting theorems from two Lean 4 libraries: Optlib and FoML. The resulting benchmark comprises 4,937 unique theorems, each derived from a single proof step or proof segment within the top-level theorems presented in these libraries. For the extracted theorems,
categorize them according their corresponding
top-level theorems, as illustrated Table 2 The results indicate that the benchmark encompasses a diverse range of theories in machine learning. The number of extracted theorems associated with each top-level theorem depends on the length the corresponding proof. On average, hundreds of theorems can be successfully extracted from each top-level theorem.
5

---

Published as a conference paper at ICLR 2026
Proving difficulty. To enable fine- line-grained evaluation of LLM-based theorem provers, we categorize theorems based on their proving difficulty. Specifically, since the proofs in FormalML are mostly procedural style, we can the proof length metric of proving difficulty (Zhang et al. 2024). We define three levels of proving difficulty, corresponding to proof lengths of 1, 3, and 5, respectively, and report the statistics of theorems across these levels in the left panel of Figure In total, there 3,924 theorems at difficulty level and 1,013 theorems at higher difficulty levels.
Premise utilizaaion statistics. We further examine the premise utilization difficulty within FormalML. In total, 1,547 theorems require the use of explicit premises. These theorems are divided into two categories based on the source of the premises: Mathlib and the local library. The right panel of Figure reports the number of theorems in each category, as well as the average number of premises.
EXPERIMENTS
In this section, we conduct a comprehensive evaluation of otatc-of-the-art LLM- LLM-based theorem provers using FormalML benchmark. Through systematic experiments, we identify several limitations and phenomena characterizing current theorem provers in practical subgoal completion tasks.
4.1 EVALUATING LLM-BASEI THEOREM PROVERS ON FORMALML
Theorem Provers
We primarily focus on the two most effective types of LLM-based provers:
Best-First Tree-Search (BFS) Methods: Each node in the search tree corresponds to a proof state, and heuristic scoring function assigns priorities nodes. The BFS algorithm is employed to explore the search space and derive the final proof. We evaluate two models in this framework: Reprover (Yang al. 2023) and BFS-Prover (Xin et al. (025).
These types of models generate a complete formal proof in a single pass directly from the problem description, eliminating the need for additional state transitions search procedures. We consider the following SOTA models: STP (Dong & Ma 2025).
Whole-Prowe Proof Generation Methods:
Goedel-Prover (Lin et al.) 2025a). Goedel-Proves V2 (Lin et al. 20255). Leavabell-Prover (Zhang Prover-Preview-77B (Wang al. 2025) DeepSeek-Prover Proverk--rvve-V.... (Xin et al. 2024 and DeepSeck-Provers
al. 2025).
(Ren et al. 2025).
In addition, we compare several automation tacties in Lean 4, such as simp, ring, linarith th, tauto, grind, and aesop. Lean 4's aesop (Limperg & From] (003) is strong and sophisticated, rule-based proof search engine. The ensemble means we combine all of the aforementioned automation tactics to try.
Metrics & Evaluation We employ the Pass@ K metric to evaluate theorem provers, where K represents the computational budget. This metric quantifies the proportion of problems for which at least one valid proof is discovered within the top K generated attempts. For BFS methods, K ExT. where E represents the number of tactics generated per expansion, and T denotes the number of expansion iterations. Unlike prior studies that predominantly focus competition-level theorem by highly challenging problems and lengthy proofs, therehy requiring large values of K, this work concentrates on the research level subgoal completion. Consequently, our benchmark emphasizes low computational budgets, corresponding to small values of K (< 8 x 50 for BFS methods and < 32 for whole -rooff generation methods).
Experimental Details For BFS methods, we implement and conduct interactive search based on the LejnDojo (Yang et al. 2023) framework. For whole-proof generation methods, we employed
vLLM (Kwon et al. 2023) to generate proofs and evaluated their correctness using kimina-lean- server (Santos et al. 2025). which provides high-throughput Lean 4 verification. All experiments strictly adhered to the published parameter configurations and prompts for each model. All computa- tions were performed on NVIDIA H800 GPUs.
Main Results In Table 3 we show the performances of current theorem provers on our FormalML. benchmark. Our experiments demonstrate that current whole-proof generation models have prelimi- narily acquired the capability to solve FormalML problems when increasing sample budgets, STP attaining the highest Pass @2 score of 63.21% %. However, their Pass @ performances remains at a relatively low level (peaking at merely 26.96%), falling short of the practical requirements for assist- ing mathematicians in completing Lean formal proofs. Notably, the BFS tree search strategy, despite consuming more computational resources, does not yield significant performance improvements

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison of automation tactics and LLM-bbsed theorem provers For-
malML. The best results are in bold, and the second-best are underlined.
Model
Sample Budget
Pass @ K (%)
Method
Size
Optim.
Prob.
Automation Tactics
simp ring
19.85
15.96
18.25
1.69
1.18
1.48
linarith
13.14
10.33
tauto
21.40
16.35
19.32
grind
15.69
11.28
13.87
aesop (Limperg & From 2023 Ensemble
45.24
40.49
43,29
51.53
44.58
48.67
Best-First Tree Search Methods
x 20
21.80
16.15
19.48
Reprover Yang al.
2023
229M
8 x 50
27.82
20.44
24.79
20
24.45
19.31
22.26
BFS- Prover Xin et al.
2025
7B
50
27.62
23.84
25.31
WWole-Proch Generation Methods
17.99
11.77
15.43
30.75
23.69
27.85
Kimina- Kimina-Prover-Pe
Wang al. 2025
7B
42,31
38.13
40,59
32
47.09
43.79
45.74
8.22
13.50
10.39
21.32
28.97
24.47
Goedel-Prover (Lin et al.
2025a
7B
37.29
41.28
38.93
32
44.34
46.11
45.07
20.85
19.31
20.21
34.61
33.15
34.01
Goedel-Prover-VV22 V2-8B
Lin al. 2025b
8B
16
45.13
43.35
44,40
32
49.67
48.67
49.26
24.11
24.09
24.10
47.33
43,30
45.68
Lecanahell-rrover Zhang 2025
7B
16
58.00
50.84
55.05
32
61.23
53.99
58.07
28.45
24.83
26.96
51.01
46.06
48.98
STP Dong Ma 2025
7B
16
62.13
57.73
60.32
32
65.19
60.39
63.21
24.29
17.29
21.41
43.31
37.39
40.88
DeepScck-Prover-VI (in al. 2024
16
$7.48
$2.56
55.46
32
61.51
56.00
59.25
17.16
16.40
16.85
37.98
37.09
37.61
DeepSeck-Provee Prover-V2 V2 (noCoT) (Ren et al. 2025
16
59.79
$2.76
56,90
32
65.08
57.73
62.06
16.23
21.58
18.43
25.73
32.56
28.54
DeepSeck- Prover-V2 (CoT)
RRen et al. 2025
16
33.54
39.36
35.93
37.39
42.51
39.50
(remaining below 30%). Additionally, most automation tactics generally underperform compared to LLMs. However, the powerful tactic aesop demonstrates strong performance, surpassing several LLMs under low sample budgets, though still falls short of LLMs under high sample budgets. Even if all these tacties were integrated together, the ensemble method still not outperforms LLMs. This suggests that LLMs possess the potential to solve subgoals, but their costs need to reduced. Furthermore, observe that: 1) the performance ranking of models exhibits inconsistencies with
7

---

Published as a conference paper at ICLR 2026
SUBGD
PGD
CD
Optin:
NAG
BCD
90%
Narter Sarppic
Narber Sample
Number Seeplack
Bartac Vesign))
Explnog
ADMM
Phoh
Plangot
of
Namber Sampla
Nember of Sampla
Bennett
Messure
Daysack Prove teatn
ALL
Paagge
ha
McDiannid
Hoeflding
Nache: Sarppis
Narber Sarphes
Narber of Kaeglis K
Forta Dorkow-
Dop 121 (ta))
Diffecit Level
Diffiuuly Level.
Dufiiclty Level
Figure 4: The left figure presents results of pass rate across various specific problem domains, while the right figure shows the performances under different difficulty levels.
the results from the miniF2F benchmark, suggesting that certain models exhibit varying degrees of overfitting to competition-level elementary theorem-proving tasks; 2) recent long-CoT provers (e.g. Kimina-Previes and Deepseek- Procee---ree---e V2) incorporating natural language assistance underperform on our Fo benchmark (with pass aass@3 32 consistently below 50%), presenting a stark contrast to their strong performance miniF2F.
Performance Distribution We present the distributional results of performance across various specific problem domains (e.g., the proof of Hoeffding's inequality). The experimental results presented in left panel of Figure demonstrate that the performance of existing models varies across different specialized areas. For instance, in the probability split, STP achieves significantly superior results on McDiarmid, far surpassing other methods. On Hoeffding, Deepseek-Proven V2 (noCoTT) delivers the best performance.
Finding 1: Existing LLMM-based theorem provers are inadequate as practical tools for assisting mathematicians in achieving subgoal completion under low computational budgets. Morcover, the performance of LLM--ased theorem provers exhibits significant disparities across different specialized areas within FormalMI.
4.2 EVALUATING PREMISE UTILIZATION ON FORMALML.
While writing practical proofs, humans often need to apply appropriate premises from local theorem libraries Mathlib. The experiments in this section are designed to evaluate the model's ability to utilize a given set of candidate premises through in-context learning to solve a proof subgoal, These experiments are carried out subset of our benchmark containing relevant premises, with the following specific design: for each proposition to be proved, we randomly sample candidate premises from a combined premise library comprising Mathlib and local premises, mixing actually correct theorems to ultimately form a candidate set containing M premises. All candidate premises are presented in a structured format within the model's context (the complete prompt template provided in Appendix[B F. Considering the current context window limitations of L.L.-based provers, this study employs four experimental settings (M 0, M = *, M = 10, A = 20) for comparative analysis. Here, M = 0 denotes the bascline with no premises, while M refers to the setting that includes only ground- -truth premises.
Results are demonstrated in Table 4 We observe that nearly all models (except for Goedel-Prover V2-8B) show improved performance when provided with accurate premises, indicating the benefit of supplying relevant contextual information. Although STP demonstrates strong performance across the full benchmark, it underperforms in premise utilization when M = 10 and M = 20. We attribute this to the absence of such data in its training set. We identify representative case in Appendix G.I In contrast, DeepSeek- Praiek-rrre---ee V2 achieves robust results under both CoT and noCoT settings, with high performance gain of approximately 10%. This indicates its strong premise utilization capability, which we believe stems from its training on extensive natural language reasoning data, thereby enhancing its general reasoning ability.
8

---

Published as a conference paper at ICLR 2026
Table 4: Pass @ 16/32 (%) of premise utilization. Relative improvements compared to M shown in green (increase) or red (decrease).
Method
M 0
M
M
10
M
20
32
32
16
32
16
32
STP
54.56
57.72
57.85 (+3.9)
61.86
52.23 (2.33)
56,43
52.17 (2.39)
56.04 61.00)
Goedel-Provee Prover
39.04
(-10.00
50.10 (.11.00)
34.58 (+)
(.700)
33.35 (40.3))
39.50
Pardll-Prove--2--88
45.24
51 51.26
45.18 6306)
50.29 com
45.05 (0.19)
49,90
46.09
50.36 (0.90
Leamaelll-Prowe Prover
(+705)
(4535)
47.45
(40.5)
(40.00)
Kinina-Prover-PPeee Preview-7B
34.71
39.56
35.16 (0:45)
40. (40.095
34,65 (0.00)
39.69 (v0.13)
35,49 (v077))
39.75 (40.9)
DeepSeek. Prover-Vi V1.5
(2000)
56.69 (4575)
46.86
51.59
(4).5)
42.79 (4.0))
DeepScek-Prooeet Prover-V2 (n)
53.46
58.37
59:47 (6600)
71.66 (113.49)
58.82
(1530)
(9.29
(-1092)
57,40 (13.8)
65, (4743)
DeepSeek- PeoSeek-Prover-.re (CoT)
37.94
44.36 (4900))
(4987))
44.34 99999)
(49.7))
42.73 (43.9)
46.74
Efficiincyy Comparison
Expert Iteration Comparison
2554
2641
70
Response Lengt
604
:90
60
Pass@ak
90-
PaskeLS
100
30
40-
EWAg00
10
135
Gookd Prover (becllax)
Goods-Proee 1-200 (esper her))
20
11x4
Lor nod Poom
20-
DoppBeck-22 (sot (bualiac)
DeepSet VS (eo( (eapen ber)
Pumer
Number of Samples K
Figure 5: The left figure illustrates the efficiency comparison among current whole-proof generation provers, while the right shows performances before and after expert iteration.
Finding 2: Existing LL-M-based provers often struggle with premise utilization when candidate set is large, resulting in suboptimal performance. Moreover, even when provided with the ground-truth set of premises, most models appear to underutilize them, with the exception of DeepSeek-Proverte remains largely attributable to limitations of the base models.
This observation indicates that the primary performance bottleneck
4.3 EVALUATING FORMALML OF DIFFERENT DIFFICULTY LEVEL
We evaluate the performance of existing whole- proof generation models at varying difficulty levels on FormalML. A larger used to assess problems higher difficulty. As shown in right of Figure experimental results indicate that model performance, both for the optimization and the probability split, decreases as difficulty increases. Notably, STP still achieves the highest results under di difficulty levels 3 and 5, with Pass @ 128 scores of 55.5% and 33.36%, respectively. Results of other models are shown in Appendix F2
Finding 3: Current LLM- theorem provers tend to exhibit significantly degraded perfor- mance when handling higher-difficuly problems in FormalML.
4.4 EVALUATING THEOREM PROVER EFFICIENCY ON FORMALML
In formal theorem proving, efficient provers crucial for practical applications. Recent studies demonstrate that RL-based long-CoT reasoning has been successfully integrated into systems such as DeepSeek-Prover V2, Kimina-Prover, and Goedel-Prover-VV These models employ deep natural- language reasoning before generating formal proofs, While they exhibit strong performance on competition- mathematical problems, our experiments reveal no significant advantage on the FormalML benchmark (See Table 3). Moreover, these models generally suffer from low reasoning efficiency. To address the trade-off between reasoning efficiency and accuracy in FormalML sce- narios, we use novel evaluation metric metric-FEfficiency wfffiiefccc--eeeiieeeee Weighted Accuracy (FWA @ K) defined as: FWA@ K Pass@ K Response Length This metric balances performance and output efficiency by normalizing the proof success rate against the response length. To the right of Figure 5 we compare the pass rates and efficiency of eight whole-proof generation models on FormalML. The results show that the three long-COT models generate the highest average number of response tokens, far exceeding other models. However, their performance is inferior to others, resulting in the lowest

---

Published as a conference paper at ICLR 2026
EWA @32 score. In contrast, STP achieves both the shortest output length and the highest pass rate on the FormalML benchmark, leading to the highest EWA@ @32 score.
Finding 4: Long-Co'T demonstrates no significant improvement in the pass rates of the subgoal completion task while simultaneously incurring substantial computational cost on MormalMI..
4.5 EVALUATING FORMALML WITH EXPERT ITERATION
We further explored the performance of expert iteration on FormalML. We extracted 92,815 problems from five repositories (mathlib, SeimeNNubbrTThoree....
PFR, PhysLean, and scilean) using the to_ theorem tactic, which 88, were used for expert iteration training. For each problem, we generated 8 candidate proofs and sampled the correct ones for training. We performed round of expert iteration on DeepSeck-Prover: V2 (noCoT) and Goedel-Prover V2-8B. Training details are shown in Appendix(] D Results shown in the right panel of Figure 5 The experimental results indicate a substantial improvement in performance after expert iteration, especially in Pass@ This suggests that expert iteration holds potential for enhancing subgoal completion tasks.
demonstrated that expert iteration enhances performance on FormalML and shows potential for improving subgoal completion capabilities.
Finding 5:
CONCLUSION
In this paper, we identify subgoal completion as a critical yet understudied task for helping mathemati- cians working research-level problems. We introduce FormalML, the first benchmark specifically designed for this task formal theorem proving. Using Lean 4 translation tactic, we systematically extract 4,937 subgoal problems from foundational machine learning theory, covering optimization and probability. Evaluations of state-of-the-an art JLM-based provers demonstrate persistent limitations in accuracy, efficiency, and premise utilization within complex proof contexts. Moreover, expert iteration has proven to be a promising direction for advancing subgoal completion. We anticipate that FormalML will catalyze the development of more capable provers support subgoal completion.
ETHICS STATEMENT
This work complies with the ICLR Code of Ethics. does not involve human subjects, sensitive data, or experiments with potential harm. All datasets are derived from publicly available formalized machine learning libraries in Lean4, with no proprietary information used. We believe the methods and results pose no ethical risks, and all authors take full responsibility for the integrity of the research.
7 REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our results. The procedures for data curation and experiments are documented in the main text and supplementary materials. And will release anonymous source code and scripts as supplementary material to facilitate reproduction of experiments.
ACKNOWLEDGEMENS
This research was supported by the Key Program of Jiangsu Science Foundation (BG2024036. BK20243012), Leading-edgs Technology Program of Jiangsu Science Foundation (BK20232003), Natural Science Foundation of China (62576162, 62025202), the Fundamental Research Funds for the Central Universities (0221 14380023), and the Frontier Technologies R&D Program of Jiangsu (BF2024059). Taolue Chen is partially supported by overseas grants from the State Key Laboratory of Novel Software Technology, Nanjing University (KFKT2023 tliyf@jj.eduu.co and Xiaoxing Ma (xxm@nju.ede Yang, Zihao Zhang, Jianuo Cao, Zhi Zhou, and Zenan Li are the contributors.
KFKT2025 Yufeng Li are the corresponding authors. Xiaowen
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. ProofNet: Autoformalizing and Formally Proving Undergraduate--Levl Mathematics. arXiv preprint arXiv:2302. 12433, 2023.
Luoxin Chen, Jinming Gu, Liankai Huang, Wenhao Huang, Zhicheng Jiang, Allan Jie, Xiaoran Jin, Xing Jin, Chenggang Li, Kaijing Ma, et al. SSeddproved Deep and broad reasoning for automated theorem proving. arXiv preprint 2xXi:2250.223766 2025.
Cheng-Han Chiang and Hung-yi Lee. Over-reasoning and redundant calculation of large language models. arXiv preprint arXiv:2401. 1467, 2024.
Kefan Dong and Tengyu Ma. Stp: Self-play Ilm theorem provers with iterative conjecturing and proving. arXiv e-prints, pp. arXiv-2502, 2025.
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXi::2502.18864 2025.
Jiewen Hu, Thomas Zhu, and Sean Welleck. minictx: Neural theorem proving with (long-) contexts. arXiv preprini 24i::2408.03350 2024.
Albert Q Jiang, Sean Welleck, Jin Peng Zhou, Wenda Li, Jiacheng Liu, Mateja Jamnik, Timothée Lacroix, Yuhuai Wu, and Guillaume Lample. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. arXiv preprint arXiv:2210.12223 2022.
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou (eds.), European Conference Machine Learning, volume 4212, pp. 282-293, 2006.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Chenyi Li. Ziyu Wang, Wanyi He, Yuxuan Wu, Shengyang Xu, and Zaiwen Wen. Formalization of complexity analysis of the first-order algorithms for convex optimization. arXiv preprint arXiv:2403. /1377, 2024a.
Chenyi Li, Zichen Wang, Yifan Bai, Yunxi Duan, Yuging Gao, Pengfei Hao, and Zaiwen Wen. For- malization of algorithms for optimization with block structures. arXiv preprint (XXi:25503.1800 2025a.
Chenyi Li, Shengyang Xu, Chumin Sun, Li Zhou, and Zaiwen Wen. Formalization of optimality conditions for smooth constrained optimization problems. arXiv preprint arXiv:2503. 18821, 2025b.
Zhaoyu Li, Jialiang Sun, Logan Murphy, Qidong Su, Zenan Li, Xian Zhang, Kaiyu Yang, and Xujie Si. A survey on deep learning for theorem proving. In First Conference on Language Modeling, 2024b.
Jannis Limperg and Asta Halkjær From. Aesop: bhite-box best-first proof search for lean. In Robbert Krebbers, Dmitriy Traytel, Brigitte Pientka, and Steve Zdancewic (eds.), Proceedings of the 12th ACM SIGPLAN International Conference on Certified Programs and Proofs, CPP 2023, pp. 253-266. ACM, 2023.
Haohan Lin, Zhiqing Sun, Yiming Yang, and Sean Welleck. Lean-STaR: Learning to Interleave Thinking and Proving, arXiv preprint 200.v:2407.100400 2024.
Yong Lin. Shange Tang, Bohan Lyu, Jiayun Wu, Hongzhou Lin, Kaiyu Yang. Jia Li, Mengzhou Xia, Danqi Chen, Sanjeev Arora, et al. Goedel-prover: A frontier model for open-source automated theorem proving. arXiv preprint arXiv:2502..7644 2025a.

---

Published as a conference paper at ICLR 2026
Yong Lin, Shange Tang, Bohan Lyu, Ziran Yang, Jui-Hui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng, Jiawei Ge, Jingruo Sun, et al. Groedel-povver-e Scaling formal theorem proving with scaffolded data synthesis and self-correction. arXiv preprint arXiv:2508. 03611,,,, 2025b.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408. 06292, 2024.
Joelle Pineau, Philippe Limcent-Lammrrr, Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). Journal of machine learning research, 22(164))1-20, 2021.
Stanislas Polu and Πly Sutskever. Generative Language Modeling for Automated Theorem Proving. arXiv preprint arXiv:2009.03330 03339,,, 2020.
ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, et al. Deepseek-prover-yy Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. arXiv preprint aXXiv2504.21801, 2025.
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg. Pengming Wang. Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475, 2024.
Marco Dos Santos, Haiming Wang. Hugues de Saxcé, Ran Wang, Mantas Baksys, Mert Unsal, Junqi Liu, Zhengying Liu, and Jia Li. Kimina lean server: Technical report. arXiv preprint arXiv:2504.21230, 2025.
Sho Sonoda, Kazumi Kasaura, Yuma Mizuno, Kei Tsukamoto, and Naoto Onda. Lean formalization of generalization error bound by rademacher complexity. arXiv preprint arXiv:2503. 1025503..660. 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: survey on efficient reasoning for large language models. arXiv preprint arXhv:250.11411, 2025.
Donald Syme. Declare: A prototype declarative proof system for higher order logic. Technical report, University of Cambridge, Computer Laboratory, 1997.
Terence Tao, Pietro Monticone, and Shreyas Srinivas. Equational theories project: Mapping out the relations between different equational theories of magmas. https://teorth. wit....ttttuu...... io/ souuational_theriees. 2024.
George Tsoukalas, Jasper Lee, John Jennings, Jimmy Xin, Michelle Ding, Michael Jennings, Ami- tayush Thakur, and Swarat Chaudhuri. PutnamBench: Evaluating Neural Theorem-Provee on the Putnam Mathematical Competition. arXiv preprint arXiv:2407.11214, 2024.
Haiming Wang, Ye Yuan, Zhengying Liu, Jianhao Shen, Yichun Yin, Jing Xiong, Enze Xie, Han Shi, Yujun Li, Lin Li, et al. DTT-Solver: Automated Theorem Proving with Dynamic- Tree Sampling Guided by Proof-level Value Function. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023.
Haiming Wang, Mert Unsal, Xiaohan Lin, Mantas Baksys, Junqi Liu, Marco Dos Santos, Flood Sung, Marina Vinyes, Zhenzhe Ying, Zekai Zhu, al. Kimina-proved preview: Towards large formal reasoning models with reinforcement learning. arXiv preprint arXiv:2504.1 10..5004.11..4 1154, 2025.
Freek Wiedijk. A synthesis of the procedural and declarative styles of interactive theorem proving. Logical Methods in Computer Science, 8, 2012.
Terry Winograd. Frame representations and the declarative/proceduuall controversy. In Representation and understanding, pp. 185-210. Elsevier, 1975.
12

---

Published as a conference paper at ICLR 2026
Huajian Xin, Z.Z. Ren, Junxiao Song, Zhihong Shao, Zhao Wanjia, Haocheng Wang. Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao. Qihao Zhu, Dejian Yang. Zhibin Gou, Z.F. Wu, Fuli Luo, and Chong Ruan. for Reinforcement Learning and Monte-Carlo Tree Search. In Proceedings of the International Conference Learning Representations, 2024.
Procckk--Pree---- Harnessing Proof Assistant Feedback
Ran Xin, Chenguang Xi, Jie Yang. Feng Chen, Hang Wu, Xia Xiao, Yifan Sun, Shen Zheng, and Kai Shen. Bfs-prover: Scalable best-first tree search for llm-based automatic theorem proving, arXiv preprint arXiv:2502.034: 03438, 2025.
Yixuan Even Xu, Fei Fang, Jakub Tomezak, Cheng Zhang, Zhenyu Sherry Xue, UIrich Paquet, and Danielle Belgrave. Neurips 2024 experiment on improving the paper-reviewer assignment.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066 2025.
Kaiyu Yang. Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem Proving with Retrieval-Augmente Lan- guage Models. In Proceedings of the International Conference on Neural Information Processing Systems, 2023.
Kaiyu Yang, Gabriel Poesia, Jingxuan He, Wenda Li, Kristin Lauter, Swarat Chaudhuri, and Dawn Song. Formal mathematical reasoning: A new frontier in ai. arXiv preprint 24Xiv:2412.1600 2024.
Xiao-Wen Yang, Zhi Zhou, Haiming Wang, Aoxue Li, Wen-Da Wei, Hui Jin, Zhenguo Li, and Yu-Feng Li. Carts: Advancing neural theorem proving with diversified tactic calibration and bias-resistant tree search. The Thirteenth International Conference on Learning Representations, 2025.
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning. arXiv preprint arxiv:2402.06332. 2024.
Zhouliang Yu, Ruotian Peng. Keyi Ding, Yizhe Li, Zhongyuan Peng, Minghao Liu, Yifan Zhang, Zheng Yuan, Huajian Xin, Wenhao Huang, et al. Formalmath: Benchmarking formal mathematical reasoning of large language models. arXiv preprint arXiv:2505.0277 2025.
Jingyuan Zhang, Qi Wang, Xingguang Ji, Yahui Liu, Yang Yue, Fuzheng Zhang, Di Zhang, Guorui Zhou, and Kun Gai. Leanabell-prover: Posttraining scaling in formal reasoning. arXiv preprint arXiv:2504.06122 2025.
Lichen Zhang, Shuai Lu, and Nan Duan. Selene: Pioneering automated proof in software verification. arXiv preprint 2rXiv:2401.0060 107663,, 2024.
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. MiniF2F: A Cross-Systen Benchmark for Formal Olympiad- Level Mathematics. In Proceedings of the International Conference on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
The Use of Large Language Models
15
Data Format Specification
15
C The Full Proof of Lemma linear -grdien
16
Training Details of Expert Iteration
16
Prompts for Premise Utilization
16
Additional Results
16
Error Analysis
16
F2 Results Under Varying Difficulty Levels
17
Case Study
18
Cases of Premise Utilization
18
G.2
Cases of Long-CoT
20
Limitations and Future Work
23
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In the course of this work, we made limited use of LLMs as auxiliary tools. Specifically, we used an LLM to assist in generating small utility Python scripts for tasks such as data preprocessing. formatting, and visualization. These scripts were only supportive in nature and did not contribute to the conceptual development, design of experiments, or the core scientific results of the paper. All key research ideas, theoretical analysis, experimental design, and writing of the main paper were carried out independently by the authors. The LLM was not used to generate or edit scientific content of the manuscript, nor was it involved in producing research hypotheses or interpretations. The authors take full responsibility for the correctness, originality, and integrity of content the paper.
B DATA FORMAT SPECIFICATION
The FormalML dataset stores each theorem a structured JSON format. Below is detailed schema with field descriptions.
"filename": string,
"line": int, #tactic_state_.eefrrrt string, "tactic": string, "tactic_state__fter":: string, string, "theorem_header": string, "formal_statement" string, "full_formal_ ttt.rormmalssttmmentttt... string, "retrieval":
string,
"library": string, "definition": string
]
Based the file COMtL/FoML/FoMMtthhii//re hotpaabiiiitt/ommmtsss.ss...m lean (Line 98), further pro- vide an illustrative example for some typical entries. In this line, the corresponding proof tactic is apply aemeasurable, aemeassrableeexp__hh. By extracting the pre- and post-tactic states, we can formulate the corresponding subgoal as follows:
theorem oetraatted_oommal_sstteeeee 1
(Ω Type u_1}
(m MeasurableSpace ?)
{p Measure ??)
[inst IsFiniteMeasure p]
(a b IR)
[X Ω + R}
μ1) (h tm (w !) Sp, X w E Set.Ice b)
(hx AEMeasurable
let : fun w => exp(I Xw);
(t', R), MEstrronlyteasurrable (e t') μ : sorry
Regarding the retrieval entry, records the library (e.g.. FoML), and specifies the definition of the retrieved lemma. For example:
lemma aemeasurable_expt
(X Ω R} (t RR
(hx AEMeasurable μ1)
Aesauuon (fun exp(1 X(w)))
15

---

Published as a conference paper at ICLR 2026
Table 5: Error type analysis for different theorem provers under pass @ 16 and pass @32.
Setting
Model
has _as_error
is valid with sorry is valid no sorry
SeepSeek-Prover-Ve
74.80
0.02
25.16
Pass 16: Optimization
SaepSeek-PPrver-.. (C6T)
63.16
10.68
15.49
Goedel-Prove
90.08
0.00
9.92
PresSeek-Prove-VIe
80.04
0.04
19.89
Pass @ 16: Probability
DeepSeek-VV V2 (CoT)
61.94
7.65
22.77
Goedel-Prover
85.20
0.00
14.80
DeepSeck-Prover- DeepSeck-Prover- (CoT)
75.09
24.87
Pass Optimization
62.98
10.76
15.49
Goedel-Prover
90.11
DeepSeek-Prowee Proek-PPrvee-II V1.5
79.88
0.04
20.04
@ass Probability
DeepSeek-V2 V2 (CoT)
61.82
7.61
22.96
Goedel-Prover
85.23
0.00
14.77
C THE FULL PROOF OF LEMMA Sinear-gradiem
private lemma linear_gradien V (EuclideanSpace I (Fin n)), HasGradientAt (fun x (EuclideanSpace (Fin n))
=>
(b . (A to x))) (AT b) x := by
intro rw intro apos
use a use apos
intro y rw
<- dotProduct_sub] rw [EuclideanSpace. sitee__ee_ttrrddtttee repeat rw [dotProduct]; simp [mul_comm] apply mul_nonneg; linarith [apos]; apply norm_nonneg
simp
D TRAINING DETAILS OF EXPERT ITERATION
Training used the AdamW optimizer with learning rate of 1e-5, cosine learning rate schedule, warmup ratio of 0.1, batch size of with gradient accumulation steps of 2 (effective batch size 16), and epochs, with bf16 precision and DeepSpeed ZeRO for distributed optimization. All experiments were conducted using GPU acceleration, and evaluation was carried out on the FormalMI. dataset.
E PROMPTS FOR PREMISE UTILIZATION
For premise utilization, we designed the following prompt, using DeepSeck-Pros V2 as an illustra- tive example.
F ADDITIONAL RESULTS
F.1 ERROR ANALYSIS
This section presents detailed breakdown error types from additional experiments conducted on three different theorem provers: DeepSeek-Prover-t V1.5, DeepSeek--Prover-V V2 (CoT), and Goedel-Prover. The results are categorized into three main types: has_error (Lean code with execution errors), is_vall valid_ww wi th_sorry (code that verifies but contains the sorry tactic), and ss__alid__ne -aaliddoo_sorry (correctly verified proofs). These proportions were calculated for both pass @ 16 and pass @ 32 metrics across different problem domains: Optimization and Probability. The tables below provide comprehensive overview of these results, allowing for direct comparison of the performance characteristies of each prover.
16

---

Published as a conference paper at ICLR 2026
The results indicate that long-CoT models such DeepSeek-V22 V2 tend to output "sorry" more frequently. but demonstrate lower rates Lean errors. Conversely, Goedel Prover seldom outputs "sorry" tactic but exhibit higher Lean error frequencies.
F.2 RESULTS UNDER VARYING DIFFICULTY LEVELS
We supplement the experimental results for all eight Phole-generation models across different difficulty levels, presented in Figure 6 and Figure 7 The results demonstrate statistically significant trend where all models exhibit performance degradation as the difficulty level increases. Among all evaluated models, DeepSeek- DeepSeek-Prover-VV Prover--r--- V2 demonstrates the most modest performance decline.
DeepSeek- Provel-PPreii V1.5
DeepScek- Proveek--Proe-22222 (CuT)
Goedel-
Optim. Masserg
Passiiik
Pass@
&
Pass@
Number Samples K
Number of Samples
Number Sumples K
Number of Samples
Propscck-PoovecV2
Gredkl-Proser
Prob.
Passsii
Pass@i
PassiGi
Number Samples K STP
Number of Samples
Number Samples
Samples K Goedel Prover
DeepSeek Prover V1.5
DeepSeek Prover V2 (CaT)
Pors@
Paoslig
Paxs@
is 12 oi
11
15 12 Number of Samples K
Number of Samples K
Number of Sumples
Number of Samples K
Difficulty Level:
Difficulty Level:
Difficulty Level:
Figure 6: Results of pass rate under varying difficulty levels (Part 1).
Ginedel-Prover-VV
Pravse
DeenSeel-- Provek--PPeeeee (noCoT)
Optim.
Passikk
Passiak
Ressien
is
Number of S'Smplls
Number of S'Smples
Number of Samples
Number of Samples
Goedel Prover V2
Leunabell Prover
Kiming Prover Preview 70
DeepSeek Prover V2 (noCoT)
Prob.
Passii
Parsick
PossiKK
11
Number of Samples Gieckl-Prove--V/
Number of Samples
Numher of Samples
Number of Samples
Leambelll Prover
Kiming
DerpSeck (noCoT)
Pussiak
ParsiiK
PussiOK
ALL
Number of Samples
Number of Samples
Number of Sumples
Number of Samples
Difficulty
Difficulty Level:
Difficulty
Figure Results pass rate under varying difficulty levels (Part 2).
17

---

Published as a conference paper at ICLR 2026
The prompt for premise utilization of DeepSeck-Prowe Prover- (CoT)
You can some of the following lemmas theorems: (all_definiige definitions)
Complete the following Iean 4 code: [formal_starn _sttttmtttttt]]
Before producing the Lean 4 code to formally prove the given theorem, provide a detailed proof plan outlining the main proof steps and strategies.
The plan should highlight key ideas, intermediate lemmas, and proof structures that will guide the construction of the final formal proof.
G CASE STUDY
G.1 CASES OF PREMISE UTILIZATION
We present case of STP where the problem was initially solved without premise utilization, but errors emerged after candidate premises were provided.
Problem
theor em eettracted.oorma__ssteeeeee
[n m N+}
[A: Matrix (Fin (m) (Fin (tn) R
[f:R)
{μроs :
[Aneo +ه
(x: Ex:uiiieeenSpace R (Fin (n))
(a:x E Set.univ)
(R)
( 0)
(y EuclideanSpee R (Fin (n))
(ydist: ||x y| S ttn (4)
(h_original_goal 1l Σ x, xl o xx
1 * Ex_1, (ly x_11 x_ 1l)l S sorry
Additional Theorems
theor em morm_one_prox-e
(lasso h= fun == / []У]]]
(x EuclideanSpace R (Fin n))
(xm EuclideanSpace IR (Fin
(tpos: 0 <
(μuрos: 0 (4)
(minpoint V 1: Fininn n, xm = Real.sign (x i) (max (abs (x i) - 14) 0))
prox_prop (t h) xm
theorem map_map
(g:S -
: ++ S):
(s.map f).map g= s.map (g.comp f)
theorem lnner_add_left
Fx
<<x + y, ス>> =<<x, 2>> <cy, 2>>
theoren em mul_pos
[a Ordinal]
(h: ())a)
(h2 mh::0= :)
0<a*
lemma id:
(fst ccmmp (inl y)) (add_neg_cancel 1)+
(snd cp).comp (Cochain.ofHom (inr 4)) (add_ _ero Cochain.offHom _
theoren em mub_sub
(a c: Ordinal):
(b =) b a * theorem id:
(id No.NonnNo
18

---

Published as a conference paper at ICLR 2026
theorem smul_add
[Zero RJ
[DistribSMul R V
(x HahnSeries R)
(yz: HahnModule V):
x- = x
theorem add @ddd_le_add_if__rte
b Ordinal]
Vn: N, + n
theorem add_right_cance
b Ordinal]
Nn:
=+
theorem abs_abs
[xy:
(h: IsCoprimee y)
IsCoprime lyl
Correct solution without premise utilization (STP)
import Mathlib
import Optlib
open Set Real Matrix Finset
open Set Real Matrix Finset NNReal
theorem extraateed {n m N+)
[A Matrix (Fin tm) (Fin fn) IR)
{p R}
(upos 0 < μ}
(Aneo : A + 0}
(x EuclideanSpace R (Fin (n))
(a : x E Set.univ)
(= R)
(epos >>>>> 0)
(y : EuclideanSpace R (Fin fn)
(ydist ||x y < E ttn μ) (hhoriginalgoaal 1p
p. Σ x_1, Ix x_11| S In Σ x_,, (ly x_1| - |Ix x_1 =
x, ly
cases' le_total (p Σ x_1, ly x_1l)
(μ * Σ x_1, |x x_1l) with h <;> simp_all [abs_of_nonneg, abs_oo_nonpos, mul_add, mul_sub,
sub sub_add_ed a____dd_e____ _ub_add__e__ad__sub]
<;> linarith
Incorrect solution with premise utilization (STP)
import Mathlib
import Optlib
open Set Real Matrix Finset
open Set Real Matrix Finset NNReal
theorem extrracted_forma formall_statemett
[n m N+}
[A: Matrix (Fin fm) (Fin fhn) [R]
(u:R) R)
{μpos (
[Anee A 0)
(x EuclideanSpace (Fin (n))
19

---

Published as a conference paper at ICLR 2026
(a:x E Set.univ)
[R)
(epos:e>0) (y: Euclideaapppae (Fin (n))
ttn/ (4) (ydist: ||x- < ///tt//
gh_original_ggal x_ by
Σ ly x
Ia * Ex_1 (ly
have : (s/ (t/ p by
have h 0 (ftn: R) == by exact_mod_cas (fun h by simp_all)
(Nat.pos_of_ne_zee)
have h2 : < by exact_mod_cast μрos
field_simp [h1, h2]
rw [div_le_iff (by pasitivity) <> nlinarith
have hi x 1x
μ xx111 x_ -xx1-xxx11:: have ha V_ 1, la
(ly -x
* x x_ by intro rw |abs_mmll abs_ abs__f__ps (by exact_mod_cass μрos)] simp_all [abs_sum]
-
have h2 14 1_1, - calc
μ * 1*I'x_1,lyx 1xx__-xx_11__
xx x_1, ||x - yll : by
exact wwuu_ll_mullpp _a____mu__o_nnnne___eees. won.nonnggeeee (by
exact Finset.sum_le__
(fun __ abs_ le__mm norm
i-i 1))) (by positivity)
-
= (y Σx_1: Fin fn, -|x bb
simp_all [Finnettssum_coontt]]
= (ttn ||x yl))> μ by ring
< (ftn (s ttn μ)) * = by gcongr
= by field_sst simp Inpos, suoss] ring
linarith
< simp_all all
linarith
G.2 CASES OF LONG-CoT
We present the output results of the Long-CoT model benchmark. The model exhibits redundant responses and suboptimal performance, which stems from issues such as hallucination and overthinking. Specifically, the Long-Cor Long-CoT model frequently misjudges the complexity of given subgoals, resulting in the generation of excessively fine-gra mee-grained subgoals (e.g.. through the use of the have tactic). This tendency overthink increases the risk of introducing erroneous statements, thereby undermining the validity of the output.
Problem
theor em Settraated_.ormaa_sstaaeeeeeee
[u_3, u_2, u_1l
[Ei Type u_1) (E2 Type u_2) [F Type u_3)
[inst: : NormeddddaCCmmrrruu En] [inst_1 InnerProductSpace IR E1]
[inst_2 FiniteDimensional H E1 [inst_3 NormedAdddComnrruup Ez]
|inst_4: InnerProducttspace Ez] [inst_5 FiniteDimensional Ez]
[inst_6 NormeddAddCommGroup F [inst_7] InnerProducttpace [R
[inst_ 8 FiniteDiienssinss R F]
(admm ADMM E E2 F) (admm_kkt Existance_dd _kkt admm)
(n: (+) f[xx E (a
univ) {yy / (a_11 yy E univ) [[aa [RB (a_2 0 & aa) (a_3 < bb) (abh aa 1)
(h_or iloorigina__gal.
aa (OptProblemAA E2) xx bb (OptProblem.Ai E2) yy + (OptProblemAe E1) (ADMM.x2 E1 -hatPred)-OOtProllee
E (OptProblem.k) Eg) yy + + bb) ((OptProblem.A E1)
(OptProblem.A: Eg) (ADMM.x2 xa E En.natPred)- Opt E E2))
20

---

Published as a conference paper at ICLR 2026
aa (OptProblem.A) E2) xx bb (OptProbIem.A) Ez) yy (OptProblem.Az E1) (ADMM.x2 E1 n.natPred) OptProblem.b E2 aa (OptProblem.A) E2) xx (OptProblemAAr E2) yy + ((OptProblem.Az E1) (ADMM.X2 n.natPred) - OptProblem.b E E2) ; by
Model Response
Prover-Preview--71
(Abbreviated; ~65 lines of repetitive reasoning and duplicate Lean code omitted from the original 83-line output.)
Internal Thinking
# Solving the Formal Statement in Lean 4
I'm going solve the problem of proving that [the goal equation] given the assumptions.
Let's think about this algebraically first. We have:
"aa + 1 (from the hypothesis "abh",
-0 <= aa and 0 <= bb" (from a_2 and "a_3")
The equation to prove: x (1*7)xx+++2 where 7 (Optrobllmmm OptProblem.b. ...
This just algebraic manipulation. Since aa + 1", have
(aa bb) = z". So the left side becomes x + z", which is the right side. In Lean, can prove this directly from h original goal. Let's complete the proof: Now have h original goal which states LHIS RHS. From algebra, (aa+bb)*z 1* -50 lines omitted: repeated explanation, substitution, linarith justification The complete proof is remarkably concise.
Proof
import Mathlib
import Optlib
Set InnerProductSpace Topology Filter InnerProduct open scoped Pointwise
theorem em exTractee__oommalastttteeeeeene u_2, (Er: Type u_1) [Ea Type
[F: u_2]
Type u_3]
[inst: NormmedddicCmmmrruu E1] [inst_1 1 InnerProducttSpace Er] [inst_2:
mll IR E.]
[inst_ Nommeeedddddddoooorrrr Eg] [inst_4: InnePPooucttpaaee Ez] [inst_5:
miniteDimensional IR Ea]
[inst_6 NormedAdddCommrrou F] [inst_] InnerProductSpace IR [inst_8:
Fini mnnl IR F]
(admm ADMM E Ez F) (adm__kkt: Existance_of _kkt adnm) (n N+) [[xx E (a: xx E
univ) E
(a_1 yy E univ) {|aa bb R] (a_2: aa) (a_3 bb) (abh aa bb = 1)
(h_ (h_original_goal. aa (OptProblem.A) E2) bb (OptProblem.A, E2) yy (OptProblem.Ag E1) (ADMM.x2 F n.natPredd-
OptProblem.b E E2
aa (CptProblem.A) E2) xx bb (OptProblem.A) E2) yy (aa bb) ((OpPProbemmA Ei) (ADMM.xg Ei n.natpred) OptProblem.b E₁ Ez)) aa (OptProblem.Ar E2) xx bb (OptProblem.A Ez) yy + (OptProblem.A E1) (ADMM.Xg E, F n.natpred)
OptProblem.b E Eu (OptProblem.Ai E2) bb (OptProblem.A) E2) ((OptPrablem.Ag E1) (ADMM.Xg E1 n.natpred) OptProblem.b E2) have :aa + bb : abh
linarith
h1]
21

---

Published as a conference paper at ICLR 2026
Model Response (DeepSeek-Prone Prover- V2 (CoT))
(Abbreviated; ~195 lines omitted from the original 237-line output.)
Internal Thinking
#if Detailed Proof and Analysis
#iif# Understanding the Problem
The hypotheses
in univ' and "a_1 1 : yy in univ" trivial (everything in univ).
The hypothesis h h_original coriginal_goal" exactly the conclusion we need, under simplifying assumptions.
3. The goal: simplify "(aa + bb) (...) to (...) using "abh bb = 1"
#ilt Simplifying Goal (aa + bb) 1= 1 because aa + bb by abh. Thus replace (aa + bb)
with
#### The Proof
by abh. Thus "aa bb) x X* (scalar mult linear).
know aa
3. Substitute into the original equation.
### Formal Proof Sketch
Replace (aa + bh) * (...)' with (...) using abh.
2. The result is exactly the RHS of the goal.
## Abstract Plan
Use abh replace (aa+bb) with 1 the scalar multiplication. Simplify LHS to RHS: (aa+bb)*X 1*X.
3. The rest follows trivially.
### btep-by-Step Abstract Plan
Simplify the scalar multiplication (aa+bb)*X = I*X using aa+bb=1. Substitute into h original_
3. Conclude: RHS of o_original goal is the goal.
#i# Lean hhvve' Statements theorem extracted formal statement 1.f. :. by have h_main (aa+bb)* 1*X = by sorry have h_final [goal] := by sorry
sorry
### Explanation
h main: scalar simplification. h final: substitute h main into h oooiiiinal goal. Final goal h final.
22

---

Published as a conference paper at ICLR 2026
Proof
theorem
(E1 Type u_1) (E2 Type
u_2) [F: Type 3_3)
[inst NormedddddCommrrru Ei] [inst_1 1 InnerProductSpace Ei] [inst_2:
Fini menesionnll IR Ei]
[inst_3: Noromecdoroom E2] [inst_4: InnerProductSpace Eg] [inst_5:
Finite@iimensional I Eu]
linst_ Normeud F] [inst_7 InnerProductSpace [
FiniteDimensional IR FJ
[admm: ADMM E₁ E2 F) [admm_kkt: Existance_or _okkt adrm) (n N+) fxx: E1 (a:
univ) (lyy E
(a_1 vy univ) {aa bb RD 2a_2:0 0 aa) 3: < bb) (abh: bb 1) (h_original_goal:
aa (OptProblemAA) E2) xx + bb (OptProblemAi E2) yy (OptProblemAq E1)
(ADMM.X2 Ei n.natPred)-
OptProblem.b E, E2
(OptProblemmA E2) +bb (OptProblemA) E2) +
(aa + bb) ((OptProblemAg E1) (ADMM.X, E, F n.natPred) - OptProblem.b E2 E2))
aa (OptProblemAA) E2) xx bb (OptProblem.A) E2) yy + (OptProblemAo E1) (ADMM.X2 E1 F n.natPred)-
OptProblem.b E2
aa (OptProblem.Ar E2) xx + bb (OptProblem.Ai E2) yy 1 ((OptProblem.Ag E1) (ADMM.x² E, n.natPred) OptProblem. E, E2) := by
have h_main (aa + bb) ((OptProblemAAz E1) (ADMM.X2 Ei F n.natPred) OptProbeem. E1 E2)
1 ((OptPrablem.Ag E1) (ADMM.xg E₃ n.natPred) OptProblem. E2) by
have h; aa +
nw [hi]
<:> simp [add_smul]
abel
have h_final: aa (OptProblemA: E2) xx (OptProblemArr E2) yy +
(OptProblem.Ag E2) (ADMM.x E. F n.natPred)
OptProblem.b E E2 = aa (OptProblemA: Eu) xx - (OptProblemAr E2) 1 ((OptProblem.a E1) (ADMM.x2 E1 n.natPred) OptProblem. E2)
calc
(OptProblem.A) E2) yy + (OptProblem.Ag E1) (ADMM.x: E1 n.natPred) - OptProblem. E₁ Ez
aa (OptPrablemAA) E2) xx
= aa (OptProblem.A, Ez) xx + bb (OptProblem.Ai E2) (aa bb)
((OptProblem.A E,) (ADMM.x2 Ei n.natPred) OptProblem. E, Ez) by
rw [hoo abel
aa (OptProblem.A; E2) xx E1) (ADMM.x2 E n.natPred)- - -OptProblemmbbbb E1 E2) by
(OptProblemAA) E2) yy + (O0tProbbemmAp
[h_main]; simp [add_smul]] abel
exact h_final
H LIMITATIONS AND FUTURE WORK
Our current benchmark primarily focuses on a subset main domains related to machine learning theory. In the future, aim to expand into more relevant fields. Additionally, plan to formalize certain aspects ML theory using Lean, with the goal encouraging more researchers to explore methods for assisting in formal theorem proving.
the use of
23


Reviews:
[{"review_id": "NPP1yMIJQ8", "reviewer": "Reviewer_7LrB", "summary": "This paper introduces FormalML, a benchmark of 4937 problems extracted from machine learning theory in a subgoal completion context (filling in sorries). The authors also use expert iteration to significantly improve the abilities of two open-source models.", "questions": "- The gains from expert iteration are very drastic for Goedel-Prover-8B and DeepSeek-V2, but the two models exhibit different patterns across sample budget. It is surprising that Goedel-Prover is showing uniform gains across all sample budgets, while DS-V2 shows very large gains for pass@1 and little gain for pass@32. Do you have any insights into this?\n- How do the insights from this benchmark transfer to theorem proving in general? Do you envision the target to be a model specialized for machine learning theory proving?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper curates a large and interesting benchmark for theorem proving in Lean sourced from existing and reputable data sources.\n- Almost all leading theorem-proving models are included and at multiple sampling budgets. \n- The authors analyze current models from various perspectives: low computational budgets, premise retrieval, and long CoT.\n- The authors show the effectiveness of expert iteration on their benchmark.", "weaknesses": "- The paper is limited to the domain of machine learning theory, which is somewhat narrow. Other than ease of data sourcing, there is no apparent reason why this needs to be the case. It would be great to expand the work to other domains as well. \n- Relatedly, while the findings are interesting, the authors do not comment on how the benchmark can inform and advance downstream research in formal theorem proving, especially in other domains. \n- As stated, the premise retrieval part of the paper does not consider embedding-style retrieval and only completion-based retrieval with the argument that using the same model enhances efficiency. However, a small embedding model could be cheap and produce better results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aD6xGr0tft", "reviewer": "Reviewer_mSD1", "summary": "The paper introduces FormalML, a Lean 4 benchmark targeting subgoal completion: filling in short but nontrivial proof steps within research-level ML theory proofs rather than producing full proofs end-to-end. The authors implement a Lean tactic, `to_theorem`, that extracts line- or segment-level subgoals from procedural proof scripts, and curate 4,937 problems spanning optimization and probability (with premise retrieval needs and complex contexts). A broad evaluation of recent LLM-based provers shows modest Pass@1, steep drops with difficulty, mixed gains from premise retrieval, and poor efficiency for long-CoT methods. They also show expert iteration improves performance, especially at low budgets.", "questions": "1. Concentration on ML theory (Optlib + FoML) may limit generality to other formal domains (algebraic topology, number theory, verification). Consider adding a small cross-domain diagnostic split (e.g., mathlib analysis/algebra lemmas) to evaluate transfer and prevent domain-overfitting.\n\n2. Consider adding refinement-at-most-N-steps interactive baselines (tool-use or repair loops) to gauge whether light interactivity beats one-shot generation.\n\n3. Table 3 lists “Goedel-Prover-V2-8B” with size \"7B\". Can you double check this is consistent?\n\n4. Slightly inconsistent terminologies: both \"Mcdiarmid\" and \"McDiarmid\" appear; both \"Godel\" and \"Goedel\" are spelt.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Novelly shifts focus from full-proof generation (competition-style) to research-adjacent subgoal completion, a practical interface between human sketches and formal verification. Benchmarks combine premise retrieval with complex research contexts (from Optlib/FoML), which most prior Lean benchmarks avoid.\n\n2. Introduces a symbolic extraction tactic (`to_theorem`) that converts procedural Lean steps into declarative subgoals at line/segment granularity, novel and broadly useful for dataset creation and tooling.\n\n3. Clear, reproducible dataset construction pipeline. Thorough experiments across tree-search and whole-proof families, multiple budgets, domain splits, and difficulty levels; plus targeted retrieval experiments and an efficiency metric. Additionally includes error analysis (sorry vs. Lean errors) and a concrete ablation on expert iteration showing meaningful gains.\n\n4. Provides a large and focused ML-theory benchmark likely to steer future model/training design (e.g., retrieval-aware training, efficiency-aware decoding).", "weaknesses": "1. Difficulty is proxied by proof length. That can correlate but isn’t identical to semantic difficulty. It would be good to add alternative signals and/or compare with human annotations.\n\n2. I would go more carefully about the claim that long-CoT \"shows no benefit\". It seems supported on FormalML but may be dataset-specific. Also, long-CoT prompts also prime models for full-proof search, not subgoal snippets. More work would be needed before concluding broadly on this.\n\n3. Authors note both single-line and multi-line segment extraction. It would be good to show how segment length affects success, retrieval load, and Pass@K. This would inform optimal chunking for future datasets.\n\n4. Authors evaluate models some of which may have seen Optlib/FoML/mathlib during pretraining. How did you guard against data leakage? Any checks that extracted subgoals aren’t near-duplicates of public training proofs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "apOQJDNRo6", "reviewer": "Reviewer_dUjc", "summary": "This paper proposes **FormalML**, a Lean 4-based benchmark for evaluating LLM performance in *subgoal completion*—a task designed to align with the vision of \"mathematicians providing high-level proof frameworks while models verify intermediate conclusions\" (as illustrated in Figure 1). The benchmark extracts 4,937 subgoals from optimization and probability theories using the custom `to_theorem` tactic, which decomposes procedural proofs into declarative subgoals. Evaluations of state-of-the-art theorem provers (e.g., STP, DeepSeek-Prover-V2) reveal underwhelming performance, though this is partially attributed to the subgoals’ nature: most can be solved with basic tactics like `rw` and `simp`, making them \"out-of-distribution\" for models not trained on such granular steps.", "questions": "1. The `to_theorem` tactic extracts subgoals from existing procedural proofs, but these subgoals are far simpler than the \"non-trivial intermediate conclusions\" in Figure 1. Do the authors have plans to refine subgoal extraction (e.g., integrating human annotations to mark meaningful research gaps) to better align with the paper’s core vision?\n2. Most subgoals can be solved with basic tactics like `rw` and `simp`, raising questions about their utility. How do the authors plan to evolve FormalML to include more complex, tactic-agnostic subgoals that reflect the \"hard technical details\" mathematicians actually struggle with?\n3. The paper notes that poor model performance is partially due to \"out-of-distribution\" subgoals. Have the authors tested whether training models explicitly on these `to_theorem`-extracted subgoals improves performance—and if so, does this training cause performance drops (i.e., \"catastrophic forgetting\") on standard full-proof generation tasks?\n4. Given that basic tactics already solve most subgoals, what concrete advantages does an LLM-based approach offer for subgoal completion? For example, can LLMs handle edge cases or premise combinations that basic tactics cannot, and if so, how is this validated in the benchmark?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Vision Alignment with Practical Research Needs**: The paper identifies subgoal completion as a critical gap in LLM-aided formal theorem proving, directly targeting the real-world workflow where mathematicians prioritize \"inspiration and intuition\" (high-level frameworks) over tedious technical details. This vision addresses a tangible pain point in mathematical research and formal verification.\n2. **Novel Benchmark for a Underexplored Task**: FormalML fills a void in existing benchmarks by focusing on subgoal completion (rather than full-proof generation) and integrating premise retrieval and research-grade ML theory contexts. It provides a structured, quantifiable foundation for evaluating LLM capabilities in intermediate proof steps.\n3. **Transparent Analysis of Model Limitations**: The paper acknowledges that poor model performance is partially due to the \"out-of-distribution\" nature of the subgoals (solvable via basic tactics like `rw`/`simp`). This honest attribution avoids overstating limitations and frames performance gaps as actionable, context-dependent challenges.", "weaknesses": "While the paper’s core vision of human-AI collaboration in formal theorem proving is compelling, the implementation of subgoal extraction via `to_theorem` is criticized for being overly simplistic and failing to fully realize Figure 1’s intended workflow. A key unresolved question is how training on these subgoals impacts models’ performance on standard full-proof generation tasks.\n\n1. **Overly Simplistic Subgoal Extraction via `to_theorem`**: The `to_theorem` tactic decomposes procedural proofs into subgoals tied to \"single-step tactic state transitions,\" which fails to capture the nuance of Figure 1’s vision. These subgoals are not the \"non-trivial intermediate conclusions\" mathematicians would need help with—instead, they are granular, tactic-dependent steps that lack the complexity of real-world research gaps.\n2. **Limited Justification for Subgoal Utility**: Since most extracted subgoals can be solved with basic Lean tactics (e.g., `rw`, `simp`), the paper does not fully explain why LLMs are needed for this task. The subgoals do not address the \"hard technical gaps\" that would truly liberate mathematicians, reducing the practical impact of the benchmark.\n3. **Incomplete Analysis of Training Transfer**: The paper evaluates model performance on subgoal completion but does not explore how training on these simplistic subgoals affects performance on standard full-proof generation tasks. This omission leaves a critical gap in understanding the tradeoffs of subgoal-focused training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7MJc9HeWar", "reviewer": "Reviewer_vdtg", "summary": "This paper introduces a Lean 4 theorem proving benchmark, FormalML, with problems extracted from two formalization projects about machine learning theory. Unlike existing benchmarks, FormalML focuses on subgoal completion, rather than proving theorems from scratch -- motivated by the fact that users of interactive theorem provers are generally interested in calling automation in the middle of a partial proof. Experiments show that the benchmark is still generally challenging for models at the 7B scale especially at Pass@1, with STP doing best at 63% Pass@32. The paper shows several other analyses of adding premise retrieval (with mixed improvements) and expert iteration (generally improving model performance).", "questions": "- What is the performance of simple automation tactics on FormalML?\n- Did you analyze the examples of theorems solved by DeepSeekProver without CoT that failed with CoT? What seem to be the most common failure modes with CoT when that happens (ie when it succeeds without it)?\n- In the retrieval experiment, how do models perform in a \"perfect retrieval\" scenario, where you give them exactly (and only) the premises used in the proof? That would give a sort of ceiling for the impact of retrieval here.\n- Was there any reason to not include other repositories (such as those used in the Expert Iteration experiment) in the benchmark? It seems to me that the pipeline should be very generally applicable.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper shows a novel angle on dataset construction for theorem proving, focusing on subgoals. The extraction pipeline should be easy to use in other repositories beyond the two used here. It's good to see this expanding the evaluation of prover models beyond mathlib and including projects from the broader Lean community, which helps it represent more use cases of Lean. I'd expect this benchmark to be used by future work in LLMs for theorem proving, which currently has significant momentum.\n\nThe paper is generally very clear and describes the pipeline and the results well.\n\nThe results and analyses are also interesting, with techniques that are generally adopted now in LLM reasoning (like long CoT in recent reasoning models) not always being beneficial. Thus, this evaluation is complementary to existing benchmarks, not just showing the same patterns (besides being well motivated).", "weaknesses": "I have some issues with the evaluation and the interpretation of the results.\n\nFor evaluation, it would be helpful to know how many of the goals are solvable with simple automation tactics in Lean, such as grind, aesop, or canonical. Users generally try these (or similar, more domain-specific, eg ring) tactics first when they want proof automation. This would help readers get a better understanding of what fraction of the goals are indeed challenging for traditional automation.\n\nFor the interpretation, I think that actually some of the models seem to do rather well. Again, depending on whether these subgoals are out of reach of traditional automation, 25% pass@1 actually does not sound bad to me. Plus, STP with Pass@32 already achieves 63%. Thus, I suggest being less bold on the claim that they \"are inadequate as practical tools\" if the reference for this is this pass@1 or pass@4 evaluation.\n\nGiven the rapid progress in this area, and progressive gains in efficiency, it's likely that smaller models on smaller budgets might achieve similar performance on this benchmark in not too long. While FormalML might be useful to track that progress, there's a risk of it being saturated quickly. It would help to perhaps include other projects that are at the moment significantly more challenging, to motivate the community to work on those. If the best results on level 3 goals (in Figure 4) are already over 60%, it's seems that the current methods are perhaps on track to tackle them with smaller budgets/models given sufficient efficiency improvements on all fronts.\n\nFor the evaluation of retrieval, it seems that there's no model doing retrieval itself, and the experiment is mainly testing the ability of models to use in-context statements. While this is a useful analysis, the descripiton is a bit misleading. For instance, the conclusion is that models \"often struggle with premise retrieval\". But retrieval here is done manually, not by the model, so the model is not struggling with retrieval, but rather in using the premises. Thus, here I suggest simply rephrasing the conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)