Process-Verified Reinforcement Learning for Theorem Proving via Lean
Authors: Minsu Kim
Se-Young Yun
Venue: ICLR 2026 Poster (2026)
Keywords: Formal Reasoning
Large Language Models
Theorem Proving with LLMs
Lean4
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We demonstrate that the Lean proof assistant itself can serve as a symbolic process oracle, supplying both outcome-level  and fine-grained tactic-level verified reward during RL.

Abstract: While reinforcement learning from verifiable rewards (RLVR) typically has relied on a single binary verification signal, symbolic proof assistants in formal reasoning offer rich, fine-grained structured feedback. This gap between structured processes and unstructured rewards highlights the importance of feedback that is both dense and sound. 
In this work, we demonstrate that the Lean proof assistant itself can serve as a symbolic process oracle, supplying both outcome-level  and fine-grained tactic-level verified feedback during training. Proof attempts are parsed into tactic sequences, and Lean's elaboration marks both locally sound steps and the earliest failing step, yielding dense, verifier-grounded credit signals rooted in type theory. We incorporate these structured rewards into a GRPO-style reinforcement learning objective with first-error propagation and first-token credit methods that balances outcome- and process-level advantages.
Experiments with STP-Lean and DeepSeek-Prover-V1.5 show that tactic-level supervision outperforms outcome-only baselines in most settings, delivering improvements on benchmarks such as MiniF2F and ProofNet. Beyond empirical gains, our study highlights a broader perspective: symbolic proof assistants are not only verifiers at evaluation time, but can also act as process-level reward oracles during training. This opens a path toward reinforcement learning frameworks that combine the scalability of language models with the reliability of symbolic verification for formal reasoning.

Published as a conference paper at ICLR 2026
PROCESS-VERIEES REINFORCEMENT LEARNING FOR THEOREM PROVING VIA LEAN
Minsu Kim KAIST AI minsu_ wasauukim@aaist.cco
Se- Se-Young Yun KAIST Al yunseyoung@kaist.acc kr
ABSTRACT
While reinforcement learning from verifiable rewards (RLVR) typically has relied on a single binary verification signal, symbolic proof assistants in formal reason- ing offer rich, fine-grained structured feedback. This gap between structured pro- cesses and unstructured rewards highlights the importance of feedback that is both dense and sound. this work, we demonstrate that the Lean proof assistant itself can serve symbolic process oracle, supplying both outcome-level and fine- grained tactic-level verified feedback during training. Proof attempts are parsed into tactic sequences, and Lean's 's elaboration marks both locally sound steps and the earliest failing step, yielding dense, verifier-grounded credit signals rooted in type theory. We incorporate these structured rewards into GRPO-style reinforce- ment learning objective with first-error propagation and first-token credit meth- ods that balances outcome- and process-level advantages. Experiments with STP- Lean and DeepSeek- Preeee-rrooev--eeee V1.5 show that tactic-level supervision outperforms outcome-only baselines in most settings, delivering improvements on benchmarks such as MiniF2F and ProofNet. Beyond empirical gains, our study highlights a broader perspective: symbolic proof assistants are not only verifiers at evaluation time, but can also act as process-level reward oracles during training. This opens a path toward reinforcement learning frameworks that combine the scalability language models with the reliability of symbolic verification for formal reasoning.
INTRODUCTION
Automated theorem proving (ATP) is one of the long-term goals of Al (Newell er 195 Com- pared to reasoning in natural language (NL), which often contains vague or ambiguous symbols, for- mal theorem proving based on formal logic and type theory provides technical and precise language for proving mathematical theorem (Churei, 1940 Fitting, 1996). Currently, interactive theorem provers (TTP) such Lean (He Moura et all 2015: Moura & Ullnch 202T), Isabelle (Nipkow et all 2002) and Coq (Barras et all (1977), serve as reliable and powerful tools for verifying mathematical proofs. Lean proofs are sequences of tactics, with automation handling routine arithmetic/logic and verification-so ITPs provide a middle ground between full automation and human guidance
By contrast, LLMs model next-token probabilities from large corpora via pre- and post-training, learning lexical correlations rather than rule-based symbolic manipulation (Brownett 2020). With further techniques such instruction tuning and Reinforcement Learning from Human Feedback (RLHF), LLMs have evolved to handle wide range of tasks, including question answering, sum- marization, dialogue (Ouyang al. 2022: Bai et all 2022). In particular, reinforcement learning (RL) approaches for reasoning tasks aim to enhance the model's reasoning ability by encouraging the generation of long chains of thought rationale (DeepSeek-At et all, 2025; OpenAl all, 2024)..
Compared to other reasoning tasks which often verify reward LLMs' response according to its final answer (Cobbe call 1021), the theorem prover can verify the correctness entire proof when LLMs respond with formal language. In this context, given the human-in-the-loop nature of ITPs, there have been growing attempts to use LLMs for formal theorem proving tasks (AlphaProof and AlphaGeometry teams, 2024 Trinh et all 202AI). LLMs act prover agents while theorem provers serve as verifiers, being used either at inference time-to search and validate tactics and premises-on for augmenting formal reasoning datasets with verified samples (Lample all 2022 Wang all

---

Published as a conference paper at ICLR 2026
2023: Ying et all MOZEE; et
Furthermore, some recent studies incorporated binary feedback from the Lean theorem prover into its online RL framework eX 20221)).
The tactic-based proof structure in Lean contains information relevant for reasoning tasks such as the positions of tactics or the nature of proof errors or failures. This structured information captures not just the outcome of proof, but also the underlying reasoning process. However, despite its potential, only a few works have explored incorporating this kind of fine-grained supervision into the training of LLMs (li al 2025). At the same time, recent RL approaches for reasoning have increasingly emphasized the use of process-based reward models (PRMs) to guide model behavior. While these models show promising performance, there is still lack of clarity around how PRMs are constructed, how the reasoning step or step reward should be defined, what training signals or datasets they should depend on (Yuan et all 2024, Luo all, 2024; Cui et all 2025)..
Unlike recent approaches that rely on PRMs or long NL. CoT (Lin et all (025abh), we directly leverage the Lean proof assistant as symbolic process oracle during RL training, without any natural-language guidance. For each generated proof, Lean provides (i) global outcome signal and (ii) fine-grained tactic-level feedback via info trees and error logs.
While fine-graine tie-ggrinedd tactic level signals are available from Lean, leveraging them effectively during RL training nontrivial. Lean outputs symbolic, tree-structured language feedback, such as proof states and error locations, whereas LLMs operate over autoregressive token sequences and learn from scalar rewards in RL. This representational mismatch creates a credit-assignment challenge: symbolic verifier feedback must be transformed into structured token-level training signals. To bridge this gap, we introduce a structured credit assignment framework for integrating symbolic verifier signals into an online RL objective, requiring three components: (i) formulation for in- corporating fine-grained signals, (ii) a principled rule for reducing Leans symbolic feedback into per-tactic scores, and (iii) mapping from per-tactic scores token-level advantages. We instanti- ate this pipeline using a tactic-level MDP, a first-error propagation rule grounded in Leans semanties, and a first-token credit assignment strategy.
We integrate the resulting per-tactic signals into Group Relative Policy Optimization (GRPO) style objective combining outcome- and process-level advantages. This enables precise, type-theoretic credit assignment grounded in verifier feedback without the need for an auxiliary PRM. Empirically, we found that incorporating symbolic verifier feedback into the RL objective consistently improves performance on MiniF2F and ProofNet, demonstrating the value of fine-grained verifier signals for reliable credit assignment in reasoning tasks. Our key contributions are as follows:
Formalizing Lean's 's Symbolic Feedback. We formalize Leans symbolic, tactic-level feed- back and reduce it into scalar training signals that enable fine-grained, token-level credit assignment.
Symbolic verifier-guided RL. We integrate outcome and tactic-level rewards derived from Lean into RL framework, providing dense and verifiable credit assignment.
Stable improvements on benchmarks. On MiniF2F and ProofNet, our approach consis- tently outperforms both outcome-only RL and vanilla baselines, yielding more stable and robust gains without NL notation or external PRM.
2 RELATED WORK
Automatic Theorem Proving An automated theorem prover typically consists of two stages.
The first is autoformalization, i.e., translating natural language mathematical statements into formal ones. LLMs have been used for this task eWi 2022), producing datasets such as M2n and Lean
(Zheng et all, 2022: Azerbayev et all, 2023; Xm et all 2024# |Ying et all 2024a). The second stage is proof generation, which can be performed step-by-step via tree search (Polu & Sutskever, 2020, Azerbayev et al]. 2024, Wu et all, 2/24 Xin
ProofNet. Deepseck-Prowe
alll 202215 or by generating entire proofs at once (Xin et all 41245: Ln all 20257) Existing approaches such as Lean-STaf and RMaxTS use Lean only step-checker during inference (Lm atll 20257; et 20246), whereas recent work has employed Lean as a whole-proof verifier during training (Wang et all 00258; Zhang et all 2025; Ren et all 2025). In this paper, we go beyond step-checking or whole- whole-proof verification by using Leans fine- greegerdded, tactic-level feedback as process-based rewards in online RL training.
2

---

Published as a conference paper at ICLR 2026
MM
Question
Policy Model
Lean Feedbock
Combined Advantage
[HEOREM AROME
T1: Intro...
Outcome Reward Whele Proof Correctoes verified by Lean
T2: apply
=(V)-0 -0 (Fail)
Parsing
T3:linarith..
Messages
Process (cactic level) parsed factics with fisst-error propagation policy
14: simp...
Firsterne all stepe irttead wringg
T1:
T3:linartth...
T4:
T5:
TS: ring..
Figure I: Overall framework for combining outcome and tactic level rewards via Lean: the proof Y is parsed into tactics , Ts. with Lean providing outcome feedback g(Y) and step-level errors (e.g., failure at T3 invalidates later tacties). Rewards then assigned to the first token of each tactic.
Reinforcement Learning in Language Models Beyond algorithmic advances such PPO
(Schulman et all 7) and GRPO al (024).. reward shaping and credit assignment remain
core challenges RL. Outcome-basee rewards (Cobbe ali. 202I)), though widely used in RLHF,
suffer from sparsity (Chan et all, 2022: Zheng et all 2023). Process-based reward models (PRMs)
address this by assigning step-level rewards (Lightman et al, 2023; Setlur all 2024; Kazemnejad
cLall ROZER Yoan 2022) Cureral 2025). Rewards can be defined implicitly (Cm etall 2025)
or explicitly via correctness annotations (Lightman et al, 2022)) or Monte Carlo rollout success rates (Wang et ar, ) but existing methods require large annotated datasets of step-level correctness.
This motivates our approach of leveraging the Lean prover itself as a process oracle, automatically verifying each step without human labels sampling. Additional discussion is in Appendix [
PRELIMINARIES
3.1
LEAN4
In Lean theorem proving, a statement to be established is represented as an initial goal and incre- mentally reduced into subgoals through sequence of tactics. Each tactic is parsed and elaborated by unifying it with lemmas or theorems in the library, generating new subgoals, and verifying their validity. The elaboration stage produces structured info trees that record proof states and error messages. Finally, the kernel ensures that the elaborated proof is theorrehoryyy consistent and constitutes valid proof for the original theorem.
Formally, let denote theorem statement provided to an LLM, and let Y be the response. a proof expressed in the Lean language. Write for the of Lean proofs and T for the set of tactics. For Y E V/ We view proof Y as a sequence of tactics (Ti, T2,... TN(Y)) parsed from the Ab- stract Syntax Tree (AST) and sorted by their starting positions, where N(Y) ) is the number of tactics in Y which aligns with the LLM's autoregressive generation process. Each tactic T; comprises corresponding tokens Yt in Y. Lean represents tactics as AST nodes; each node encodes the tactics syntactic structure and binding context, and may carry metadata such as error messages, proof states, and an index through which users (or training frameworks) can interact with Lean. If a tactic does not appear in the error log, then it has been elaborated successfully and passed Lean's 's internal rule-based verification, which guarantees that the step is locally sound under dependent type theory. Thus, any tactic not marked as an error constitutes a verified reasoning step-even it does not contribute to clos- ing the proof because some subgoals remain later tactics fail. In other words, Lean ensures tactic- level soundness, while proof-level completeness depends on whether the entire sequence resolves all goals. Leveraging this parsing and validation feedback, we define the parsing function y T to be the sequence obtained by sorting TacSet(YY): f(Y) (Ti...., T Tw(Y)). We also define the global scoring function y + [0, 1], where g(Y) if passes the Lean verifier and other- (1,d1,d))
wise, and the per-tactic scoring function & {(Y, (Y,T) E,, T TacSet(Y))
3

---

Published as a conference paper at ICLR 2026
Specifically.
1, if gY)
((V = di, else if g(V) = and T has no errors in Lean,
d2, else if g = and contains errors. Combining these components, represent Lean's is role via 4 Lean 10. (T (1,d1,d2)),
Lean(Y) (o(Y), ((,, ( (1
3.2 TACTIC-LEVEL MDP
We define a tactic-level Markov Decision Process (MDP) as the tuple M (S,A, F, m). The state space 8 contains partial formal proofs; each S is the proof prefix produced so far. The action space A coincides with the tactic space T; each action a A a single Lean tactic. The reward function
S x A - I assigns a tactic-level reward r(s. a). The transition function F x A + S is deterministic: Sj+1 = F(s,. Q2) = 3j + aj, where + denotes concatenation of the tactic a; the proof 8; time step Transitions are pure concatenations: Lean feedback affects r, not F. Let Stem S be EOS absorbing states. Let m E S be the initial state. In Section extend this formulation with outcome- and tactic-level rewards derived from the Lean theorem prover to obtain the final training signal.
3.3 CREDIT ASSIGNMENT IN REINFORCEMENT LEARNING
PPO assigns a sparse -ee--enee reward and propagates credit with value model with Gener- alized Advantage Estimate (GAE), reducing variance at the cost of extra learning complexity; full details are deferred to Appendix
In contrast, REINFORCE style GRPO optimizes directly from verifiable whole-trajectory rewards without a value model. For a prompt q, we sample responses { from Told and obtain rewards Ti A normalized, response-level advantage is applied uniformly to all tokens of
A Fi mean(r)
std(r)
The objective is (GR = T
Teaa (Wil ④) A., clip no (ye 4)
To (y ④)
min
I-c, (.) DKL(Toll
Toas (ya 4)
We make this dense and sound by injecting Lean-derived tactic advantages into GRPO: the outcome signal remains at response level, while tactic-level signals are mapped to tokens at the first token of each tactic (Sec. B.. This preserves GRPO's 's simplicity while addressing sparse credit.
4
METHOD
4.1 DEFINE TACTIC-LEVEL REWARDS
In the previous section, we modeled the correctness of proofs generated by the Lean proof as- sistant and parsed and verified each tactic within Y. We introduce reward mechanism that integrates both outcome-basse and process-based signals explicitly into the RL framework. Specifi- cally, we employ nutcome-based reward defined through function gYY), similar approaches used by (DeepSeek-AI et al], 2(25),, as global reward evaluating the entire proof. Additionally, we define process-based reward (YY, T), assessing the correctness validity at the level of individual tactics T E Y. Unlike implicit rewards or Monte Carlo estimations typically interpreted as process rewards, our method explicitly assigns correctness-baat based rewards each tactic step.
Assume that, analogous to the GRPO training rollout framework, given a question an LLM gener- ates a group of responses (Y1, Yy...., YG). Lean produces an outcome- based rewards:
Toutcome (Y;) = 9(v))
We define the nuteome-based advantage for any token V/., in response Y, as: Aoutcome, (Y.)) (CC)) .(C)))
((G)))
std(g(())..... g(C))
4

---

Published as a conference paper at ICLR 2026
Beyond binary outcome verification signals, we further design elaborate rewards based on the AST feedback produced by the Lean parser as in section B.T. We leverage this AST feedback to distin- guish between different kinds of tactics: for example. whether tactic is elaborated successfully (i.e., type-correct and locally sound), but may still leave unresolved subgoals that prevent the proof from being completed,
whether it has type errors parser-level mismatches. Thiis structured feedback allows us to assign more fine-grained process-based rewards. Since, we sorted the tree node contain- ing proof state by increasing order, we apply First Error Propagation rule when mapping Lean's feedback into tactic-level rewards as (Lu et all, 2024: Lightman et all 2025). Given sequence of tacties (Ti...., ,T): once an error is observed at Tj we propagate this failure all subsequent tactics, i.e., every Ta with l 2 j is treated as erroneous for the purpose of reward assignment.
1,
g(Y) = 1.
Let j = min ( : T contains an error).
(b)
= d1, g(Y) = O and k < j and no error, d2, (Y) O and k 2 j.
Unlike Lean, which parses proofs into a tree structure, the LLM generates proofs in an autore- gressive, causal manner. Once the first erroneous tactic 7₃ occurs, the continuation T.+1..... Tn conditioned on an invalid prefix, and therefore cannot constitute a valid reasoning process. First- error propagation enforces this principle by assigning error signals to all subsequent tactics, ensuring causal and type-theoretic credit assignment.
For any arbitrary response Y,, composed tactics Y; = [T,,, T2 ...). if we set 83, ag as the state and tactic Ti, at step j in response Y,, the process-basee based reward for tactie Tag is:
pproress (8), (j) "prroees, i.j = <Y, Tūş).
The corresponding process-based advantage is
Aroccess.i, ", mean(g(()).....(((.))))
Here, the subtraction of the mean outcome reward serves as a dynamic baseline reflecting the dif- ficulty of the problem q GRPO algorithm. If the problem is easier, the mean outcome reward becomes higher, thus penalizing incorrect proofs and their tactics more heavily. Conversely, for more challenging problems, the lower baseline imposes less severe penalties.
4.2 INTEGRATING LEAN INTO TACTIC-BASED REINFORCEMENT LEARNING
We then integrate these two types of advantages into the standard GRPO objective as follows.
Ai,t Aputcome, i, + 1 =
A process, i, a(i.))
where s(i, 1) E {1..... N} is the index of the tactic containing the token in Y,, first(T:.) (Ti)) indicates the first token the tactic. i.e,, we assign the tactic advantage only to the first token of each tactic. We applied the advantage Ai, into GRPO objective function:
L(0)=
E B (1)) (1
Σ
Σ
min (pre (pre clip(An, 1-,, 11,,,1+6)) e) A..) - Duu [To mlll
To( Vi.ce)
where Pi.t To (2, Yi.ce)
This formulation explicitly leverages both the global correct-
ness signal Aoutcome, i from proof outcomes and the detailed, tactic level correctness assessment Aprocess, a(i.))- By combining them into single advantage Ar,, we enrich the learning signal provided to the LLM based proof generator under the GRPO framework.
In Appendix DI we provide mathematical grounding and interpretation of our method via a potential- based reward shaping. We can interpret our credit assignment potential-basee reward shaping, where the potential function of a state defined as the probabil- ity that the current proof prefix be completed into valid Lean proof. We treat the error state as an absorbing state and set its potential to zero (see Appendix D for details).
discrete approximation of
Rather than propagating cumulative rewards across entire proof trajectory, collapse credit assignment to veriffed,, tactic-level signals. In general RL, a suboptimal step may still obtain positive return if later rewards are high, but in mathematical proof, this could be unsound: once
5

---

Published as a conference paper at ICLR 2026
Model
Model size
Budget'
MiniF2F-TTsst ProofNet- -est
Whole-Proof Generation Methods
DeepSeek-Phove--I..-SFT (Xin el all 2024F
7B
32
46.2% + 0.2
14.3% + 0.3
64
47.5% + 0.1
15.05% +
DeepSeei-Prover-V..SRR (Am stall 20247)
7B
48% =
16% =1
48.8% +
17.4%
0.6
Prove--- etl (0250)
7B
56.9% +
15.6% 0.5
64
57.9% - 0.5
16.7% 1
STP-Lean (Dong & Ma M0Z5
7B
32
55.9% 0.2
17.2%
64
56.7% 0.2
19.1%
0.4
STP-Lean + Ours
7B
32
57.1% + 0.8
18.6%
0.3
59.2% + 0.5
19% + 0.3
DeppSekk-Pove-V.554577
7B
54.9% = 0.7
16.8%
57.2% + 0.2
17.7% +
Deepseck-Prvee..1555S51 Ours
7B
32
56.3%
+ 0.6
17.6% 0.8
64
57.8% %
+ 0.4
18.5% + 0.3
Tree Search Methods
Lean-STaR
7B
64
X 50
46.3%
IntemLLM2-MMthh-Pin Plus--B (Ying el al, (0246)
7B x 32
100
48.8%
LnterLLM..5-SSepPPrrv DeepSeek P...er---P RMaxTS (Km eral RIZER)
7B
x 32 x 600
58.5% + 0.9
7B
3,200
55.0% + 0.7 21.5% + 0.8
Table 1: Budgets for whole-proof methods denote the sample budget (N) per problem; for tree- search methods, budgets denote the authors reported search expansions county. We compare with InternLM family and DeepSeck-Prover based tree search methods for fair comparison with our method. Bold indicates the best number within the whole-proof block. All our GRPO-style runs use the same STP subset, generations per query, and 15s Lean timeout. The notation μ + aindicates the mean and the standard deviation each.
tactic fails, all subsequent steps are invalid under first-error propagation. Empirically, return-based credit led to unstable optimization, as requires a value function or auxiliary estimator normal- ize scale and reduce variance. Hence, we adopt simpler formulation that combines normalized outcome-level signals with tactic-level rewards, without computing returns (See Appendix G
EXPERIMENTS
5.1 EXPERIMENTAL. SETUP
trained on 10k samples randomly drawn from the STP dataset (3.26M proofs). Proofs were verified via Lean through REPL interface, with 15s timeout per attempt. Baselines included STP-Lean and DeepSeek-Pront Prover-VIL V1,5-SFT, the latter additionally fine-tuned on 500k STP samples before RL. We used non-CoT prompt, response styles in etal 20246). We used tactic-level rewards d, = -0.05 and d2 = -0.1 for the main experiment. Full hyperparameters and training details are provided in Appendix
5.2
MAIN RESULTS
In Table the results on both the MiniF2F and ProofNet datasets demonstrate that training with tastic-based advantage via Lean consistently enhances model performance across most evaluation settings. For the STP-Lean model, our method improves MiniF2F performance up to +2.5%p (4), and ProofNet performance by +1.4%p @pass@32), while showing negligible decrease of -0.1%p on pass@64. 64. Similarly, for DeepSeek-Prowee V1.. our approach achieves marginal yet consistent increases across all benchmarks.
Budgets are not directly comparable: tree-search budgets count expansions/verifier calls at inference, whereas our budgets count whole-proof samples. Our aim to improve single-shot generation under different compute regime.

---

Published as a conference paper at ICLR 2026
Model
Model Size Budget MiniF2F- Test
ProotNet-
Test
STP + Outcome only (GRPO)
7B
55.7%
17.4% + 0.6
64
57.9% 0.5
19% + 0.3
STP +Tattic only
7B
32
55.6% 1 0.6
18.3% +
64
56.8%
0.6
17.9% + 0.8
STP + Outcome+Tactic RL (ours)
7B
32
57.1%
0.8
18.6% = 0.3
64
59.2%
0.5
19% 0.3
DeepSeck-Prover-VI... Prover-PPvvv-- V1.5+ Outcome only (GRPO)
7B
32
55.3% 0.4
16.8%
64
57.4% 0.4
17.6%
DeepSeek- only
7B
32
54.9% 0.7
16.8% 0.8
64
57.8% +
17.6% 0.3 0,3
DeepSeck-Prover-VI.. Outcome+Tacti Tactic RL (ours) 7B
32
56.3% 0.6
17.6% 0.8
64
57.8%
0.4
18.5% + 0.3
Table 2: Ablation study of STP-Lean with various verifier methods MiniF2F-Test and ProofNet- Test benchmarks.
Across both MiniF2F and ProofNet, leveraging Lean as a process-level oracle yields consistent, stable gains outcome-only reinforcement learning, without increasing training cost. In partic- ular, in Table M when applied to DeepSeek-Prowe Prover models, GRPO fails to yield any gains on the ProofNet-Test set, and in some cases underperforms relative to the supervised baseline. This highlights a key limitation of purely outcome- based credit assignment: it often lacks stability and fails to provide consistent guidance for proof search.
By comparison, tactic-level credit assignment yields more reliable improvements. While minor drops appear in some settings, generally provides stable gains over outcome-only GRPO. For ex- ample, on MiniF2F (pass @ 64), STP-Lean + Ours improves by +2.5%p over the bascline, compared +1.2%p with GRPO. As shown in Table D and 2 tactic-based training consistently matches or surpasses both the supervised baseline and GRPO. Importantly, this stability comes with almost no extra cost: since both methods already use REPL interactions with Lean, the additional sorting and scoring overhead is negligible.
Compared to strong search-based baselines (e.g.. InternL families, DeepSeek-Prove Prover- RL+RMaxTS), our single-shot, whole-proof training approaches their reported accuracy (e.g.. 59.2% vs. 58.5% pass@66 64 on MiniF2F) while avoiding large search-time compute.
0.37
Qutcorre + TBci AdAntagg
0.30
335
D4ttome Deeeens
3.30
3.35
21
0.97
Outrme beTE Marcop
3:3
0.31
359
350
200
250
69
68)
0.000
Tactic Aduutagee Only
3.004
2035
0.900
20.0
Ouccome Betk lacri édhontage Only (JFFO Backee
20.5
0.500
2400
1.00
(lccaal Steep
330
(10b3) Shop
330
Figure 2: Training dynamics showing (a) outcome reward, (b) tactic reward, (c) entropy, and (d) mean of response length during reinforcement learning.
7

---

Published as a conference paper at ICLR 2026
5.3 ANALYSIS
The Role of Outcome and Tactic Rewards. Integrating both outcome-level and tactic-level sig- nals yields more effective learning than employing either signal in isolation. Outcome-only RL, as in GRPO, is constrained by the sparsity of binary feedback: improvements are gradual and the final performance plateaus at relatively low level (Figure ("))) In contrast, tactic-only training provides dense feedback but lacks global objective, resulting in premature convergence. When combined, outcome rewards serve as a global objective function, while tactic rewards provide local credit as- signment, enabling both rapid progress and higher performance. This complementary relationship is further reflected in Figure (ib), where tactic-only supervision's 's tactic reward plateaus, but outcome- tactic combined rewards continue to increase steadily. The results in Table supports this finding: outcome signals enforce proof-level correctness, while tactic signals supply verifiable intermediate feedback; only their integration consistently improves performance across benchmarks.
Entropy and Proof Length. The use of fine-grained rewards influences exploration not by in- discriminately broadening the search space but by focusing learning on more informative decision points. As shown in Figure [(c), outcome+tactic training converges to lower entropy than tactic-only and outcome-only settings, indicating that the policy becomes more decisive as training progresses. This does not correspond mode collapse: Figure [(d) shows that the average proof length remains stable across all methods, suggesting that the performance gains are not attributable to trivial length- ening of outputs. Instead, denser intermediate rewards appear to reduce the need for broad stochastic exploration, guiding the model toward more efficient proof strategies.
Tactic to Token Level Credit Assignment. After defining tactic-level rewards, next step is how to distribute them across tokens. In our main method, the tactic advantage is assigned only to the first token of the tactic. For comparison,
conducted ablations where the tactic advantage was instead (i) distributed to all tokens of a tactic, (ii) assigned only to the last token, (iii) keep first token reward distribution, but additionally choose 10% tokens within the tactic with respect high entropy. As Wang el all (2025d) showed that high entropy tokens could be reasoning drive tokens, we speculated that this method can automatically select the tokens for serving as fork in formal reasoning. Assigning credit to the first token of each tactic achieves the most stable and consistent improvements, as evidenced by Table B Alternative strategies do not yield comparable gains and in some cases even degrade performance. This outcome aligns with the semantics of Lean proofs: the first token corresponds to the tactic keyword (c.g.. int ro, apply, have), determining the subsequent proof strategy and constrains the structure of subgoals. Concentrating credit on this decision point enhances the models ability to select tactics appropriately, resulting in more reliable downstream reasoning. This finding is also aligned with (Fang et al. 2025), showing that focusing on key tokens during training improves performance on long-contex, tasks.
Reward Strategy for Tactic-level Feedback. For tactic-level feedback to be effective, it must reflect the sequential dependency of proof construction, account for task difficulty, and distinguish between partially correct and erroneous steps. The first-error propagation rule ensures that once an error occurs, subsequent tactics are treated invalid; removing this rule significantly reduces perfor- mance (Table 9) because once the first eтror occurs, the remaining tactics are evaluated in an invalid context and cannot salvage correctness. Incorporating a iffficulty-normalizer baseline further stabi- lizes training, while its absence leads to degraded results. Finally, differentiating penalties between partially correct tactics and outright erroncous ones proves essential: collapsing these into a single penalty dj = de yields inconsistent outcomes- improvements on MiniF2F but declines on ProofNet. These results indicate that an effective tactic-level reward scheme must combine sequential error propagation, difficulty-aware normalization, and differentiated penalties in order to provide stable and semantically faithful learning signals. In the sensitivity analysis of Appendix D assigning dif- ferent values di and da leads robust performance, tending outperform the GRPO baseline and yielding the strongest improvements on MiniF2F.
Effeet of Verification Timeouts When using Lean as a verifier, long proofs can lead to excessive verification time, so we introduced timeout thresholds of 5, 10, 15, and 30s (Figure B). A 5s limit gave the worst results, since even relatively simple proofs often exceeded this window and produced too few valid reward signals. In contrast, 10-30s yielded much stronger performance, with 15s giving the best overall balance. Interestingly. 10-15s sometimes outperformed 30s despite the shorter allowance. We attribute this to the fact that discarding overly complex proofs biases training
8

---

Published as a conference paper at ICLR 2026
Model
Model Size
Sample Budget
Minil-2F- Test
Proof
Test
All tokens
56.3% 0.6
18.1% 0.8
64
57.8% + 0.7
18.1% = 0.8
Entropy-based
7B
32
56.4% 1 0.2
17.9% 0.8
64
57.1% + 0.5
18.5% 0.3
Last token
7B
32
56.7% + 0.9
17.2% =
64
57.5% + 0.6
17.7% 0.5
First token
7B
32
57.1% % 0.8
18.6% = 0.3
59.2% 0.5
19% 0.3
Table 3: Ablation study of STP-Lean on how to distribute tactic-level advantages across tokens.
Model
Model Size
Sample Budget
MiniF2F Test
ProofNet Test
No First Error
7B
32
56.4% = 0.9
17.4% + 0.3
64
58.2% + 0.7
18.3% + 0.3
No Baseline
7B
56.7% 0.2
17.9% 0.3
57.4% + 0.7
18.3% + 0.5
Same tactic reward
7B
32
57.7% 0.2
17.6% 0.6
64
58.7% 0.8
18.1% 0.6
Outcome+ Tactic RI. (ours)
7B
57.1% + 0.8
18.6% 0.3
59.2% + 0.5
19% + 0.3
Table 4: Ablation study on reward strategies for tactic-level feedback in STP-Lean. Additional experiments include removing the first-error propagation policy (No First Error), removing the base- line extraction (No Baseline). and using equal penalties for all tactics (Same tactic reward).
toward shorter and more efficient proof strategies. This effect is amplified in our setting because we evaluate non-CoT responses purely by Lean verification (without NL commentary): longer outputs are not only slower to check but also more error-prone. As a result, shorter verification limits encourage the model to generate concise, canonical proofs, which hypothesize leads to better generalization at test time.
69
(M)
20
15
passie
pasug33
p225064
pasuge4
59
10
100
490
timeout
timeout
Figure 3: Ablation study of STP-Lean on different Lean verification timeouts (5, 10. 15, and 30 seconds) during outcome+tactic based training. We report evaluation performance on the MiniF2F and ProofNet benchmarks (b), and the maximum response length observed during training (c).
Qualitative Analysis We conduct qualitative analysis better understand the differences be- tween our tactic-reward-based approach and the baseline STP model. Specifically, we examine proof's from two benchmark problems: Imo 1960 p2 the MiniF2F benchmark and 14 from ProofNet (See Appendix H). Table 3 presents the proof generated by our tactic-reward model, while Table E shows the corresponding proof from the STP model. The key difference in the first example lies in how the upper bound x < 45/8 is established. The STP model attempts to use the nonlinear inequality tactic nlinarith, which results in an error. By contrast, our tactic-reward model learns to penalize such invalid tactic choices. Instead, it carefully applies previously proven assumptions and intermediate lemmas before invoking nlinarith, thereby producing a correct and more robust proof. The second example comes from the ProofNet benchmark (1 14 exercise). As shown in Table III the tactic-reww rewird model begins by normalizing the problem using a rewrite tactic. In contrast, the baseline model in Table [[ skips this normalization step and directly attempts inequality manipula-

---

Published as a conference paper at ICLR 2026
tions, which ultimately causes the proof to fail. Analysis for Failure case is in Appendix H These anecdotal examples illustrate plausible mechanisms; for definitive evidence, see Table II.
CONCLUSION
We introduced a reinforcement learning framework that uses the Lean proof assistant as process- level reward oracle. Unlike prior outcome-only methods, our approach leverages Lean's parsing and validation to provide both global outcome signals and fine-grained tactic rewards, integrated into a GRPO objective. This enables denser, verifiable credit assignment: outcome rewards enforce proof-level success, while tactic rewards guide step-level reasoning. Experiments on STP-Lean and DeepSeck-Provee Prover-Vo.5 show consistent improvements MiniF2F and ProofNet, with stable gains achieved by assigning tactic rewards to the first token each tactic and first error propagation method. Overall, proof assistants serve only checkers at inference but also structured feedback sources during training, pointing toward more stable and effective RL for reasoning.
LIMITATIONS
We not compare against learned PRMs, as they rely on natural- -anuuugl CoT supervision and large annotated datasets that are not yet available for Lean. Our models also generate pure Lean proofs without long CoT, leaving open how to design tineggraieed rewards for long-form reasoning. In addition, tactic rewards in method were fixed scores (d1, d2), which proved effective but somewhat sensitive across different models and datasets. Developing general advantage estimators and lcage-scale lactic-level datasets remains important future work.
ETHICS STATEMENT
This research does not involve human subjects, personal data, sensitive information that could raise ethical concerns. All experiments were conducted on publicly available formal mathematical datasets, and the proposed models are solely trained and evaluated for automated thcorem proving tasks.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide a comprehensive description of our and train- ing process Section For further details such as hyperparameter, version Lean, we intro- duced it in Appendix B. We utilized Huggingface and trl library for our experiments, and we plan to release the source codes to facilitate future research.
ACKNOWLEDGEMENT
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) [No. RS-2022-11200311 De- velopment of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manip- ulation of Everyday Objects, 90%] and [No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST), 10%].
REFERENCES
AlphaProof and AlphaGeometry teams. solving international mathematical URL
Ai achieves silver-medal standard olympiad problems.
Google DeepMind
Blog, July 2024.
nd. google /iscccive//
ai-sol ves mo mw-mmobllloooooooo ems aaris--at--i attaat-ss s1 ver-meda leveel Accessed: April 2025.
Zhangir Azerbayev, Bartosz Piotrowski. Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-locn mathe- matics, 2023. URL https::
abs /2302 12433.
10

---

Published as a conference paper at ICLR 2026
Zhangir Azerbayev, Hailey Schoelkopf. Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q. Jiang, Jia Deng. Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2024. URL /
org/ abs/2310..... 22.5//33001113111
Kaito Baba, Chaoran Liu, Shuhei Kurita, and Akiyoshi Sannai. Prover agent: An agent-based framework for formal mathematical proofs, 2025. URL htt /arxiv. 2r2/bb5/2006.
192.3
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Haffield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shanua Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Ólsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204 055862 2022. URL https:/
Corpus ID: 188778
Bruno Barras, Samuel Boutin, Cristina Comess, Judicaël Courant, Jean- Chun-hhisttpphe Filliâtre, Ed- uardo Giménez, Hugo Herbelin, Gérard Huet, César Muñoz, Chetan Murthy, Catherine Parent- vigouroux, Christine Paulin-Mohring, Amokrane Saïbi, and Benjamin Werner. The coq proof assistant reference manual Version 6.1. 06 1997.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL wwtp:/xwvworaaaaoooo http://222...22///2222002.51..11 2005.
Meng Cao, Shuyuan Zhang, Xiao-Wec Chang, and Doina Precup. Scar: Shapley credit assignment for more efficient rlhf, 2025. URL https://aax.ii wttpaaa//2555.222000
Alex J. Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in re- inforcement learning from human feedback, 2024. URL 00782
/2402.
Alonzo Church. A formulation of the simple theory of types. Journal of Symbolic Logic, 5(3): 114-115, 1940. doi: 10.2307/2266866.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxix:
prc/ abc/abs//e 221.. 14168.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. URL https: /ttp://arvvvv..ooo
Bps/ / 2DS/2502.01455
Leonardo Mendonça de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. The lean theorem prover (system description). In CADE, 2015. URL https::/ /api.
semant wwtpooooo.....///oraaaaaaa000000
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu. Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang
11

---

Published as a conference paper at ICLR 2026
Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning. 2025. URL www.hhttt:/saatt /ttpp//////xxoo.o////////////
12948.
Kefan Dong and Tengyu Ma. Stp: Self-play Ilm theorem provers with iterative conjecturing and proving, 2025, URL
00212.
Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling?, 2025. URL attp:////aaaa..o//aaaaaaaaaaa/// 23771.
Melvin Fitting. First-order logic and automated theorem proving (2nd ed.). Springer-Verlag. Berlin, Heidelberg, 1996. ISBN 0387945938.
Xingguang Ji, Yahui Liu, Qi Wang, Jingyuan Zhang. Yang Yue, Rui Shi, Chenxi Sun, Fuzheng Zhang, Guorui Zhou, and Kun Gai. Leanabell- prover-v2: prover:: Verifier-integrated reasoning for formal theorem proving via reinforcement learning, 2025. URL 08644
Albert Q. Jiing, Sean Welleck, Jin Peng Zhou, Wenda Li, Jiacheng Liu, Mateja Jamnik, Timothée Lacroix, Yuhuai Wu, and Guillaume Lample. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs, 2023. URL
12283
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment, 2024. URL htt.h:tts://aat
abs /aabb///41... 01679
Guillaume Lample, Marie-Anne Lachaux, Thibaut Lavril, Xavier Martinet, Amaury Hayat, Gabriel Ebner, Aurelien Rodriguez, and Timothée Lacroix. Hypertree proof search for neural theorem proving, 2022. URL https:: /ttp:///axivv.v..ii www.ii.org/aae wttp:orraabbb.......
Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, Keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang, Xiang Wang. Junyang Lin, and Dayiheng Liu. Cort: Code-integrated reasoning within thinking, 2025. https:::
09820.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. 2023. URL
Haohan Lin, Zhiqing Sun, Sean Welleck, and Yiming Yang. Lean-star Learning to interleave thinking and proving, 2025a. URL https: //arxvv org. abb /2407 10040.
12

---

Published as a conference paper at ICLR 2026
Yong Lin, Shange Tang, Bohan Lyu, Jiayun Wu, Hongzhou Lin, Kaiyu Yang, Jia Li, Mengzhou Xia, Danqi Chen, Sanjeev Arora, and Chi Jin. Greder- A frontier model for open-source automated theorem proving, 2025b. URL bttps://arxiv.cm /ttp///////////////////////aaa// http//bs/22333 /2002 07640.
Yong Lin, Shange Tang, Bohan Lyu, Ziran Yang, Jui-Hui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng, Jiawei Ge, Jingruo Sun, Jiayun Wu, Jiri Gesi, Ximing Lu, David Acuna, Kaiyu Yang, Hongzhou Lin, Yejin Choi, Danqi Chen, Sanjeev Arora, and Chi Jin. formal theorem proving with scaffolded data synthesis and self-correction, 2025c. n
Scaling
7/arxiv.
2208 03613.
Jianqiao Lu, Yingjia Wan, Zhengying Liu, Yinya Huang. Jing Xiong, Chengwu Liu, Jianhao Shen, Hui Jin, Jipeng Zhang, Haiming Wang. Zhicheng Yang. Jing Tang, and Zhijiang Guo. Process- driven autoformalization in lean 4, 2024. URL https: aaxx www.vv..rg.aaa... 2406. UT940.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. URL https::/aax.v..rrrtttrr abs/
Leonardo Moura and Sehastian Ullrich. The Lean 4 Theorem Prover and Programming Language, pp. 625-635. 07 2021. ISBN 978-3-030-79875-8 doi: 0211007/97----03--99877777
A. Newell, J. C. Shaw, and H. A. Simon. Empirical explorations of the logic theory machine: case study in heuristic. In Papers Presented at the February 26-28, 1957, Western Joint Computer Conference: Techniques for Reliability, IRE-AIEE-CMM '57 (Western), pp. 218230, New York, NY, USA, 1957. Association for Computing Machinery. ISBN 9781450378611 1. doi: 10.1145/ 0555567.1455603 URL https: 11t.:/01111111/15555553333333311 1555.
Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Confer- ence on Machine Learning, ICML 199, pp. 278287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122.
Tobias Nipkow, Markus Wenzel, and Lawrence C. Paulson. Isabelle/HOL: a proof assistant for higher-order logic. Springer-Veriie Verlag, Berlin, Heidelberg, 2002. ISBN 3540433767.
OpenAI, Aaron Jaech, Adam Kalai, Adam Lever, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas. Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Gianbattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren. Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg. Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen,
13

---

Published as a conference paper at ICLR 2026
Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhow, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang. Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agar- wal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Tay- lor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. hRtps:t:://aae httpa//aww..orrrsssa/aaaaasa//// /ttp//aabb//11.111222
Azim Ospanov, Farzan Farnia, and Roozbeh Yousefzadeh. Apollo: Automated Ilm and lean col- laboration for advanced formal reasoning, 2025. URL https://arxiv /ab//////////// abs 2505.
05758
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https::://............tt...ts /ttss///axvvvv.. wwwp:iv.orrraaeoeooo wtt.rr//a////......o 02155
Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving, 2020. URL
abs /2009. 033993.
Z. Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang. Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, Z. F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu, Wenjun Gao, Daya Guo, and Chong Ruan. Deepseek-prover--22 Advancing for- mal mathematical reasoning via reinforcement learning for subgoal decomposition, 2025. URL
nttps::/arxxnl orq/ aos www./////......1
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL
wwtp://wwwwwwbb/oo.ooooooo 06347]
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning, 2024. URL pttps://arxiv http://w.rrg/b/////////e 2btp//bb///220110
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL htttps:/aaxxiv.org/
2402
03300
Trieu Trinh, Yuhuai Tony Wu, Quoe Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476-482, 2024. URL kttps: hwwp:///www....... wwwur..
com/ http//ttccee//41155-11
023-
Haiming Wang, Ye Yuan, Zhengying Liu, Jianhao Shen, Yichun Yin, Jing Xiong, Enze Xie, Han Shi, Yujun Li, Lin Li, Jian Yin, Zhenguo Li, and Xiaodan Liang. DT-solver: Automated theorem prov- ing with dynamic-tree sampling guided by proof-level value function. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pp. 28632-12646 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18.18653/v 1tt33/1112000 1/2011/22zza acl-long..76. URL
https: // /ttps::/acclase aantoo wtw.::lo...oo..o..... wwt.oyy..oo.....eee/// -ong. 7067l
14

---

Published as a conference paper at ICLR 2026
Haiming Wang, Mert Unsal, Xiaohan Lin, Mantas Baksys, Junqi Liu, Marco Dos Santos, Flood Sung, Marina Vinyes, Zhenzhe Ying, Zekai Zhu, Jianqiao Lu, Hugues de Saxcé, Bolton Bailey, Chendong Song, Chenjun Xiao, Dehao Zhang, Ebony Zhang, Frederick Pu, Han Zhu, Jiawei Liu, Jonas Bayer, Julien Michel, Longhui Yu, Léo Dreyfus-Schmide Lewis Tunstall, Luigi Pagani, Moreira Machado, Pauline Bourigault, Ran Wang, Stanislas Polu, Thibaut Barroyer, Wen-Ding Li. Yazhe Niu, Yann Fleureau, Yangyang Hu, Zhouliang Yu, Zihan Wang, Zhilin Yang. Zhengying Liu, and Jia Li. Kimina-prover preview: Towards large formal reasoning models with reinforce- ment learning. 2025a. URL ptEps://aare:
ora/ abs/ /204 11354.
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations, 2024.
URL https: org/obgg///ooo / 2312. 08935
Ruida Wang, Yuxin Li, Yi R. Fung, and Tong Zhang. Let's reason formally: Natural-formal hybrid reasoning enhances I1m's math capability, 2025b. URL https: arxiv. http//aawiiv.......a.....
127703
Ruida Wang. Rui Pan, Yuxin Li, Jipeng Zhang, Yizhen Jia, Shizhe Diao, Renjie Pi, Junjie Hu, and Tong Zhang. Ma-lot: Multi-agent Ican- long chain-of-thought reasoning enhances formal theorem proving, 2025c. URL htt::::: /arxiv. www. htcp//bs/22555 2503 03205.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang. Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song. Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning, 2025d. URL
2106.
Yuhuai Wu, Albert Q. Jiang, Wenda Li, Markus N. Rabe, Charles Staats, Mateja Jamnik, and Chris- tian Szegedy. Autoformalization with large language models, 2022. URL prEtps://arxi:
org/abs/ 22005//225... 12615.
Zijian Wu, Suozhi Huang, Zhejian Zhou, Huaiyuan Ying, Jiayu Wang, Dahua Lin, and Kai Chen. 1nternlm2.5-steeprrove Advancing automated theorem proving via expert iteration on large-scale lean problems, 2024. URL
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseck-prover, Advancing theorem proving Ilms through large-scale synthetic data, 2024a. URL httLhttss//aiitt www.oorg//bbb//aoee///.ee
abs abs///b//2233/ /2405 14333
Huajian Xin, 7. 7. Ren, Junxiao Song. Zhihong Shao, Wanjia Zhao, Haocheng Wang. Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang. Zhibin Gou, Z. F. Wu, Fuli Luo, and Chong Ruan. Deepseek-prover-v133 Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search, 2024b. URL
a0s/ 2408/2008.0 08152
Huaiyuan Ying, Zijian Wu, Yihan Geng, Jiayu Wang, Dahua Lin, and Kai Chen. Lean workbook: A large-scale lean problem set formalized from natural language math problems, 2024a. URL
https://arxix www.or///aooo. wttp///////...../// 03847.
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang. Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang. Kai Chen, and Dahua Lin. Internlm-math: Open math large language models toward verifiable reasoning. 2024b. URL
abs 2402. 06332.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL Actps: http://axxvvo..////. abs/ 14476
15

---

Published as a conference paper at ICLR 2026
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels, 2024. URL https://axxiv. orq/ abs/ 2412. 01981.
Jingyuan Zhang, Qi Wang. Xingguang Yahui Liu, Yang Yue, Fuzheng Zhang, Di Zhang, Guorui Zhou, and Kun Gai. Leavabell-prover: Posttraining scaling in formal reasoning, 2025. URL
attps://arxixs wwwxiv.org/ab..... www.abs////a.oooooooooooo.oo.... 2504.
Kunhao Zheng. Jesse Michael Han, and Stanislas Polu. Minif2f: cross-system benchmark for formal olympiad-level mathematics, 2022. URL /t..ttts.//aaav..r//////200....
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang. Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language models part Ppo. 2023. URL /tttDs///a7x...or///ab////// 04964
Thomas Zhu, Joshua Clune, Jeremy Avigad, Albert Qiaochu Jiang, and Sean Welleck. Premise selection for a lean hammer, 2025. URL
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A ADDITIONAL RELATED WORKS
Automatic Theorem Proving An automated theorem prover typically consists of two stages. The first the process of translating mathematical statements written in natural language into formal statements. Wu all (2022) utilized large language models to translate mathematical questions into formal languages such as Isabelle and HOL.. This process, known as autoformalization, is primar- ily used for constructing datasets intended for formal reasoning. Benchmarks or training datasets such as MinilF2,, LeanWorkbook, ProoINet, Deepseck-Prover have employed LLMs to translate natural language mathematical statements into formal expressions, contributing to the creation of gugh-quality formal reasoning datasets (Zheng et al. 2022: Azerbayev et al. 2023% Xin all 20225 Ying et all 2024R)
The second stage involves generating a formal proof from the translated formal statement. This proof generation process is typically divided into two approaches: one involves step-by-step inference, such as tree search during inference time (Polu & Sutskever, 2020: Azerbayev et all, 20244 Wi et all
2024; Xmn 2024D), and the other generates the entire proof at once (f 20244: Lin
2025)) Ospanov et all (2025); Waan et all (2024G) IBaha el all (20255 use Lean compiler as agent
for complementing formal reasoning ability of LLMs, while (Dong Ma (025) enhances formal
reasoning by augmenting problems via conjecture. (Dang et al., 2023) presents a unified framework that combines both autoformalization and proof generation a single pipeline.
Existing methods such as Lean-STaR and RMaxTS ct 22025 Xin et 20246) utilize Lean as a step-checker during inference, generating steps sequentially and searching optimally via tree search to find valid proofs. In contrast, in this paper, similar to (Wang et all, 2025a: Zhang et all 2025; Ren etall 2025: Tretall 2025), we utilize Lean as a whole-proof verifier during the training stage. Additionally, beyond merely providing correctness checks for the entire proof, we leverage Lean's parsing and elaboration capabilities to validate each individual tactic step, integrating this step-level validation into the training process. In other words, we employ the Lean proof assistant as a macces--based reward model for validating the correctness of each reasoning steps. (Lightman atall 20233..
Unlike prior work that leverages dense feedback from proof assistants, our research takes a different perspective: we rely solely on the sule-based signals of the symbolic engine, without introducing any natural language. Approaches such as Lin el all (2025am)); Wang et all (2025H); ell (21225) exx ploit natural language reasoning In contrast, our method improves performance exclusively through reward signals provided by Lean, without any reliance on natural language.
form of annotation enhance LLMs formal reasoning abilities.
Reinforcement Learning in Language Models While developing applying algorithms such as PPO (Schulman et all, 2017)) and GRPO (Shao et all 2022) plays a sigmificant role in reinforcement learning, reward shaping and credit assignment
central challenges in reinforcement learning. (Cobbe et all, 2021) introduced a reward model based on the outcome of a response. However, similar other areas of RLHF, this approach suffers from the limitation of sparse rewards (Chan ctall 20244 Zheng et all, 2023).
To address this, process-based reward model (PRM) assigns step-level rewards during inference to guide rationale generation (Lightman et all, 2023), and can also be used to reward responses during training (Setiur et all, 2024 Kazemnejad all 20ZE). (Yuan ef all 2022) al 2025) derived an implicit PRM from the ORM without any data annotation or additional training. When assigning scores to reasoning steps, (Lightman all 2123)) defined the reward as the correctness of each step, which required substantial human annotation effort. (Wang et all 2722)) instead adopted Monte Carlo approach, defining the score of step as the proportion of successful rollouts originating from that step. While recent PRM approaches show promise in natural language reasoning, they require large annotated datasets of step- correctness. To the best of our knowledge, no such dataset exists for Lean formal theorem proving, making direct comparison with learned PRM baseline infeasible. This further motivates our approach of leveraging the Lean verifier itself as a process oracle. In contrast, our process-based reward leverages the Lean theorem prover to automatically
17

---

Published as a conference paper at ICLR 2026
verify the correctness of each step, thereby eliminating the need for human annotators sampling many proofs steps.
Our work also be interpreted through the lens of reward shaping (g et 1999) Prior ap- proaches have explored different mechanisms for distributing reward signals: Chan el all (20244 leverages the internal attention patterns of LLMs assign higher weights to important tokens, Cad stall (2025) employs Shapley values to allocate credit across actions, and Kazemnejad et all (2(Z4)
Monte Carlo rollouts to estimate and distribute rewards over intermediate steps. In contrast,
our method relies on an external parser-the Lean theorem prover-to parse tactics and assign reward the first token, thereby implementing a form of credit assignment.
B EXPERIMENTAL DETAIL
Data. We randomly sampled 10k instances from the STP dataset (3.26M total) for RL training. we applied an additional supervised fine-tuning step on 500k STP
For SeeySeck-Prove--I... samples before RL, since the vanilla model produced low-quality proofs during RL training.
Verification. We use Lean 4.9.0-re1 for all experiments in the paper. During training, used a REPL (read-eval-prins loop) interface with Lean to verify proofs and assign outcome- and tactic- level rewards. Each proof attempt given maximum of 15 seconds for verification; longer runs were treated as failures (both outcomes, tactic rewards are zero).
RL configuration. For GRPO training, we used G generations per prompt, sampling temper- ature 0.9, KI. coefficient 0.04, clipping 0.2, and the DAPO upper bound 0.28 (Yu all 2025). Tactic-level rewards were fixed at dj -0.05 and d2 -0.1 for partially valid and erroneous tactics in the main experiments, respectively. All experiments used non-CoT prompts, following
Xin aL (202216).
Training details. We fine-tuned the models with LoRA (rank 64, 64) using bf16 precision. The AdamW optimizer was used with learning rate of 1.0 x 10-5 Maximum response length was to 1024 tokens during both training and evaluation.
Evaluation. For decoding we used temperature 1.0 and top-p 0.95. We re-evaluated all baselines under the same non-CoT and budget settings (32/64 samples). All reported results are from the final checkpoint.
Compute.
Training was conducted on x NVIDIA A6000 GPUs, requiring approximately 21-23 hours.
C HYPERPARAMETER ABLATIONS ON d;
Setting
Model Size
Sample Budget
MiniFFF Test
ProofNet Test
STP-bascline
7B
32
55.9% = 0.2
17.2% t
64
56.7% + 0.2
19.1% + 0.4
GRPO baseline
55.7% 1
17.4% + 0.6
64
57.9% + 0.5
19% = 0.3
dj 0.05, d2 0.10
7B
32
57.1% 1..1%++00 0.8
18.6%
64
0.5 59.2% 0.5
19% +0%+..3
dj d2 0.10
37.7%
17.6%
64
58.7% + + 0.8
18.1% + 0.6
di 0.05, d2 00.5
7B
32
57% 0.4
17.6% 0.3
64
59.2% + 0.5
18.6% + 0.8
Table 5: Ablation study tactic-level penalties d1 d2. We compare outcome-only GRPO base- line with three variants (d,, dg) settings. Results are reported as pass@32 and pass @ 64 (%) on MiniF2F and ProofNet test sets. The experiment is couducted with STP-Lean model.
18

---

Published as a conference paper at ICLR 2026
This ablation shows that introducing gap between dj and d2 makes the method more robust: per- formance remains consistently above the GRPO baseline, with stable gains across different penalty scales and especially clear improvements MiniF2F.
D
PROMPTS
For training and evaluation, used non-COT evaluation followed by (Dong & Ma. 2025) and (Xinn
R (024)). The examples are introduced Table B.
Prompt Template
Complete the following Lean code: :n'n ... lean4 \n( header H formal_ statement))
Table 6: Prompt template used in training and evaluation. We selected non-COT generation which appropriate with our MDP setting
Prompt Example
Complete the following Lean 4 code:
"'-eenn
import Mathlib import Aesop
set option maxteartbeats 0
open Bigoperators Real Nat Topology Rat
theorem theorem_exercise_ 2011 2 257 (G Type*)
[Group G] [Fintype
G (h Fintype. G
2)
(x
GG)
x
( x y G, x ==y x) let p x
G, a2)
a G, an =
by
Table A training sample used training and evaluation. We selected non-COT generation which is appropriate with our MDP setting.
E CREDIT ASSIGNMENT IN REINFORCEMENT LEARNING
Let Ye be the t-th token of R denote the reward model, To represent the policy model, and Tet be the reference model. L denote the response length and be coefficient controlling the distance between the policy and the reference policy. In PPO, the token-level reward at po-
sition defined as: rx y1) = R(x,y)1(y+ = L) B Bog
To (y+llz) where the non-zero <=((w(=))
reward A(x 1/ is assigned only to the last token. For all other tokens, only a KL divergence penalty is applied via a log ratio log ne(sew>) Direct usage rewards can lead high vari- ance; therefore, PPO reduces variance by utilizing learned value model V.This value network assigns value each token Yts from which the Teifpral--ifferrncce (TD) error computed as: 5, rt + >VV (3 (ye) where is discounted factor. Then, the advantage for each token is recursively calculated as follows: Ar 8z. A, = 8, + 7dAr41, for = L 1, L 2.... 1. Subsequently, because the computed advantages A, can exhibit high variance during exploration, normalization or similar techniques are applied, resulting in the final adjusted advantage A, This adjusted advantage is then utilized in the PPO loss defined as:
(y 2x)
Te (
1,, A1 1
I. CLIP (0) = E2 min
A,, clip
(2)
Teaa (Vr x))
Tea (1) x)
19

---

Published as a conference paper at ICLR 2026
In contrast, DEINFORCE-bassee methods such GRPO and RLOO have proposed algorithms that optimize policies directly from verifiable rewards without requiring a value model, due to concerns about the computational cost and estimation capability associated with training value networks.
GRPO generates multiple response groups { (t)) for a given question q from an old policy Fold- Subsequently, a reward function outputs reward = { [r()) ) for each response group. If we set Vi. as t-th token index of response ys The advantage for yi.r. Ar, is then computed by normalizing these rewards as follows: As. = "==mean(r)
std(r)
This advantage is uniformly assigned to each token Vi.t constituting the response y4. Subsequently, this identical token-level advantage is utilized in calculating the following loss:
LGRPO (0) BxPP(Q). (u))2
min wo(wne ②)
A clip To(Mn. (g) Ka.a (VE. q)
1 14c) Ars) 33 DKi. (To Teae) }
(3)
G
STa. (Vi,a (g)
As our goal is to integrate Lean-based fine-gr fine-grained rewards to GRPO algorithm, method keep balance between value-based PPO's fine-gres ane-ggrined credit assignmethod and GRPO's s simple unifrom credit assignment approach.
20

---

Published as a conference paper at ICLR 2026
F GENERATED PROOFS EXAMPLES
imo_1960, _160_p21000 in minif2f from tactic-reward--rrined model
import Mathliß ib import Aesop
set_option maxHeartbeats 0
open Rig0perators Real Nat Topology Rat
theorem imo 11e__9660__0 p2 (x \mathbb(R\) (h_0 0 1 x)
(b_1 (1 Real (1 + 2 x)) 2 0)
(h_2
/ (1 Real.sqrt (1 2 x)) 2 < + 9)
-1 / 2)
45 / :- by
have h_3 0
(1
x) := bb linarith
have h_4 4
Real.sqrt (1
x) Real. sqrt_nonneg(11 + x)
have h_5 5 0
(1 2
(1 +
x) :- by pesitivity
have h_6 0 (1 Real.sgrt (1 + 2 x)) 2
apply sq_ of ne zero
intro h apply h_ nlinarith
constructor
nlinarith Isq sqrt (show 0 + from by linarith), __
have h_7 x < 45 / := by
apply it_of_not_ge intre hge have 8 + x ~ 2 / (1 Real. (1 2 x)) 2 2+x+9 == by rw [ge_iff_le] Iw [ic_div_iff 6_6]
mlinarith [sq_sqrt (show 0 + x from by linarith), bge] linarith linarith
Table 8: Proof of imo 1960_p2 from MiniF2F generated by tactic-reward-trained model.
21

---

Published as a conference paper at ICLR 2026
smo__160__p2 p2 in minif2f from baseline model
import Mathlib
import Aesop
set_option maxtleartbeats
open Bigopcrators Real Nat Topology Rat
theorem imo _mo__960_p2
\mathbb(R))
(h_0 :
x) (h_1 (1 Real sqat. (1 x)) ~ 0)
: / (1 Real.sqrt
2 x)) ~ 2 9)
-11 2)
x
by
have h_4 (tt (1 x))
have h_3 0
by positivity x) Real Real.sgrt_nonneg
Real.sqrt by linarith
have h_ Real.sqrt
2
have
x=
[Real. fqeallsqrt_le_iff..
constructor
à linarith
à mlinarith have h_7 4 2 (1 Real. (1 + x)) 2 2
by assumption
have 8 8 [div it_ is h_3l at h_7 nlinarith have h_9 ldiv ise h1 at h
/ 2) := by
45 8 1= by
nlimarith
exact h_8, h
<;> resumption
<;> assumption
<j> assumption ion
<;> assumption
<;> assumpt lon
<;> assumption
<j> essumption
assumption
assumption
Table 9: Proof of imm__160_pp2 from MiniF2F generated by the baseline STP model.
exercise 1xercise___14inn in ProofNet from facticrreward-rrained model.
import Mathlib
open Topology Filter Real Complex Topclogicallpaace Finset open scoped Bigoperators
theorem exercise_l_14
(2 : \mathbb(C]) (h abs 2 1)
: (abs (1 + 2) 2 (abs (1 - 2)) 2 4 : by
rw
Complex. saapllex.norns_ee__aae
simp
complex..ormsqg.co h]
ring_nf
<;> simp [h, htt.eex.ooommm_eeddbbe
<;> ring_nf
Table 10: Proof exercise 1 14 from ProofNet generated by our www.eeewaard---iiieeee model.
22

---

Published as a conference paper at ICLR 2026
exercise
1_14 __ciie___14444 ProofNet from baseline
import Mathlib
open Topology Filter Real Complex Topologiical$pace Finset open scoped Bigoperators
theorem exercise_l_14
(2 : \mathbb(C])
abs 2
: (abs (1 2) ^ 2 (abs (1 2)) : by
have h_C : abs ^ 2
by simp [h]
simp [pow_twe, abs_add, abs_sub, h_0] ring_nf
simp_all [sq_abs]
Table 11: Proof of exercise_] 1 1_14 from ProofNet generated by the baseline STP model.
23

---

Published as a conference paper at ICLR 2026
G
RESULTS ON RETURN-BASED ADVANTAGE
Model
Model Size
Sample Budget
MiniF2F Test
ProofNet Test
Return
7B
32
55.3% + 0.4
18.1% t 0.3
64
57.5% + 0.2
18.6% + 0.3
Outcome+ Tactic RL (ours)
7B
32
57.1% = 0.8
18.6% = 0.3
64
59.2% + 0.5
19%
0.3
Table 12: Ablation study on reward strategies for return- based advantage.
sovurage
2119
Octore bac Advarage
5130
3250
6.0
25
0.511
125
300
Figure 4: Training dynamics showing (a) outcome reward, entropy, and (c) mean of response length during reinforcement learning in our method and return-based method.
24

---

Published as a conference paper at ICLR 2026
H FAILURE CASE AND LIMITATION OF OUR METHOD
amo_2019 __p in MiniF from tactic-reward-trained model.
import Mathlib
import Aesop
set_option maxHeartbeats
open BigOperators Real Nat Topology Rat
theorem imo iio__2019_pp (f \2 \2)
(\forall a b, (2 a)
E (E (a ))))
liff (Vforall 2,
0)
\n (lexists c, \forall 2, fz constructor intro h
0) := by
intro z
have h_1
have h_2
have h_3
have h_4
have h_5
have h_6
have h_7 :=
have h_8 :=
have h_9
2
simp at h_1 h_2 h_3
norm_num at h_ h_2
b_7 8
h
omega
intro
intro a
have h_1
(2
have h_2
have h_3
cases' h_1 with h_1 h_1 <;> cases! h_2 with h_2 h_2
\n <;> cases' h_3 with h_3 h_3 <;> simp_all
<;> omega
Table 13: Proof of imo _mn__201_____1111 _p in MiniF2f generated by our decticrrwward-traine model.
Consider function f 2 satisfying
Va, b E Z, The task is prove that necessarily of the following holds:
f(2a) + 2 (bb) = {(f(a+b)).
ii V2 Z, f(z) 0,
(ii) 3c E Z, V f(z) 2z cc
Our model first introduced the assumption
h Va E Σ,
+ ((2)))2(())))) (((at)
and then instantiated it at several concrete pairs to create hypotheses h; (e.g.. hn = 1(0, 0). h 10, ..). After some local simplification steps (e.g.. simp, norm_numm attempted to close the goal using the omega tactic, decision procedure for Presburger arithmetic (linear integer arithmetic).
However, the omega call produced the first Lean error. While our method correctly assigns the d2 penalty to this failing omeqa tactic under first-error propagation, it does not penalize the preceding tactics (intro, have, simp) because they elaborate successfully and thus appear locally valid. In other words, although introducing and instantiating h; is not logically incorrect, this route is strate- gically unproductive for this problem: the remaining goal still involves quantifiers, disjunction, and
25

---

Published as a conference paper at ICLR 2026
uninterpreted function which lie outside omega's theory. Consequently, current scheme only punishes the terminal failing step and fails to capture that the earlier (locally successful) steps did not make meaningful progress toward solving the global goal.
I LARGE LANGUAGE MODEL USAGE
In preparing this manuscript, we made limited use of large language models strictly for writing assistance. Specifically, used ChatGPT-5 and Gemini-2.5 to improve grammar, enhance clarity of expression, and polish the overall presentation.
J MATHEMATICAL GROUND FOR TACTIC REWARD
In this section, we provide a conceptual interpretation of tactic-level rewards using a simple value model and potential-basee reward shaping. Our goal is not to claim formal optimality guarantee, rather to clarify how the structure of our discrete Lean-based rewards aligned with an idealized concept of proof success under First error propagation assumption with potential function.
J.1 DEFINE VALUE FUNCTION
Consider Lean proof trajectory
Ti
81 T2
Tv SN.
Su
where denotes the prefix of tactics (Ti.... T--1). 1. and sn refers to completed proof.
adopt the modeling assumption already used the main paper:
From the first error propagation, once the first erroneous tactic occurs, the proof can longer be repaired into a valid Lean proof.
Formally, let be the index of the first tactic for which Lean reports error. Then all states with t 2 lie in an absorbing failed state. Instead of assuming independent Bernoulli errors, we consider more general and realistic model with conditional valid probabilities. For valid prefix 8k-1, let
((*-1)) 1) = Pnoo error at step k 8k-1 is valid).
Under the first error propagation assumption, we define the value function the probability of eventually producing a valid proof from valid prefix is
V(s.) = P(success
8t) II ((*k-1)- kD8
Along a valid trajectory, we have
V(s)) V(s++1) ((())
2 V(sr;;
so Vss)) is monotone increasing until errors founded. if the first error occurs at step j. the success probability collapses to zero:
V(s;) =(.. =0.
Qualitatively, this yields the following structure:
For a successful proof, V(s)) increases from small value so V(sN) 1.
For a failed proof, (8x) increases along the correct prefix, and then drops to 0 at the first error and stays at 0 afterwards.
Thus, the ideal value function encodes (i) Monotone growth along error-free prefixes and (ii) Irre- versible collapse after the first error.
This structure motivates using stronger positive credit for tactics on an error-free prefix and negative neutral credit after the first failure.
26

---

Published as a conference paper at ICLR 2026
J.2 POTENTIAL-BASED REWARD SHAPING WITH V(s)
The value defined in previous section suggests a way to define potential-based shaping. In an ide- alized MDP setting where the environment state is exactly 8, and the agent has access the value function (*), one could define a potential
=
where f is any monotonically increasing transformation (e.g.. f(v) v). A shaped reward can then be written as
r T Toutcome, + +**0(*)+1) 5x (st);
where T Vontcome,t is the sparse end-of ofdoof reward derived from g Under the assumptions of (gg atall 1999), such potential-based shaping preserves the set of optimal policies.
Intuitively, using Φ* (s) = f(V(s)) means that the potential is highest on globally successful trajec- tories, increases along error-free prefixes, and collapses after the first error, similar with the structure of (s) in previous section. The corresponding
term
(8++1) - x5* (8t)
acts as a local improvement signal: is positive along valid contexts, negative when the value collapses the first error, and zero afterwards.
In our setting, however, we neither assume access to the true V()). We therefore view this potential- based construction a normative model that suggests the qualitative shape of desirable local credit signal, rather than as source of formal optimality guarantees.
J.3 LEAN-BASED DISCRETE APPROXIMATION AS QUANTIZED LOCAL SHAPING
In practice, we did not estimate V(s) 4* (*(s) explicitly. Instead, we exploit Lean's symbolic feedback (AST errors, first-error propagation) to construct discrete tactic-level scores.
For proof Y, with first error index j (if any), recall that we define
1, if gYY) 1,
(Y,
d1, if g(Y) and < j, d2, if g(Y) and 2 j.
1> d1 d2.
as the process-level reward for tactic T,
Conceptually, (Y, T;) is a coarse, Lean-driven quantization the ideal local improvement signal suggested by the value model. States globally successful trajectories receive the highest score (1); states on error-free prefixes of failed proofs receive an intermediate score (dz): and states at or after the first enror receive the lowest score (d2). This partitions trajectories into three regions whose ordering (success > pre-error > post-error) aligned with the ordering of (s)) implied by previous section.
Assuming 7 = and finite horizon, any such per-step process reward sequence can be written as difference of state potential. For fixed trajectory Y of length N, define t backwards by
(sNN O, (((++)) (ss)) processs,s N=N 1,...,0.
By construction,
so the total shaped reward becomes
Toutcome.t Tprocess,t Toutcome, ((s++1) (ss)).
This potential & is not intended as an estimate of the true value function V(s); rather, it is an implicit potential induced by discrete Lean-based scoring rule. The key point that its level sets respect the same qualitative ordering (success > pre-error > post-error) as the ideal value model, providing theoretically motivated yet practical shaping signal.
27

---

Published as a conference paper at ICLR 2026
J.4 DISCUSSION AND LIMITATIONS
Our analysis suggests the following:
Under a first-error propagation assumption, an ideal value function V(s) for Lean proofs increases along error-free prefixes and collapses to zero after the first error.
Our discrete tactic-level scores ( TT) 1, d1, d2 can be viewed quantized local improvement signal, capturing this qualitative structure without estimating V(s) explicitly.
For any given trajectory, the resulting process rewards Vprocess,t potential-based shaping term 7process, ((+++)) - ((s)) for a suitable potential ④.
be written as
Consequently, our use potential- shaping should be understood as theoretical framework that explains the structure of our rewards and motivates design choices, rather than as strict proof that our procedure preserves the optimal policy for the original sparse outcome reward. Empir- ically, we observe that this verifier-informed, discretized shaping leads to more stable training and consistent improvements over outcome-only GRPO on MiniF2F and ProofNet. We emphasize that do not claim any formal optimality guarantee for this shaped reward in large-scale LLM and Lean setting; the potential-based perspective is used purely as conceptual framework for designing and interpreting our tactic-level rewards.
28


Reviews:
[{"review_id": "lHk8IXleQw", "reviewer": "Reviewer_kG7S", "summary": "The paper studies reinforcement learning for formal mathematics in Lean. It proposes to view Lean compiler feedback not just as a binary final reward, but as more fine-grained process supervision: a large penalty is assigned to the tactic that produces the first error message. Continued RL experiments on DeepSeekProver-V1.5 and STP models suggests that this method performs better than standard outcome-only RL.", "questions": "- values for $d_1$ and $d_2$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is sensible and clearly described, the experimental setup is sound, the experimental results are good.\n- The problem of credit assignment is highly relevant: both in code generation and (natural language + formal) math, the state of the art uses outcome-only reinforcement learning, which forgoes the finer, localized signal that compilation error messages or generative reward models provide.", "weaknesses": "- The presentation is overly convoluted (e.g. suggestion: remove TacSet etc, write: \"viewing a proof $Y$ as a sequence of tactics $(T_1, ..., T_n)$ parsed from the AST\") and lacks crucial details (which values for $d_1$ and $d_2$ are chose? which one is larger?).\n- The method could be grounded more theoretically: first come up with a credit assignment scheme, then explain that any token-level advantage sequence can be integrated into GRPO. The optimal way of assigning credit is via $r_t + V(s_{t+1}) - V(s_t)$ where $V$ represents the expected value under the policy $\\pi$. Rewards $r_t$ are still sparse at the end of the sequence. Under the modeling assumption \"no recovery after the first error\" that the authors rightly put forward in the context of Lean, $V(s_{t+1})$ is zero after the first error. $V(s_0)$ is estimated from the $n$ rollouts in GRPO. Inbetween, there is incomplete information to fill in for $V$ precisely, but the constant extrapolation suggested in the paper is likely suboptimal. Under a model of independent Bernoulli-distributed error variables at each tactic, the value $V$ should *increase* over the progression of the trajectory until reaching a value of 1 upon success or until the first error. I do not expect the resulting assignment scheme to produce fundamentally different behavior, but it would be a more theoretically sound derivation of a scheme similar to the one used in the paper. See: potential-based reward shaping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "04AW0r4kJ3", "reviewer": "Reviewer_6nHT", "summary": "Authors demonstrate Lean proof assistant's fine grained feedback on proofs can be harnessed as tactic level process rewards that produce improved RL training results. They compare outcome only, process only, and their hybrid method and show improved training dynamics. They demonstrate solid results on miniF2F and ProofNet and contribute detailed ablation studies for several design decisions.", "questions": "- has there been any opportunities to further scale training? the differences between different conditions seem fairly small on the outcome reward chart in figure 2, and RL is infamously unstable (even in the R1 paper poor trends (negative/plateau) are observable for hundreds of steps before another rise). I totally understand if it's a resource limitation, but curious to see if the patterns highlighted here hold at scale.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- thorough definitions and descriptions of their procedures\n- clean idea that leverages the additional signal afforded by powerful proof assistants\n- thorough ablation studies confirming the significance of various design decisions", "weaknesses": "- analysis limited to full proof generation-- search and iterative repair remain important techniques in automatic theorem proving\n  - this point is slightly biased towards the human perspective, but theorem proving is often understood as a search problem\n\n- nit: reward definition hard codes values given to \"valid steps\" before incorrect steps. This measure of validity is only in the syntax sense, right? The tactic is not necessarily contributing to a correct proof. The empirical analysis with d1/d2 shows that it should be treated distinctly from syntactically incorrect tactics which makes sense, but beyond this ordering, selecting a fixed value doesn't seem grounded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pQOxoHmTjh", "reviewer": "Reviewer_4VNo", "summary": "This paper proposes using the Lean proof assistant as a symbolic process oracle during reinforcement learning training for theorem proving. The key idea is to parse generated proofs into tactic sequences and leverage Lean's elaboration feedback to assign tactic-level rewards in addition to outcome-level rewards. The authors integrate these structured rewards into a GRPO-style objective with first-error propagation and first-token credit assignment. Experiments on STP-Lean and DeepSeek-Prover-V1.5 show modest improvements on MiniF2F and ProofNet benchmarks over outcome-only baselines.", "questions": "The main questions and concerns are detailed in the Weaknesses section above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Clear formulation and motivation: The paper provides a well-structured formalization of Lean as a process oracle, defining parsing function f, outcome function g, and tactic-level scoring function φ. The MDP formulation is clear and the integration with GRPO is technically sound.\n\n2. Comprehensive ablation studies: The authors conduct thorough ablations on token-level credit assignment (Table 3), reward strategies (Table 4), timeout settings (Figure 3), and hyperparameter sensitivity (Table 5, Appendix C), demonstrating careful experimental work.\n\n3. Stability analysis: The comparison with outcome-only GRPO shows that tactic-level supervision provides more stable training dynamics, with analysis of entropy and proof length providing useful insights.", "weaknesses": "1. **Fundamental semantic limitation without learned PRM comparison (Critical)**: The paper's most significant limitation is that Lean's tactic-level feedback only provides **syntactic/type-theoretic correctness**, not **mathematical progress or strategic value**. As the authors acknowledge in Appendix H (imo_2019_p1 example), tactics that elaborate successfully (e.g., introducing hypotheses h_1 through h_9) receive no penalty despite being mathematically unproductive. The method only penalizes the final failing tactic (omega), missing that the entire proof strategy was flawed from the beginning.\n\n   More critically, the paper does not compare against learned process reward models (e.g., Lightman et al., Wang et al.) that could potentially capture mathematical progress beyond syntactic verification. While the authors claim no Lean tactic-level dataset exists, this actually highlights the core limitation—**symbolic verification captures local soundness but misses global strategic value that learned PRMs aim to model**. The absence of this comparison makes it impossible to assess whether the symbolic approach is fundamentally limited compared to learned alternatives, or whether combining both approaches could address the semantic gap.\n\n2. **Marginal empirical improvements**: The performance gains are quite limited:\n   - MiniF2F: +1.2-1.4% at pass@32, +2.5% at pass@64\n   - ProofNet: +1.4% at pass@32, essentially no gain (-0.1%) at pass@64\n   \n   These improvements are within typical variance ranges and raise questions about whether the added complexity is worthwhile. The gains are particularly underwhelming considering that the method requires calling Lean's parser and elaborator during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IzbnMkfz4D", "reviewer": "Reviewer_sb7Y", "summary": "The paper uses Lean as a verifier for process oracle rewards (which are tactic-level rewards in the case of theorem proving). It then incorporates it into a GRPO-style reinforcement learning objective with first-error propagation and first-token credit methods that balances outcome- and process-level advantages. The authors report that on MiniF2F and ProofNet, the proposed approach outperforms both outcome-only RL and vanilla baselines.", "questions": "Have you thought about other more fine-grained tactic-level rewards?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper is well-written and easy to understand. This includes a dedicated formalization of the problem and clear mathematical representation.\n- The approach is straightforward. Once the additional process rewards are defined, incorporating them with GRPO seems standard and I believe reproducible.", "weaknesses": "- My major concern is novelty. Leveraging feedback from ITPs has long been known to (and practiced by) the ML for theorem proving community. The paper doesn’t offer much other than demonstrating that combining it with GRPO did bring some benefits empirically. This is particularly concerning because the tactic-level rewards being considered here essentially only distinguish between “applicable” and “inapplicable” tactic applications. While useful, they are not novel signals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)