VERINA: Benchmarking Verifiable Code Generation
Authors: Zhe Ye
Zhengxu Yan
Jingxuan He
Timothe Kasriel
Kaiyu Yang
Dawn Song
Venue: ICLR 2026 Poster (2026)
Keywords: code generation
formal verification
verifiable code generation
AI for math
theorem proving
AI for code
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: Large language models (LLMs) are increasingly integrated in software development, but ensuring correctness in LLM-generated code remains challenging and often requires costly manual review. Verifiable code generation---jointly generating code, specifications, and proofs of code-specification alignment---offers a promising path to address this limitation and further unleash LLMs' benefits in coding. Yet, there exists a significant gap in evaluation: current benchmarks often focus on only individual components rather than providing a holistic evaluation framework of all tasks. In this paper, we introduce VERINA (Verifiable Code Generation Arena), a high-quality benchmark enabling a comprehensive and modular evaluation of code, specification, and proof generation as well as their compositions. VERINA consists of 189 manually curated coding tasks in Lean, with detailed problem descriptions, reference implementations, formal specifications, and extensive test suites. Our extensive evaluation of state-of-the-art LLMs reveals significant challenges in verifiable code generation, especially in proof generation, underscoring the need for improving LLM-based theorem provers in verification domains.
The best model, OpenAI o3, achieves a 72.6% code correctness rate, 52.3% for specification soundness and completeness, and a mere 4.9% proof success rate (based on one trial per task). We hope VERINA will catalyze progress in verifiable code generation by providing a rigorous and comprehensive benchmark. We release out dataset on https://huggingface.co/datasets/sunblaze-ucb/verina and our evaluation code on https://github.com/sunblaze-ucb/verina.

Published as a conference paper at ICLR 2026
VERINA: BENCHMARKING VERIFIABLE CODE GENER- ATION
Zhe Ye', Zhengxu Yan', Jingxuan He', Timothe Kasriel', Kaiyu Yang" Dawn Song' University of California, Berkeley, Meta FAIR
ABSTRACT
Large language models (LLMs) are increasingly integrated in software develop- ment, but ensuring correctness in LLM-generated code remains challenging and often requires costly manual review. Verifiable code generation-joing ating code, specifications, and proofs of sode-specification alignment- a promising path to address this limitation and further unleash LLMs' benefits in coding. Yet, there exists significant gap in evaluation: current benchmarks often focus only individual components rather than providing holistic evaluation framework of all tasks. In this paper, we introduce VERINA (Verifiable Code Gen- eration Arena), a high-quality benchmark enabling a comprehensive and modular evaluation of code, specification, and proof generation as well as their compo- sitions. VERINA consists of 189 manually curated coding tasks in Lean, with detailed problem descriptions, reference implementations, formal specifications, and extensive test suites. Our extensive evaluation of state-of-the-ar L.LMs reveals significant challenges in verifiable code generation, especially in proof generation, underscoring the need for improving -ased theorem provers in verification domains. The best model, OpenAI o3, achieves 72.6% code correctness rate, 52.3% for specification soundness and completeness, and a mere 4.9% proof suc- cess rate (based verifiable code generation by providing a rigorous and comprehensive benchmark. and We release dataset on httpss/wwggginngaae..ooodassssaa our evaluation code on https//gghhalccomm/mmbuuzzu////e
gener-
trial per task). We hope VERINA will catalyze progress in
INTRODUCTION
Large language models (LLMs) have shown strong performance in programming (Jain al. 2025 Jimenez et al. 2024; Chen et al. 2021) and are widely adopted in tools like Cursor and GitHub Copilot to boost developer productivity (Kaliamvakou), ALM--generate code becoming prevalent in commercial software (Peters) 2024) and may eventually form substantial portion of the world's code. However, due their probabilistic nature, LLMs alone cannot provide formal guarantees for the generated code. As a result, the generated code often contains bugs, such functional errors (Wang et al.) 2025) and security vulnerabilities (Pearce et al.] 2022). When L.Lee code generation is increasingly adopted, these issues can become productivity bottleneck, as they typically require human review to be resolved (Finley). Formal verification presents a promising path to establish correctness guarantees in LLM--generaed code but has traditionally been limited to safety-critical applications due to high cost (Gu et al.] 2016 Leroy et al.) [0166] Bhargavan et al.) 2013). Similarly how they scale up code generation, LLMS have the potential to significantly lower the barrier of formal verification. By jointly generating code, formal specifications, and formal proofs of alignment between code and specifications. LLMs can offer higher levels of correctness assurance and automation in software development. This approach represents an emerging programming paradigm known as verifiable code generation (Sun et al., 2024 Yang al. 2024).
Given the transformative potential of verifiable code generation, it is crucial to develop suitable benchmarks to track progress and guide future development. This is challenging because verifiable code generation involves three interconnected tasks: code, specification, and proof generation. We need to curate high-quality samples and establish robust evaluation metrics for each individual
'lll data processing and experiments were conducted outside Meta.

---

Published as a conference paper at ICLR 2026
Table 1: A comparison of VERINA with related prior works on LLMs for code generation and verification. We characterize whether each work supports the three foundational tasks for end-to-end verifiable code generation: CodeGen, SpecGen, ProofGen (Section 4.1). means fully supported, O means partially supported, O means unsupported. If ProofGen is supported, we specify the proving style: automated theorem proving (ATP) or interactive theorem proving (ITP). For works supporting multiple tasks, we annotate if these tasks are supported in modular and composable manner. Overall, VERINA offers more comprehensive and high-quality benchmarking compared to prior works.
CodeGen
SpecGen
ProofGen
Proving Style
Compositionalty
Language
Benchmurks
HumunEval Chon et al. 2r23 MBPP
Austin
2021
Python
Dafny- Symthesi [Mis K024
Dafty
Dufny
Loughrlle ei KyA5
ATP
Dafny
miniCidePron Trein Welle 2074
0
ITP
Lean
FVAPPS Doughhny Malla
ITP
X
Lean
alZpostcond (Eadees cl
Python, Java
Clover (Sun 20 pu003
ATP
X
Dafny
AlphaVerus Axparre al 2024
ATP
X
Rust
Tochnigues
SpecGen ola
AutoSppec 2
ATP
ATP
X
CIC++
lava
SAFE ci 40363
ATP
X
Rust
Autu Y15g 21 2025
ATP
Rust
Laure] Maccugg
ATP
Dafny
ATP
Java
Baldur 51 2023
Selene Zhang ot al. 2024
ITP
Isbellle
Rango
Tloenpsin & R1
2025 PALM Lu a a 2014
ITP
Cop
VERINA
ITP
Lean
task, while also composing individual tasks to reflect real-world end-to-end usage scenarios where LLMs automate the creation of verified software directly from high-level requirements. Existing benchmarks, as discussed in Section 2 fall short as they lack comprehensive support for all three tasks (Loughridge al. 2025 Aggarwal et al. 20244 Chen et al.] (025), quality control (Dougherty & Mchtal 2025), robust metrics (Misu et al.] 2024), or a modular design (Sun et al. 2024).
To bridge this gap, we introduce VERINA (Verifiable Code Generation Arena), high-quality bench- mark to comprehensively evaluate verifiable code generation. It consists of 189 programming challenges with detailed problem descriptions, code, specifications, proofs, and comprehensive test suites, We format these problems in Lean (Moura & UlIrich 2021), a
programming
language with rapidly growing ecosystem and applications both formal mathematics (Mathlib community 22200 Mathlib Community 2022) and verification (de Medeiros et al. [202aa] Hietala &
2024). Lean has become the one of the most popular platforms for LLM- proving and verification, demonstrated by breakthrough results like AlphaProof (Google Deep 2024) and production adoption at organizations like AWS (de Moura), with ongoing efforts to use
theorem-
Lean for verifying mainstream languages like Rust (Ho & Protzenko 2022). We provide additional discussion on the choice of Lean in Appendix A
VERINA is constructed with careful quality control. It draws problems from various sources, including MBPP (Misu et al., 2024; Aussin et al. 2021). LiveCodeBench (Jain et al. 2025). and LeetCode, offering a diverse range of difficulty levels. All samples in the benchmark are manually inspected and revised to ensure clear text descriptions and accurate formal specifications and code implementations. Moreover, each sample also includes comprehensive test suite with both positive and negative cases, which achieves 100% code coverage and passes the ground truth specification.
VERINA facilitates the evaluation of code, specification, and proof generation, along with flexible combinations of these individual tasks. We utilize the standard pass @ k metric (Fan et al. 2024) with our comprehensive test suites to evaluate code generation. For proof generation, we use the Lean compiler to automatically verify their correctness. Furthermore, we develop multi-stage evaluation pipeline that systematically assesses generreee specifications by combining theorem proving and comprehensive testing, providing a practical and robust way to score their soundness and completeness against our ground truth specifications.
The high-qualins samples and robust metrics of VERINA establish it as a rigorous platform for evaluating verifiable code generation. On VERINA, we conduct a thorough experimental evaluation of ten state-of-the-ee general-purpose. LLMs and three LLMs or agentic frameworks specialized in theorem proving. Our results reveal that even the top- perf-orffnigg general-purposs LLM, OpenAI o3 (OeenAl), struggles with verifiable code generation, producing only 72.6% correct code solutions, 52.3% sound and complete specifications, and 4.9% successful proof in one trial. Among theorem- proving I.Mss, the best model, Goedel Prover V2 32B (Lin et al. 2025), achieved an 1 11.2% proof
2

---

Published as a conference paper at ICLR 2026
1 Description of the coding problor in natural lanquaqo
2 Remove an clement from given erray of integera at spectried indox. The resuiting array should the original slenenia excupd renoved eloment zenain unchanged, and alanents after it aze shifted one position the left.
the given index. Elesenta before the
Gode inplementaa ion
def senoveRlenent (a Acray lnt) (k Bat) phuprecend renoveBlemett_pre s.arascIdx!
k) Acray Int :*
Pre condit loc
9
del Conoveeienent_.re la Assay Int) ik : Mat] : Prop :-
10
X s.91ze -- the indox musc de anallor than the array si2e condition
det semoveE.eeentt_por (a Acray Inc) (k Hatl [1es: Azray Inc) (upreccodd removeBlement_ppa a
13
Prop :-
14
res.xise s.xise 1 Only une element is resoved
(Y L,
elenents before index renain unchanged The elenencs after index are serifed by ane position
restill a[4]t)
17
LV -, 1 ces.aian -t 2 k sea[i]! = ati 1 :llt
18 - Proof foroot body onicted for bravicy]
theoren remnvallement tenevellemen...oo post Jommovell ennenn a legrecond) Ruprecood s- by socty Test ca562
(8: krray Int)
(h.precond renovel) prent..pre
21
22
[s +11, 3, 513 (e
(ree +00, 2, 4, 513 - Poviiiive text with valid inpute and out
NegaLive
24
I8 :
11,
3.
513
C
Inpute viotate the pre-condition at Mine 12
25
:
41,
2,
3,
4,
513
C
2) (res +[1, 2, 41) put violiter the peat condit ino at Line 16
26
3.
513
C
(ews 422, 2, 4, 513
Dulpul viulates the pest cosurooniiiio ton ai. Liow 17 Output wiolater the post-condit
free
+0,
413
18
Figure 1: An example instance of VERINA, consisting of problem description, code implementation, specifications (pre-condition and post-condition), a proof (optional), and comprehensive test cases. Note that we select this instance for presentation purposes and VERINA contains more difficult ones.
success rate in one trial. Interestingly, iterative refinement using Lean compiler feedback can increase the proof success rate to 20.1% with 64 refinement steps. However, this approach significantly raises costs and the success rate remains low. These findings underscore the challenges of verifiable code generation and highlight the critical role of VERINA in advancing the field.
2 BACKGROUND AND RELATED WORK
We present works closely related to ours in Table and discuss them in detail below.
Task support for verifiable code generation. Writing code, specifications, and proofs for a verified software component time-consuming when done manually, Although various studies have explored using I.LMs to automate these tasks, they primarily focus on individual aspects, failing to capture the full spectrum of verifiable code generation. Benchmarks like HumanEval (Chen et al. 202T) and MBPP (Austin et al.) (021) have sparked impressive progress on L.LM-based code generation but do not handle formal specifications or proofs. Many craffication-oouusee efforts target only one two tasks, while assuming the other elements are provided by the human user. For example, DafnyBench (Loughridge et al. 2025) and miniCodeProps (Lohn & Welleck 2024) are two benchmarks designed exclusively for proof generation. Moreover, AutoSpec (Wen et al. 2024) and SpecGen (Ma et al. 2025) infer specifications and proofs from human-written code.
To the best of our knowledge, Dafny-SSnthesis (Misu al. 2024) and Clover (Sun al. 2024 the only two works that cover all three tasks, like VERINA. However, they target automated theorem proving using Dafny (Leino) 2010). while VERINA leverages interactive theorem proving in Lean. Moreover, they have relatively small numbers of human-writter samples (50 and 62 respectively). In contrast, VERINA provides 189 high-quality samples that are manually validated and undergo rigorous quality assurance (Section
Automated and interactive theorem proving, A major challenge in formal verification and verifi- able code generation lies in tooling. cnffiicationoorientte languages like Dafny (Leino (010) and Verus |Lattuada et al. (023) leverage SMT solvers for automated theorem proving (De Moura & Bjørner 20088] Barrett & Tinelli (018)) and consume only proof hints, such as loop invariants (Pei ct al. (023) and assertions (Mugnier al. 2025). However, SMT solvers handle only limited proof domains and behave as black boxes, which can make proofs brittle and hard to debug (Zhou et al.) 2023). Interactive theorem proving (ITP) systems like Lean provide a promising target for verifiable code generation with LLMs. ITPs support constructing proofs with explicit intermediate steps. This visibility enables LLMs to diagnose errors, learn from unsuccessful steps, and iteratively refine their proofs. Recent work shows that LLMS can generate proofs human level in math competi-
3

---

Published as a conference paper at ICLR 2026
tions (Google DeepMind 2024). Prior verification benchmarks in Lean include miniCodeProps (Lohn Welleck 2024) and FVAPPS (Duuhherty & Mehta 2025). miniCodeProps translates 201 Haskell programs and their specifications into Lean but is designed for proof generation only. FVAPPS con- tains 4,715 Lean programs with L.M--generaeed specifications from fully automated pipeline that lacks human validation and quality control. In contrast, VERINA provides human-verified samples and captures all three foundational tasks in verifiable code generation.
Task compesitionality. A key strength of VERINA its modular design, which enables flexible evaluation of not only individual tasks but also their combinations (Section 4.2). This compositionality captures diverse real-world scenarios from specification-guu -ogiidddngddd code generation to end-to-ent verifiable code generation enabling a comprehensive assessment of different aspects of verifiable code generation. This modularity also facilitates targeted research on specific weaknesses, such as improving proof generation. On the contrary, all other prior works lack full compositionality For example, Dafny-Synthest (Misu et al. 2024) and Clover (Sun et al., 2024) mix specification and proof generation into single task, lacking support for separate evaluation of each.
VERINA: DATA FORMAT, CONSTRUCTION, AND QUALITY ASSURANCE
We describe the VERINA benchmark, its data construction pipeline, and quality assurance measures.
3.1 OVERVIEW AND DATA FORMAT
VERINA consists of 189 standalone programs, annotated with natural language descriptions, code, specifications, proofs, and test cases. The code, specification, and proof are all written in Lean. An example is illustrated in Figure I consisting of:
Natural language description (Line 1-4): informal description of the programming problem, capturing the intent of the human developer.
Code (Line 5-7): ground truth code implementation that solves the programming problem.
Specification (Line 8-17): ground truth formal specification for the programming problem. consists of pre-condition, which states properties the inputs must satisfy, and post-condition, which states desired relationship between inputs and outputs.
Proof (Optional, Line 18-20): formal proof establishing that the code satisfies the specification. Ground truth proofs are optional in VERINA, as they are not required for evaluation. Model- generated proofs can be checked by Lean directly. Nevertheless, we invest significant manual effort writing proofs for 46 out of 189 examples as they help quality assurance (Section 3.2
Test suite (Line 21-27): a comprehensive suite of both positive and negative test cases. Positive tests are valid input-output pairs that meet both the pre-condition and the post-condition. Negative tests are invalid inputs-output pairs, which means either the inputs violate the pre-condition or the output violates the post-condition. These test cases are useful for evaluating model- generrted code and specifications, as detailed in Section 4.1 They are formatted in Lean during evaluation.
Benchmark statistics. Table 2 presents key statistics of VERINA. Natural language descriptions have a median length of 110 words, ensuring they are both informative and detailed. Code ranges up to 38 lines and specifications up to 62 lines, demonstrating that VERINA captures complex tasks. With median of positive tests and 12 negative tests per instance, the constructed test suites provide strong evidence for the high quality and correctness of VERINA.
Table 2: Statistics of VERINA.
Metric
Median
Max
Words in Description LoC for Code
110
296
38
LoC for Spec.
62
Positive Tests
13
Negative Tests
12
27
3.2 BENCHMARK CONSTRUCTION AND QUALITY ASSURANCE
VERINA consists of 189 problems sourced from different origins. We employ meticulous data cura- tion process that combines careful translation, thorough manual review, and automated mechanisms, leading to rigorous and high-quality benchmark for verifiable code generation.
4

---

Published as a conference paper at ICLR 2026
To construct VERINA, first consider MBPP-DFY-55 (Misu et al., 2024) our data source. It consists of MBPP (Austin et al. 202T) coding problems paired with human verified solutions in Dafny, Each instance contains a natural language problem description, code implementation, specifications, proof, and test cases. We manually translated 49 problems into Lean, refining and verifying each translation. To extend the benchmark, we added 59 more human-authored Dafny instances from CloverBench (Sun et al., (024). These were translated into Lean using OpenAI o3-mini with few-shot prompting based on our manual translations, followed by manual inspection and correction.
Additionally, VERINA incorporates problems adapted from student submissions to a lab assignment in course on theorem proving and program verification. Students, both undergraduate and graduate, were encouraged to source problems from platforms like LeetCode or more challenging datasets such as LiveCodeBench (Jain et al. 2025). They formalized and solved these problems in Lean, providing all necessary elements in VERINA'S ' format (Section 3.1]. We carefully selected the most suitable and high-quality submissions, resulting in 81 benchmark instances. In addition, we manually reviewed and edited the submissions to ensure their correctness. During evaluation, we observe problems adapted from student submissions are generally more difficult than problems translated from Dafny datasets on all models, with detailed analysis provided in Appendix D
Quality assurance. During the data collection process, we consistently enforce various manual and automatic mechanisms to ensure the high quality of VERINA:
Detailed problem descriptions: The original problem descriptions, such as those from MBPP-DFY. 50, can be short and ambiguous, making them inadequate for specification generation. To resolve this, we manually enhanced the descriptions by clearly outlining the high-level intent, specifying input parameters with explicit type information, and detailing output specifications.
Full code coverage with positive tests: Beyond the original test cases, we expanded the set of positive tests to ensure that they achieve full line coverage on the ground truth code. We created these additional tests both manually and with LLMss We leveraged the standard coverage py tool to verify complete line coverage, since Lean lacks a robust coverage tool. This approach aligns with common practices for assessing functional correctness across languages (Cassano et al.) 2023. Roziere et al. 2022). For Python reference implementations, we either used the original MBPP code generated implementation from the enhanced problem description via OpenAl's 04-mini with manual validation. To further ensure coverage transferability, we manually inspected all benchmark instances and confirmed that our test suites also achieve 100% line coverage on the Lean ground truth implementations.
Full test pass rate on ground truth implementations and specifications: We evaluated both the ground truth implementations and specifications against our comprehensive test suites. All ground truth implementations and specifications successfully pass their respective positive tests, confirming the quality of the implementations and specifications in VERINA.
Necessary negative tests: We mutated each positive test case to construct at least three different negative tests that violate either the pre- or the post-condition, except when the function's output has boolean type, in which case only single negative test can be created. These negative tests are explicitly categorized based on whether they violate the pre-condition or the post-condition to enable separate and precise evaluation of each specification component. We made sure that our ground truth code and specifications do not pass these negative tests.
Preventing trivial code generation: VERINA allows providing ground truth specifications as an optional input for the code generation task (discussed in Section (4.I). We crafted all ground truth specifications such that they cannot be directly used to solve the coding problem. This prevents LLMs from generating an implementation trivially equivalent to the specification. As a result, the model must genuinely demonstrate semantic comprehension of the reference specification and non-trivial reasoning to generate the corresponding implementation.
Manual review and edits: Each benchmark instance was manually reviewed by at least two authors, carefully inspecting and editing them to ensure correctness and high quality.
4 EVALUATING VERIFIABLE CODE GENERATION USING VERINA
VERINA enables comprehensive evaluation of verifiable code generation, covering foundational tasks asks-code, code, specification, and proof generation and their combinations form end-to-end
5

---

Published as a conference paper at ICLR 2026
pipeline from natural language descriptions to verifiable code. We also introduce novel framework for a reliable automatic evaluation of model-generated specifications.
4.1 FOUNDATIONAL TASKS AND METRICS
As shown in Figure 2 all three foundational tasks include natural language descriptions and function signatures (Lines 7, 11, and 15 in Figure as model inputs, which captures human intent and enforces consistent output formats, facilitating streamlined evaluation.
Description Signature
Deseription Signature
Deseription
Signature
SpecGen
Spec.
CodeGen
Code
Code
ProofGen
Proof
Code
Spec.
Spec.
Figure 2: VERINA'S three foundational tasks.
Dashed arrows represent optional inputs.
Specification generation (SpecGen). Given description, signature, and optionally code imple- mentation, the model generates formal specification. Next, we formally define the soundness and completeness relationships between the generated specification and the ground truth specification. Then, we describe our multi-stage evaluation pipeline to assess whether these relationships hold.
Let & denote the set of programs that satisfy the ground truth specification and & the set that align with the generated specification. An ideal generated specification should achieve I o, which entails two properties
soundness (Ø S Φ): it "small enough" cover only correct programs, and (ii) completeness (ø C Φ): it is "large enough" to cover all correct programs. Since specifications consist of pre-conditions and post-conditions, let and p denote the ground truth and model-generanl pre-conditions, respectively, and Q and Q the corresponding post-conditions. VERINA, we define the soundness and completeness of P and Q as follows:
P is sound iff ((T) = P(z), where T are the program's 's input values. Given the same post- condition (e.g.. Q), is more difficult for program satisfy p than P. This is because i allows more inputs, which the program must handle to meet the post-condition. As a result, the set of programs accepted by P subset of those accepted by P.
P is complete iff VT. P(z). Given the same post-condition, the set of programs accepted by P is now superset of those accepted by P, since p more restrictive than P.
Q is sound iff VT, y. P(x) A Q y) = Q y), where is the output value. For any valid inputs w.r.t. P. the set of output accepted by Q is subset of those accepted by Q, establishing soundness. Symmetrically, Q is complete iff VI, y. P(x) ^ Q(E,y) Q(E,W).
To practically and reliably assess whether the above relationships hold, we develop a multi-stage evaluator based on theorem proving and comprehensive testing, as shown in Figure 3 Given soundness completeness relationship R, the evaluator first attempts to prove using M.M--bseed theorem provers, they provide formal guaran- tees when proof is successful. When the prover is inconclusive, e.g. current LLM-based provers (as detailed
Prove and -B using L.M
Proves if & holds
Yes
No
proce
hobde
Sumplaty to It using tosts
not
due the incapability Appendix C.5). the evaluator proceeds with practical testing-bar based framework using our comprehensive test suites. this resting-bsses process, we check It against concrete values in test cases. Specifically, we distinguish between negative tests that violate pre-conditions and those that violate post-conditions, applying them separately to evaluate the corresponding specification component.
Dezides if R'holds
Canmot
New
No
decide
miight lold
Property-hasell
mos hold
Teating for R'
Counteresample?
Cannet
No
Yos
test
R might
Jt doos boc hold
Unknowa
helld
For example, to evaluate Q's soundness, we check if P(F) A Q( y) Q y) holds for all test cases (z. y) in our test suite. We denote this simplified version of R as R'. For many cases, e.g., the specification in Figure [ Lean automatically determine if R' holds (Selsam et al.
Figure 3: Our evaluator for specification generation.
2020 and we return the

---

Published as a conference paper at ICLR 2026
corresponding result. Otherwise, employ property- based testing with the plausible tactic in Lean (Lean Prover Community 2024). generates diverse inputs specifically targeting the remaining universally and existentially quantified variables in R', systematically exploring the space of possible values to test R'. In Appendix C.5 we provide a detailed description of how we implement these metrics in Lean.
Since our evaluator integrates proof and testing, it can certify R holds when a formal proof of R succeeds, and it can certify R does hold by producing counterexamples. When only testing passes without a proof, the evaluator returns I might hold, reflecting strong empirical evidence that R holds. While cannot formally establish R holds, remains highly robust in this regard, due to our comprehensive test suite with both positive and negative tests, which achieve full coverage on ground truth code implementations. Lean's property-base testing cannot handle small number of complicated relationships some testcases, for which our evaluator returns unknown. To further enhance the accuracy of our metric, we repeat our evaluation framework to check -R. We compare the evaluator outcomes on R and -R, selecting the definitive result whenever the other yields unknown.
Our final metrics for SpecGen include individual pass bass@k scores (Chen et al. 2021 for soundness and completeness of all generated pre-conditions and post-conditions, as well as aggregated scores that soundness and completeness hold simultaneously for pre-condition, post-condition, and the complete specification. Since our specification evaluator may return unknown, we plot error bars indicating the lower bound (treating unknown R does not hold) and upper bound (treating R holds).
To illustrate our metric, consider the ground truth pre-condition k s.size at Line 12 of Figure ! and model-generated pre-condition can be determined as unsound using the positive test (s A [[1, 2, 3, 4, 51) (k 4), while sisize
s. size
and k sisi 1. k s.size
incomplete based on the negative test (s : #11, 2, 3, 4, 51) (k : 5).
We provide more examples of our metrics for specification generation in Appendix
Code generation (CodeGen). Given a natural language description, function signature, and op- tionally specification, the model generates code implementing the desired functionalityy Following standard practice, we evaluate the generated code by running against positive test cases VERINA and reporting the pass @+ k metric defined by Chen et al. (2021). In Section 4.2 we will explore evaluating code by proving its correctness with respect to the formal specification.
Proof generation (ProofGen). Given a description, signature, code, and specification, the model generates formal proof in Lean to establish that the code satisfies the specification. This task evaluates the model's ability to reason about code behavior and construct logically valid arguments for correctness. We use Lean to automatically check the validity of generated proofs, and proofs containing placeholders (e.g., the sorry tactic) marked incorrect.
4.2 TASK COMBINATIONS
VERINA enables combining the three foundational tasks to evaluate various capabilities in verifiable code generation. These combined tasks reflect real-world scenarios where developers utilize the model to automatically create verified software in an end-to-end manner. Such modularity and
Gececated Cole
Clste Evsluntor
Pare Fither Falls
Geneested Prasf
CodoGen
Code Evalustor
Orund Thrck
Proof Evslunter
Speifiction
Ground Trush Spetihottion
Prouftion
Generated Proef
Proof Evalustur
Pass
Eitbur Fulls
Spedien
Geservte Syue.
Spee Rei fuater
Fither Falls
All Piss
Spection
Generated Spac:
Eralustor
Spee
Ticck
Coca
Poodieu
Prost
Proof Konioente
Figure 4: Combinations of VERINA'S foundational tasks: apecification-guiiee code generation (top left), specification inference from code (bottom left), and end-to-end verifiable code generation (right). Natural language descriptions and function signatures omitted in the figure for brevity.
7

---

Published as a conference paper at ICLR 2026
compositionality highlight the generality of VERINA, which encompasses various tasks studied in previous work (Table IT. Three examples of combined tasks are (Figure 4):
PrecificationGuudder Code Generation: Given a natural language description, function signature, and the ground truth specification, the model first generates the code and then proves that the code satisfies the specification. This aligns with tasks explored in FVAPPS (Dougherty & Mchta 2025 and AlphaVerus Verus (Aggarwal et al. 2024
Specification Inference from Code: Developers may have the code implementation and want the model to annotate it with a formal specification and prove their alignment. This corresponds to the setting in AutoSpec (Wen et al.] 2024), SpecGen (Ma et al. (222), and SAFE (Chen al.. 20025).
End-to-Ee End Verifiable Code Generation: For even higher degree of automation, developers might start with only a natural language problem description and instruct the model to generate code and specification independently, and then generate the proof. This captures the scenario in Dafny- Synthesis (Misu et al., 2024) and Clover (Sun et al. 2024). In this task, we specifically require the model to generate proof that the generated code satisfies the ground truth specification. This prevents the model from generating definitionally equivalent code and specifications to trivialize the proof, ensuring the evaluation reflects the model's true verification capability.
In these task combinations, a crucial design consideration is the dependency between code and specification. For example, in specification-guided code generation, it is important to assess how beneficial the ground truth specification is beyond the natural language description, which already captures the developer's intent. Additionally, for end-to-end verifiable code generation, it is essential decide the order of the CodeGen and SpecGen the output of CodeGen, place SpecGen before CodeGen, or run them independently (as in Figure 4 We experimentally explore these design choices using VERINA Section [5] Concurrent with our work, CLEVER (Thakur et al.) 2025) introduces 161 manually crafted problems sourced from HumanEval (Chen et al., (021) with ground truth specifications. However, CLEVER only supports the SpecGen task and the specification-guice
make SpecGen dependent on
code generation setting and cannot capture the full spectrum of workflows that VERINA enables through both individual and compositional tasks. We
provide detailed comparison in Appendix C.4
EXPERIMENTAL EVALUATION
Experimental setup. We evaluate a diverse set of ten state-of-the-art general-purposs LLMS and three L.MSs or agentic frameworks specialized in theorem proving. We leverage 2-shot prompting to enhance output format adherence, with the 2-shot examples excluded from the final benchmark. For each task, we primarily report the pass@) prompts, output formats, and LLM setups in Appendix
@ I metric (Chen et al. 202I). We provide detailed input
do
Code Score
Spec Coundd&Camllete Score
Proof Score
100
100
100
80
60
11.4
40-
20
224
05 12
32
Figure 5: pass@1 1 performance of LLMS on VERINA'S 's three foundational tasks.
All foundational tasks are challenging, especially ProofGen. As shown in Figure code generation generally achieves the highest success rates across models, followed by specification generation, while proof generation remains the most challenging with pass@ rates below 4.9% for all general purpose models. All three tasks pose significant challenges for current general purpose LLMs, with constructing Lean proofs that the implementation satisfies the specification being particularly hard and requiring specialized theorem proving capabilities. This also means that for any combined task involving ProofGen, e.g., the ones in Section 4.2 LLMs' performance will be heavily bottlenecked by the ProofGen subtask. Among the evaluated models, 04-mini, o3, Claude Sonnet 3.7, Claude
8

---

Published as a conference paper at ICLR 2026
Opus 4.0, and Gemini 2.5 Flash demonstrate relatively stronger performance across tasks. We report detailed results on pre-condition and post-condition soundness and completeness in Appendix D where observe that generating sound and complete post-conditions is generally more difficult than pre-conditions.
CFT 4o-irl
GTT-4 (ls de Opua
c6-mil DeepSeek va
93
GFT 4.2 Gerrin Flych
Caste Sonner 3.7
Qwnn 2353-A228-104
Cade&Spec: Score
Eno-to-End Score
100
100
40
412
40-
12.4
20.4
20
155
20
8.3 333
Figure 6: pass @1 performance of LLMs on VERINA'S 's end-to-end verifiable code generation task.
ProofGen the major bottleneck for end-to-end verifiable code generation. We further evaluate the models on the most challenging setting: end-to-end verifiable code generation, as defined in Section 4.2 We report the & Score, where both generated code and specification should be correct, and the End-to-End Score, where additionally the proof verifying the generated code against the ground truth specification should be correct. As shown in Figure (6] simultancously generating correct code and specifications is difficult, with the leading model, 03, achieving only 41.2%. Furthermore, the evaluation results confirm that ProofGen is the bottleneck in end-to-end verifiable code generation setting, with the leading model, 04-mini and o3, achieving only 3.2%.
c4-mini
GP
Claude Sonnet 3-7
Prover (14-m
Genini 255 flash Iter ative Refinements
Goosel Prover V2 328
DeepStek Prover V2 7B Direct Generation
Proof Score
100
80
60-
20
4.9
11.2
Figure 7: pass@1 1 for ProofGen
Figure 8: pass l performance of selective LLMs on ProofGen using proof refinement (left) and direct generation (right).
across models and proving agent.
Specialized provers and agentic methods improve proof success rate. Given the limitations of general-purpose LLMs, we extend our evaluation to specialized theorem- shorremproovin models and agentic approaches. As shown in Figure 7 Goedel Prover V2 32B (Lin et al. 2025) and DeepSeek Prover V2 7B (Ren et al. 2025) achieve higher proof success rates compared to general-purpon
models. We further evaluate Copra (Thakur al. 2023), an agentic theorem-proving framework based on tree-search. We use o4-mini as the backbone model and allow at most 64 LLM queries for each sample. Copra demonstrates clear improvements over direct single-pass generation.
Iterative proof refinement shows meaningful improvements. For ProofGen task, besides pass @ 1, we also extend the evaluation of the four general-purpose models (o4-mini, GPT 4.1, Claude Sonnet 3.7, Gemini 2.5 Flash) alongside two specialized LLM- -provers (Goedel Prover V2 32B (Lin et al.
2025) and DeepSeek Prover V2 7B (Ren et al., 2025) )) We evaluate them with iterative proof refinement, where the evaluated model receives Lean verifier error messages and is prompted to revise its proof, and with direct generation, where the evaluated model generates responses independently without Lean feedback in each iteration. For all methods, rounds of iterations, for up 64. This metric investigates how much additional interaction helps repair the proof that single-pass generation would miss, and whether providing Lean verifier feedback improves success rates compared to independent generation attempts.
report pass @ k, the success rate after k
As shown in Figure iterative proof refinement reliably outperforms direct generation at matched query budgets on both general purpose and proof- -ppecsscccficc models, underscoring the value of Lean verifier feedback. A detailed breakdown by problem difficulty is provided in Appendix D

---

Published as a conference paper at ICLR 2026
GPT dounini
GPT
camnhi DeesSeck V3
GPT 4.1
Cisude Sornet 3.7
Clusd Opus 4.0
Owen 2338 1228-2228-100
Gentiil 2.5 flash
Code
Coundlsoo
100
ground Spec
Mes.--t-.. Bot
46-
Figure 9: Impact of contextual information on CodeGen and SpecGen performance.
Providing ground truth specification benefits CodeGen. Providing ground truth specifications as context consistently improves CodeGen performance across models. Since the ground truth specifications cannot be used directly as code (as explained in Section 3.2 all CodeGen improve- ments rely on semantic understanding of the reference specification. On the contrary, providing ground truth code as context shows minimal or negative improvement for SpecGen. While it is possible for LLMs to directly use the ground truth code in the specification, manual inspection of our evaluation results reveals no evidence of such behaviors. This is likely because using code as specification is uncommon in standard development practices, and our prompts C.3 ask LLMs to focus on constraining code behavior rather than replicating implementation details. The asymmetry in using ground truth information for CodeGen versus SpecGen suggests that formal specifications effectively constrain and guide code synthesis, while verbose code implementations may introduce noise to over-constrain specification generation rather than providing helpful guidance. Moreover, replacing ground truth with LLM- -LM-ggnnrrted artifacts generally degrades performance, indicating that combined tasks are more challenging than individual tasks.
Qualitative case studies. We present detailed qualitative case studies with analysis of failure modes and success patterns across different tasks in Appendix B
CONCLUSION AND DISCUSSION
We have introduced VERINA, a comprehensive benehmark comprising 189 carefully curated examples with detailed task descriptions, high-quality codes and specifications in Lean, and extensive test suites with full line coverage. This benchmark enables systematic assessment of various verifiable code generation capabilities, and our extensive evaluation result presents substantial challenges that expose limitations of state-of-the-art language models on verifiable code generation tasks. We hope that VERINA will serve valuable resource by providing both rigorous evaluation framework and clear directions towards more reliable and formally verified automated programming systems.
Limitations and future work. Despite advancing the state-of-the-an in benchmarking verifiable code generation, VERINA has several limitations. First, its size (189 examples) is modest, scaling to larger dataset suitable for finetuning likely requires automated annotation with L.M assistance. Second, it emphasizes simple, standalone coding problems, which is well-suited for benchmarking but not fully representative of complex real-world verification projects (Klein et al. 2009 Leroy et al., 2016). Our results demonstrate that current models struggle with VERINA, especially on ProofGen, with performance dropping substantially on harder instances (see Appendix D). indicating these fundamental capabilities must improve before tackling more difficult verification challenges. Third, while our current evaluation pipeline overcomes the limitation of current L.M theorem provers using comprehensive testing, the future advances in LLM theorem prover capabilities can enable stronger formal guarantees. Fourth, extending VERINA to ATp-based verification system like Dafny (Leino) 2010) or Verus (Lattuada al. 2023) can strengthen VERINA'S generalizability but requires significant effort, and we leave this as an important future work. Finally, while Lean programs in VERINA are newly written, the underlying task topics are drawn from widely used sources, posing a risk of data contamination. We provide detailed data contamination analysis and discussion in Appendix B
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This material is based upon work supported by the National Science Foundation under grant no. 2229876 and is supported in part by funds provided by the National Science Foundation, by the Department of Homeland Security, and by IBM. Any opinions, findings, and conclusions or recom- mendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or its federal agency and industry partners.
ETHICS STATEMENT
We adhere to the ICLR Code of Ethics and ensure compliance with all relevant dataset licenses, as detailed in Appendix C.I All data used in this work are publicly available and collected strictly for academic research purposes with proper citation and attribution.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our work. All code, benchmark datasets, and evaluation pipelines introduced in this paper are included in the supplementary materials, accompanied by detailed instructions for setup and usage. The dataset construction processes are described in Section 3.2 The evaluation metries are described in Section Additional implementation details and experimental settings are described in the appendix.
REFERENCES
Pranjal Aggarwal, Bryan Parno, and Sean Welleck. AlphaVerus, Bootstrapping formally verified code generation through self-improving translation and treefinement. arXiv preprint arXiv:2412.06176. 2024. 2
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint AXXv:2108.00732 2021. 3
Clark Barrett and Cesare Tinelli. Satisfiability modulo theories. Handbook of model checking, 2018.
Clark Barrett, Swarat Chaudhuri, Jim Grundy, Pushmeet Kohli, Fabrizio Montesi, and Leonardo de Moura. CSLib: A Foundation for Computer Science in Lean 4. https://www. http::///www.... csw.b. io/ Accessed: 2025-12-1. [6
Karthikeyan Bhargavan, Cédric Fournet, Markulf Kohlweiss, Alfredo Pironti, and Pierre-Y Yver Strub. Implementing TLS with verified cryptographic security. In Symposium on Security and Privacy, 2013. 0
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-EE: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. IEEE Transactions Software Engineering, 2023. doi:
10.1109/TSE.222332277444 5
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 1xXiv:2107.03774 2021. 1
18
Tianyu Chen, Shuai Lu, Shan Lu, Yeyun Gong, Chenyuan Yang, Xuheng Li, Md Rakib Hossain Misu, Hao Yu, Nan Duan, Peng Cheng, et al. Automated proof generation for Rust code via self-evolution. In International Conference on Learning Representations (ICLR), 2025. 2 g
Joseph W Cutler, Craig Disselkoen, Aaron Eline, Shaobo He, Kyle Headley, Michael Hicks, Kesha Hietala, Eleftherios loannidis. John Kastner, Anwar Mamat, et al. Cedar: A new language for expressive, fast, safe, and analyzable authorization. Proceedings of the ACM on Programming Languages, (000PSLA 2024.

---

Published as a conference paper at ICLR 2026
Markus de Medeiros. Muhammad Naveed, Tancrede Lepoint, Temesghen Kahsai, Tristan Ravitch, Stefan Zetzsche, Anjali Joshi, Joseph Tassarotti, Aws Albarghouthi, and Jean- Saan-Baptiste Tristan Verified foundations for differential privacy. In Programming Language Design and Implementation (PIDI), 2025a. 2
Markus de Medeiros, Muhammad Naveed, Tancrède Lepoint, Temesghen Kahsai, Tristan Ravitch, Stefan Zetzsche, Anjali Joshi, Joseph Tassarotti, Aws Albarghouthi, and Jean-Baptiste Tristan. Verified foundations for differential privacy. Proceedings of the ACM on Programming Languages,
9(PLDD)::094-1111, 2025b. T6
Leo de Moura. How the Lean language brings math coding and coding math.
https:/ www. http::/:/wwwwwaaaanoooo.. amazon math-to-codiig wwwt:::/ccccccccccccecccococccte http:://www..aaoo..ccccccoo//oo/
lean- anquage bribbring brings-
coding math Accessed: 2025-09-24. 2 16
Leonardo De Moura and Nikolaj Björner. Z3: An efficient SMT solver. International conference
on Tools and Algorithms for the Construction and Analysis of Systems, 2008.
Quinn Dougherty and Ronak Mehta. Proving the coding interview: A benchmark for formally verified code generation. arXiv preprint arXiv:2502.05714, 2025. 2
Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K Lahiri. Can large language models transform natural language intent into formal method postconditions? Proceedings of the
ACM on Software Engineering. 2024. 2
Wen Fan, Marilyn Rego, Xin Hu, Sanya Dod, Zhaorui Ni, Danning Xie, Jenna DiVincenzo, and Lin Tan. Evaluating the ability of large language models to generate verifiable specifications in verifast.
arXiv preprint
2024. 2
Klint
Finley.
How developers spend the time they save thanks to Al coding and htt.://w......aaa.aaaa.caaaannaa aireewaiiooww developers Accessed: 2025-
tools.
05-10. [
Emily First, Markus N Rabe, Talia Ringer, and Yuriy Brun. Baldur: Whole-proof generation and repair with large language models. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023.
Google DeepMind. AI achieves silver-medal standard solving international mathematical e/di http:///gggggggoaeoo//////o/////
olympiad problems. https:://deppmindd..gg -ew.eeessss-----re---raa-l------
2024.
Ronghui Gu, Zhong Shao, Hao Chen, Xiongnan Newman Wu, Jieung Kim, Vilhelm Sjöberg, and
David Costanzo. CertiKOS: An extensible architecture for building certified concurrent OS kernels. In Symposium Operating Systems Design and Implementation (OSDI), 2016.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. Neural Information Processing Systems (NeurIPS). Datasets and Benchmarks Track, 2021.
17
Kesha Hietala and Emina Torlak. Lean into verified software development. https
// //aws www.aaaaw......caaaaaooo hhtp:/llos////// http://logsssppooooooe hptp///penoooeeaaaeaeeaa http://wwwooooooooooooooo////aaa into verified software
development// 2024.
Son Ho and Jonathan Protzenko. Aeneas: Rust verification by functional translation. Proceedings of the ACM on Programming Languages, 6(ICFF)::11-741, 2022, 2 16
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.122 12186. 2024. [6
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama Koushik Sen, and lon Stoica. LiveCodeBench: Holistic and contamination free eval- uation of large language models for code. In International Conference on Learning Representations
(ICLR), 2025.
17
12

---

Published as a conference paper at ICLR 2026
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference Learning Representations (ICLR), 2024. [
Eirini Kalliamvakou. Research: Quantifying GitHub Copilot's Impact on Developer Productivity and
github- crccuub-copiii
Accessed: 2025-05-10.
Omar Khattab, Amav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self- improving pipelines. In International Conference on Learning Representations (ICLR), 2024.
18
Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, Thomas Sewell, Harvey Tuch, and Simon Winwood. seL4: Formal verification an OS kernel. In Symposium on Operating systems principles (SOSP), 2009.
T0
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:221 15533, 2022. T6
Andrea Lattuada, Travis Hance, Chanhee Cho, Matthias Brun, Isitha Subasinghe, Yi Zhou, Jon Howell, Bryan Parno, and Chris Hawblitzel. Verus: Verifying rust programs using linear ghost types. Proceedings of the ACM on Programming Languages, 2023.
16
Lean Prover Community. Plausible: A property testing framework for Lean 4 that integrates into the tactic framework. lnttps://githbb wttp:///aaaoooeeeeooooooaaaa///a
2024. 7
leanprover. leanprover/LNymm: Armv8 Native Code Symbolic Simulator in Lean. https://
github. wttp.///eaaoooer/e//ssss./ Accessed: 12-1-12-1. [6
K Rustan M Leino. Dafny: An automatic program verifier for functional correctness. In International Conference on Logic for Programming Artificial Intelligence and Reasoning (LPAR), 2010.
16
Xavier Leroy, Sandrine Blazy, Daniel Kästner, Bernhard Schommer, Markus Pister, and Christian Ferdinand. CompCert-a formally verified optimizing compiler. In Embedded Real Time Software and Systems (ERTS), 2016.
T0
Yong Lin, Shange Tang. Bohan Lyu, Ziran Yang, Jui-Hui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng. Jiawei Ge, Jingruo Sun, et al. Goedel-provers Scaling formal theorem proving with scaffolded data synthesis and self-correetion. arXiv preprint arXiv:2508.085: 03613, 2025. 2 9
Evan Lohn and Sean Welleck. miniCodeProps: a minimal benchmark for proving code properties. arXiv preprint arXiv:2406.11915 2024.
Chloe Loughridge, Qinyi Sun, Seth Ahrenbach, Federico Cassano, Chuyue Sun, Ying Sheng, Anish Mudide, Md Rakib Hossain Misu, Nada Amin, and Max Tegmark. DafnyBench: A benchmark for formal software verification. Transactions on Machine Learning Research, 2025.
Minghai Lu, Benjamin Delaware, and Tianyi Zhang. Proof automation with large language models. In International Conference on Automated Sofwware Engineering (ASE), 2024.
Lezhi Ma, Shangqing Liu, Yi Li, Xiaofei Xie, and Lei Bu. SpecGen: Automated generation of formal program specifications via large language models. In International Conference on Software Engineering (ICSE), 2025. 3
Mathlib community. The Lean mathematical library. In Certified Programs and Proofs 2020)
2020.
13

---

Published as a conference paper at ICLR 2026
Mathlib Community. Completion of the liquid tensor experiment. community htt.ha/...//bllooo/ttt/eeeeeeeee finaat///ieeeeaa// 2022.
https://leanprot
Md Rakib Hossain Misu, Cristina V Lopes, Iris Ma, and James Noble. Towards Al-assisted synthesis of verified Dafny methods. Proceedings of the ACM on Software Engineering. 2024. 2 5
Leonardo de Moura and Sebastian Ullrich. The Lean theorem prover and programming language. In International Conference on Automated Deduction (CADE), 2021.
Eric Mugnier, Emmanuel Anaya Gonzalez, Nadia Polikarpova, Ranjit Jhala, and Zhou Yuanyuan. Laurel: Unblocking automated verification with large language models. Proceedings of the ACM on Programming Languages, 2025.
Nethermind. Clear-Prove Anything About Your Solidity Smart Contracts.
https
7/ www. oggpeeermmnnn.../oooll/g/og////e proverrrrve- iiggthhnngggg about your-
sol -ttiity-saattttoootoroeeaaaaeeee Accessed: 2025-12-1. 16
OpenAl. Introducing OpenAl o3 and 04-mini. ptttps://peeaa..com/iii
introducing--3- 04 miini/
Accessed: 2025-09-24. 2
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of Github Copilot's code contributions, Symposium on Security and Privacy, 2022.
Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. Can large language models reason about program invariants? In International Conference on Machine Learning (ICML), 2023,
Jay Peters.
More than a quarter of new code at Google is generated by AL
https: /topss//wwwrr //ww://wwwwwwoo theverggeeeeeeeeeeeeee www.ccmmmmm///m
cccm//2224400///228887///0o2oee0 2024.
generated-a1-g3-222
ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, et al. DepSeek-Proverr V2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. arXiv preprint arXiv22504.28811, 2025. g
Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. Leveraging automated unit tests for unsupervised code translation. 2022. 5
Daniel Selsam, Sebastian Ullrich, and Leonardo de Moura. Tabled typeclass resolution. arXiv preprint arXiv:2001.04301 2020. 6
Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett. Clover: Closed-loop verifiable code
generation. In International Symposium Al Verification, 2024.
4
17
40
Amitayush Thakur, George Tsoukalas, Yeming Wen, Jimmy Xin, and Swarat Chaudhuri. An
in-context learning agent for formal theorem-proving, arXiv preprint arXiv:2310.04355 2023.
Amitayush Thakur, Jasper Lee, George Tsoukalas, Meghana Sistla, Matthew Zhao, Stefan Zetzsche, Greg Durrett, Yisong Yue, and Swarat Chaudhuri. Clever: A curated benchmark for formally
verified code generation. arXiv preprint arXiv:2505. 13938, 2025.
S 23
Kyle Thompson, Nuno Saavedra, Pedro Carrott, Kevin Fisher, Alex Sanchez-Stemi Yuriy Brun, João F Ferreira, Sorin Lerner, and Emily First. Rango: Adaptive auumenel- proving for automated software verification. In International Conference Software Engineering (ICSE), 2025. 2
Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models. In International Conference Software Engineering (ICSE), 2025.
14

---

Published as a conference paper at ICLR 2026
Cheng Wen, Jialun Cao, Jie Su, Zhiwu Xu, Shengchao Qin, Mengda He, Haokun Li, Shing- Shing-Chi Cheung, and Cong Tian. Enchanting program specification synthesis by large language models using static analysis and program verification. In International Conference Computer Aided Verification (CAV), 2024. 2
Chenyuan Yang, Xuheng Li, Md Rakib Hossain Misu, Jianan Yao, Weidong Cui, Yeyun Gong, Chris Hawblitzel. Shuvendu Lahiri, Jacob R Lorch, Shuai Lu, et al. Autoverus: Automated proof generation for rust code. Proceedings of the ACM on Programming Languages, 900PPLA22;
3454-3482, 2025.
2
Kaiyu Yang, Gabriel Poesia, Jingxuan He, Wenda Li, Kristin Lauter, Swarat Chaudhuri, Dawn Song. Formal mathematical reasoning: A new frontier in AL. arXiv preprint arXiv: arXtv:2412.16075, 2024. 0
Lichen Zhang, Shuai Lu, and Nan Duan. Selene: Pioneering automated proof in software verification. In Annual Meeting of the Association for Computatioonl Linguistics (ACL). 2024. 2
Yi Zhou, Jay Bosamiya, Yoshiki Takashima, Jessica Li, Marijn Heule, and Bryan Parno. Mariposa: Measuring SMT instability in automated program verification. In International Conference on Formal Methods in Computer-Aided Design (FMCAD), 2023. 3
15

---

Published as a conference paper at ICLR 2026
A EXTENDED DISCUSSION ON THE USE OF LEAN IN VERINA
This appendix provides extended discussion on the rationale for using Lean in VERINA, demonstrates its growing role in production code verification beyond mathematics, and explains how VERINA'S 's findings transfer to broader verification ecosystems including automated theorem proving (ATP) systems like Dafny (Leino) 2010) and Verus (Lattuada et al., 2023).
Lean in Production Code Verification. While Lean originated in formalizing mathematics, recent years have witnessed Lean's substantial adoption for production code verification across diverse domains. Amazon Web Services (AWS) has invested heavily in Lean as verification infrastruc- ture for critical production systems (de Moura). For example, AWS's Cedar project (Cutler et al.) 2024) employs Lean to verify security properties of their policy language for cloud services au- thorization, handling authorization decisions for millions of resources. The LNSym (laanprover) project demonstrates Lean's is capability in low-level verification by providing a symbolic simulator for Armv8 machine code, enabling verification at the hardware- soottae interface. Additionally, SampCert (de Medeiros et al.) 2025b) represents a verified implementation of randomized algorithms systems programming language has created demand for formal verification tools, and Lean has emerged as a viable platform. Aeneas (Ho) & Protzenko 2022) translates Rust programs into Lean for formal verification, enabling developers to prove properties about safe systems code. In blockchain, the Clear project (Nethermind) provides an interactive formal verification tool for Ethereum smart contracts using Lean, addressing the critical need for mathematical guarantees in high-stakes environments. Beyond direct production use, the CSLib project (Barrett et a represents a collaborative effort across academic institutions and industry partners to formalize undergraduuae--lveè computer science in Lean, establishing reusable foundations for future verification projects. These diverse applications demonstrate that Lean's 's adop- tion extends well beyond mathematical formalization into practical software verification domains, validating its relevance as VERINA'S platform.
deployed in production AWS services. The rise of Rust
Transferable Insights Across Verification Paradigms. While Lean's syntax differs from verification systems leveraging Automated Theorem Prover (ATP), the fundamental challenges in verifiable code generation are largely shared across verification paradigms. Both ITP and ATP frameworks require: (i) generating correct code and sound, complete specifications, and (ii) identifying key properties such as loop invariants for constructing proofs. VERINA'S 's CodeGen and SpecGen tasks evaluate capabilities equally critical in ATP systems, which require the same semantic understanding regardless of surface syntax. For example, generating a sound and complete pre/post-conditions same semantic understanding specifying these properties correctly, not in the syntactic representation. For ProofGen, ATP systems automate proof search via SMT solvers but are not guaranteed to succeed on complex properties. When automation fails, LLM must generate additional guidance through assertions and annotations, which requires similar reasoning capabilities VERINA evaluates through explicit proof construction. Furthermore, Lean's 's dependent type system offers stronger expressiveness than the SMT-based specifications used in ATP systems, enabling verification of programs with higher-order functions and specifications. This greater expressiveness ensures that insights from VERINA generalize to ATP systems and beyond. These insights inform development of M.ss-siie code generation and verification workflow in both ITP and ATP paradigms, demonstrating that VERINA'S 's findings extend beyond Lean-specific details address fundamental challenges in verifiable code generation.
Dafny requires the generating pre/post-conditions in Lean. The difficulty lies in
B EXTENDED DISCUSSION ON DATA CONTAMINATION ANALYSIS
VERINA draws algorithmic problems from popular sources, raising the possibility that models may have encountered similar problems during training and undermine the evaluation. To address this, we conducted systematic data contamination analysis and discuss how VERINA properly mitigates these risks.
Direct contamination analysis. We performed N-gram overlap analysis between VERINA'S 's Lean ground truth solutions and the bigcode/the-stan pretraining dataset (Kocetkov al. 2022), which contains approximately 550 million rows of coding files sourced from GitHub. Following standard decontamination practices like Qwen-2.5 Coder (Hui et al. 2024), we conducted 10-gram overlap
16

---

Published as a conference paper at ICLR 2026
detection and found zero matches. This confirms that VERINA'S Lean artifacts novel and not present in public pretraining corpora, therefore no risk direct contamination.
Verification tasks differ fundamentally from simple code generation. In verifiable code generation LLM are required to perform specification and proof generation beyond code generation, which are fundamentally different skills. Our evaluation shows that even for algorithmically familiar problems, the best LLMs struggle significantly with formal specification generation and proof generation, demonstrating that memorized algorithmic solutions do not transfer verification tasks in current LLMs, effectively eliminating indirect contamination risks. The stark contrast between CodeGen and ProofGen success rates, despite both potentially benefiting from algorithmic familiarity, demonstrates that LLMs cannot formally reason about algorithms they may have seen and thus suggest a lack of deep understanding of the algorithm they use.
C DATASETS AND DETAILED EXPERIMENTAL SETUP
C.1 LICENSE
We ensure compliance with all relevant licenses: MBPP-DFY-50 (Misu et al. 2024) is licensed under GPL-3.0, while both CloverBench (Sun et al. 2024) and LiveCodeBench (Jain al. 2025) use MIT licenses. Our datasets VERINA will be licensed under GPL-3.0. Consistent with established research practices (Hendrycks al. 202 Jain al. 2025). we only use publicly available materials from competitive programming platforms such as LeetCode. Our collection and use of these problems strictly for academic research purposes, and VERINA involves no model training or fine-tuning processes.
C.2 MODEL. CONFIGURATIONS AND COMPUTE
Table 3 presents the configuration details and total experiment costs for all twelve evaluated LLMs. For LLMs, we use a temperature of 1.0 and a maximum output token budget of 10,000. For reasoning models, use default settings of reasoning efforts budgets. We host DeepSeek Prover V2 7B, Goedel Prover V2 32B, and Qwen 235B-A22B locally using NVIDIA H100 80GB GPUs. We other LLMs through APIs, for which provide the total cost and cost per million tokens. The costs marked with asterisks include the additional expenses incurred during iterative proof refinement experiments, which required up 64 refinement attempts per datapoint.
Table 3: Detailed configurations and costs for evaluated LLMs.
Price ($/IM tokens)
Vendor
Model Name
Checkpoint
Type
Cost
(Input / Output)
4o-mini
gD1-4o-mini-20244-7-18
$0.15 $0.60
$10.94
GPT 4o
gpt-4o-2024-08-06
$2.50 $10.0
$153.01
OpenAl
GPT 4.1
12t-4.1-2025-04-14
APT
$2.00 $8.00
$453.72"
04-mini
od-mini-2025-00-16
API
$1.10 $4.40
$894.38'
o3
03-2025-04-16
API
$2.00 $8.080000 00
$121.70
Claude Sonnet 3.7
clmud-3377:smme222255201
API
$3.00 $15.0
$777.60'
Anthropic
Claude Opus 4.0
claude-qpus-42222555144
API
$15.00 $75.0
$1197.39
Google
Gemini 2.5 Flash
gemin-225-faahhppeeiivv----11
$0.15 $0.60
$295.20'
DeepSeek V3
DeepSeck.V3.0324
API
$1.25 $1.25
$51.15
DeepSeek
DeepSeek Prover V2 7B
AkepSeckk-rroee--2277
GPU
Qwen
Qwen3 235B-A228
Dwen3-235B-A228--P5
GPU
Gocdel-LM
Gocdel Prover V2 32B
Goodel-Prova--2-3228
GPU
Including costs for iterative proof refinement experiments.
17

---

Published as a conference paper at ICLR 2026
C.3 PROMPTS
We employ a consistent 2-shot prompting approach across all models and tasks to enhance output format adherence and task understanding. The 2-shot examples excluded from the final benchmark evaluation. For each problem instance, we sample 5 responses from each model and calculate pass @ metrics (Chen et al., 2021) using these samples to ensure robust evaluation statistics. We utilize DSPy (Khattab et al. 2024) for structural prompting. We provide the detailed prompts in the following: Prompt for CodeGin, Prompt for SpecGen, Prompt 3 for ProofGen, and Prompt for ProofGen with iterative refinement. For DeepSeek Prover V2 7B and Goedel Prover V2 32B, we used their own prompt templates for ProofGen to achieve optimal performance. Our control experiments revealed that using the standard DSPy prompts for these models resulted in 0% success rate due to severe cesrructionfoolowwee failures. Specifically, they are not able to produce parsable output formats using the standard DSPy prompts.
18

---

Published as a conference paper at ICLR 2026
Prompt (CodeGen)
Instructions
You are expert in Lean programming and theorem preving.
Please generate Lean program that finishes the task described
4 in
'task_des dessskdeccrrption using the template provided in 'task_ted task_template'. The 'task_template is Lean code snippet that contains
placeholders
(warpped with ((H) for the code
generated.
The program should:
Be well-dccumented with comments if necessary
Follow Lean 4 best practices and use appropriate Lean syntax and features
Do NOT use lean syntax or features
DO NOT import Std or Init
:int:
Use alil! instead of alil when is array list when necessary
Input Fields
task description
Description of the Lean 4 programming task to be solved.
task template
Lean 4 template with placeholders for code generation and optional reference specification.
Output Fields
imports
Imports needed for 'code', Keep it empty if not needed.
code_aux
Auxiliary definitions for 'ode'' Keep empty not needed.
code
Generated Lean 4 code following the template signature and complete the task.
19

---

Published as a conference paper at ICLR 2026
Prompt (SpecCien)
Instructions
You are expert in Lean programming and theorem proving. Please generate Lean specification that constrains the program implementation using the template provided in 'task_template". The 'task_ "eakk_emplate" is Lean code snippet that contains
+ placeholders
(warpped with ((H) for the spec be generated. The preconcition should be as permissive possible, and the 4 postconnition
should model a sound an complete relationship between input and output of the program based the "task_description".
The generated speci specification should:
Be vell-aocumented with comments if necessary
Follow Lean 4 best practices and use appropriate Lean syntax and features
DO NOT use Lean syntax or features
DO NOT import Std or Init Only use 'precond_aux'' or 'postcond_aux' when canact express the precondition or postcondition in the main body of the specification
add 8 (reducible,, simp] attribute to the definitions in
+ precond_aux or 'postcond_aux'
Hint:
a[il! instead of [[i] when is array or list when necessary
Input Fields
task description
Description of the Lean 4 programming task to be solved.
task template
Lean template with placeholders for specfication generation and optional reference code.
Output Fields
imports
Imports needed for 'precond' and 'postcond', Keep it empty if needed.
precond_aux
Auxiliary definitions for 'precond', Keep it empty if not needed.
precond
Generated Lean 4 code speci fying the precondition.
postcond_aux
Auxiliary definitions for 'postcond', Keep it empty if not needed.
postcond
Generated Lean code specifying the postcondition.
20

---

Published as a conference paper at ICLR 2026
Prompt (ProofGen)
Instructions
You are expert in Lean programming and theorem proving. Please generate a Lean proof that the program satisfies the + specification
using the template provided in 'task_ Seask_teplaaee. The 'task_template is Lean code snippet that contains
placeholders
(warpped with ((H) for the proof The proof should:
generated.
Be well-documented with comments if necessary
Follow Lean 4 best practices and use appropriate Lean syntax and features
Do NOT use lean syntax or features
DO NOT import Std or Init
Do NOT use cheat codes like 'sorry'
Hint:
Unfold the implementation and specification definitions when
4 necessary
Unfold the precondition definitions at h_precond when necessary
Input Fields
task task description
Description of the Lean programming task to be solved.
task template
Lean template with code and speci fication to be proved, and placeholders for proof generation.
Output Fields
imports
Imports needed for 'proof'. Keep it empty if not needed.
proof aux
Auxiliary definitions and lemma for 'proof'. Keep it empty if not needed.
proof
Generated Lean 4 proof that the program satisfies the specification
21

---

Published as a conference paper at ICLR 2026
Prompt (ProofGen with Iterative Refinement)
Instructions
You are expert in Lean programming and theorem proving. Please generate a Lean proof that the program satisfies the + specification
using the template provided in "tasktteppatee The 'task_template is Lean code snippet that contains
placeholders
(warpped with ((H) for the proof The proof should:
generated.
Be well-documented with comments if necessary
Follow Lean 4 best practices and use appropriate Lean syntax and features
Do NOT use lean syntax or features
DO NOT import Std or Init
Do NOT use cheat codes like 'sorry'
Hint:
Unfold the implementation and specification definitions when
4 necessary
Unfold the precondition definitions at h_precond when necessary
Furthermore, 'prev_error' is the message from the previous 4 proving
attempt.
Please use the 'prev_imports', 'uu and 'prev_proof'
references improve the generated proef.
You can ignore unused variable warnings in the error message.
Input Fields
task description
Description of the Lean 4 programming task to be solved.
task template
Lean template with code and speci proved, and placeholders for proof generation.
prev impprts
Previously generated imports for reference.
prev-proof
Previously generated proof auxiliary for reference.
prev-proof
Previously generated proof for reference.
prev_error
Error message from the previous proving attempt.
Output Fields
imports
Imports needed for 'proof'. Keep it empty if not needed.
proof aux
Auxiliary definitions and lemma for 'proof', Keep it empty if not needed.
proof
Generated Lean 4 proof that the program satisfies the specification.
22

---

Published as a conference paper at ICLR 2026
C.4 COMPARISON WITH CLEVER
As summarized in Table 4 CLEVER (Thakur et al. 2025) only supports evaluation of specification generation and specification-guieed code generation. It lacks evaluation support for code generation, proof generation, specification inference from code, and fully end-to-end verifiable code generation. In contrast, VERINA fully covers all three foundational tasks and their flexible combinations, enabling more comprchensive assessment of realistic verification workflows.
Moreover, CLEVER's SpecGen evaluation assumes access sound and complete ground truth specification for certification. However, if such ground truth specification is already available, there little practical value in generating another, as developers would simply use the existing one. This reliance on ground truth specifications therefore limits CLEVER's applicability and prevents it from reflecting real-world scenarios. contrast, VERINA employs a combined evaluation framework for specification (Section 4.1 leveraging both formal proving and comprehensive testing, which can reliably assess specification quality even when formal proof's are inconclusive.
Table 4: A detailed comparison of VERINA with the concurrent work CLEVER (Thakur et al. 2025 supported tasks in verifiable code generation. means fully supported, means unsupported.
Foundational Tasks (Seciio
Task Continations SSeclo
GodeGen
SpeeGen
Prosftie
Specileation-Ciie Guiiie Gade Genrration
Specilictionn
From Cide
Verilable Oade Guneration [Donc Code Spac Porn))
(Dee - Unle
(llce
Spece
(Uido=Sppc - Preeds
[Dooc- -
Ende Phoof)
(Doc Code Spac Phool)
23

---

Published as a conference paper at ICLR 2026
C.5 IMPLEMENTATION OF EVALUATION METRICS IN LEAN
In Section[4.] we provide a high-level description of our evaluation metrics for the three foundational tasks of verifiable code generation. Now describe how implement these metries in Lean 4.
Proof evaluation. We directly evaluate generated proofs using the Lean compiler and filter out any proofs containing placeholders, as described in Section 4.1
Code evaluation. We evaluate generated code on unit tests using #guard statements in Lean 4, ensuring the implementation produces correct outputs for given inputs. The evaluation harness for generated codes is illustrated in Figure TO
Lnguei Mathiib import Plausibie
Definal Lons Fox codee EceooveBleeeni ondited fox bruvity
Rvaluate code cocrrotnee waing powitiie test cases
4guazd cenoveBlement (412, 2, 3, 4, 511 (2) Byy soccy) -- [#11, 2, 4, 51) -hould paas
Figure 10: Example (verina.basica, ba.bsiic_2)) Evaluating the correctness of L..M-generated code using unit tests in Lean 4.
Specification evaluation. Recall Section we define the soundness and completeness of pre-condition P and post-condition Q in relation to their ground truth counterparts complete iff VT. P( = P(T); (iii) Q is
P and Q: (i) P is sound iff VI. P(z) = P(T); (ii) sound iff VΣ, P(T) A (( v) = (z, v): (iv) Q is complete iff vz, y. P(T) A Q Q v).
Our specification evaluation pipeline first attempts to establish the soundness and completeness of generated specifications against the ground truth using LLM-based provers. When the proving step is inconclusive, the evaluator proceeds to testing, where we only require that and are from our test suite. Our quality assurance process in Section 3.2 ensures that all ground truth pre-conditions and post-conditions pass our positive tests and not pass our negative tests. Therefore, we can simplify the soundness and completeness metrics follows:
Deciding the soundness of P is equivalent verifying whether P(z) holds for all positive tests in our suite. This because for all negative tests Σ, P(z) does not hold, making P(z) P(T)
true by default. For positive tests I, P(#) holds, and P(T) + P(z) is true iff P(z) is true.
Similarly, deciding the completeness of P is equivalent to verifying whether P(T) does not hold for all negative tests in our test suite.
The soundness of Q can be evaluated using our negative test cases.
The completeness of Q can be evaluated using our positive test cases.
For each test evaluation, we employ the two-step approach described in Section 4.1 First, check if the relationship (with the specific test case incorporated) is directly decidable in Lean on the test case via decide. If not, we proceed to property-based testing using plausible tactic. The evaluation implementation in Lean 4 is illustrated in Figures
and 12
To further examine the role of proofs within our evaluation pipeline, we analyze how often LLM- based provers succeed in establishing the soundness and completeness of generated specifications against the ground truth. In this setup, we use 04-mini and Claude Sonnet 3.7 construct Lean proofs for the required logical relationships and compare the results with the testing-based evaluation results. Table summarizes the outcomes. Proof success rates are very low, below 4% across all cases, while testing recognizes more than 40% of generated specifications sound and complete. We have examined all specifications marked as sound and complete by formal proofs. We observe that whenever proofs succeed they always agree with testing, confirming their validity. However, when proofs fail but testing reports correctness, manual inspection of 20 randomly selected disagreements shows that the testing outcome is always correct.
These results indicate that while proofs provide the formal guarantees the evaluation results when they succeed, current L.M provers are incapable of serving as reliable metric with high inconclusive
24

---

Published as a conference paper at ICLR 2026
rates. Testing- festing-based evaluation methods achieve high empirical accuracy and reliably identify sound and complete specifications even when proof' are inconclusive and therefore play an important role in ensuring robust and comprehensive specification evaluation when the proving-basee evaluation is inconclusive. LLMss proof capabilities and stability are rapidly improving with newer prover models, stronger proof search agents, and new automation tacties like grind. This will make our SpecGen metric increasingly powerful over time.
We further analyze the sensitivity of our SpecGen evaluation (on 04-mini results) the property- based testing budget, varying the number of generated test instances from 10 to 2,000 across random seeds. Table demonstrates that VERINA'S choice of 1,000 test instances sufficient, as increasing the budget to 2,000 instances provides minimal additional benefit, and the standard deviations across seeds are small, All evaluation components contribute meaningfully determination, demonstrating the comprehensiveness of our specification evaluation approach. The most variable component (property- based testing) contributes less than 13% of cases. As LLM--based theorem provers continue to improve, expect the "Guaranteed to Hold (proved)" percentage to increase, providing more formal guarantees the SpecGen evaluation.
the final
Table 5: Evaluation of generated specifications for soundness and completeness. Rows indicate the model that generated the specification, while columns indicate the prover used to check correctness. The last column shows results from our testing-bact based evaluation.
Proved sound and complete by (%)
Spee generated by
Sound and complete by testing
o4-mini
Claude Sonnet 3.7
04-mini
3.7
1.6
51.0
Claude Sonnet 3.7
3.7
2.6
41.6
Table 6: Sensitivity analysis of SpecGen evaluation results (o4-mini) with varying property- -based testing budgets. Results are averaged over random seeds. The "Unknown" column includes the standard deviation across seeds.
Guarunfeed to Hold
Might Hold (%)
Guaranteed to Not Hold
Cannot Compile
Test Budget
Unknown (%)
(Proved) (%)
Unit Tests
PBT
(Counterexample) (%)
(%)
10
3.7
38.7
2.1
21.0
10.5 + 0.41
24.1
100
3.7
38.7
21.3
7.3 + 0159
24.1
38.7
5.7
7.2
24.1
2000
3.7
38.7
5.7
21.3
7.2 - 0.11
24.1
25

---

Published as a conference paper at ICLR 2026
inpozt Mathiib
2
izport Plausibie
Dofinitions for pro-condition [ssnore2lomenr_proooo enitted for bravity
Evaluate precoad souodnesa aich positive test casea (2)) Iquard decids Sennveelleens_peee
example : (remiveflenert ppecund (+11, 2, 3, 4, 81) (21) by Should pass
9
unfoid renoveElenert precoon
10
simp_alii (config
feilIEUcchaged :n calso }) samp lounfig :- FeilICUnchanged plausible lcontig
aunina: :- 1020, naxSize 100, nunBetries 20, candenSeed some 42)) exemple e : (rrnngee ementt precord (+)), 2, 3, 512 (2)) :- by Showld fail
13
14
unfold removellenerl powuondd
simp_alit
failICuchanged false
step (onnfig :o Fflhhhaede falae [+]
17
plausible lountig == auninas :- 1000, naxBise :- 100, musieliiin :- 20, candonneed i- sume 421)
18
precond cosplet 4guazd decide In Lemove@lement.pteecoo (4[3]) (2)):
neget
21
example : sesnovallenerr_prrcooo (B[1.) (2)) s- by -- Should pass
22
unfold rempveElenert prrecond
simp_allt lconZig [ COnhhaandd Calae
stmp (config se FUtchaane =- false [+] plausible lconfiig =- nunfra: == 1000, naxßize :- 180, nunBetriea :- 20, rardenSeed (- some 4213 example wwinveeleneett_poo
25
26
(A[1]) (2))
Should Fail
renoveElener precond
simp jconfig =- FfilTEfnehangee =- falae }) samp (ounfig := FailICUnchanget :- false 3) [x] 29
plausio'e lconfig numfrss 1000, naxSize 100, nunBetries 20, randenSeed 4233
Figure 11: Example (verina basic.29). Evaluating pre-condition soundness and completeness using unit tests in Lean 4.
2. Ergort Math 1b
Engort Plauxible
Beflort fons for post-gond-tion (eresvel)eeent prettppptootdtdot onlited brevicy
Evaluate posccond completenezs with posilive ceac cases
#guard example : (remiveElemert puosttttppootoud (#(1, 2, 3, 51) (2) (412, 2, 511 (by sarsyl) by
[renovel)
post.oond
2,
byy sorry))
Shoold pasa
9
untoid removeElenert, wwweeettpssttoorrreeeeeeeeeo postcord
10
a11 lconfig samp (ountig s= FwilI[Umchangee
Fail FFUnchange :- Falae 1)
false p. plausia.e Icontig = nuntrst 1- 1000, naxSize 180, nunRetries 20, randonSeed some 4213
13
exemple : ((renove wweeeeeeeeeee....mmmm postiond 181", 2, 3, 4, 5)) (2) (KE), 2, 4, 513 by serry)t i- by Should d fa
14
unfold senoveElenwtt ound
ssmp all! Icontig
TeUnchanged :- sinp (onnfig :o FiihhFUhheengee falae [x]
17
plausible lountig = auninas :- 1000, naxBise =- 100, iusBettiva :- 20, candundeed :- sume 421)
Kvaluste poscoond
negat
20
tquard decide 1n wetreesieeett_psceeeeeee 1f11, 2, 3, 4. 513 121 (tL, 3, 511 bby serry)
21
example : //rensveelement postcoon 141", , 2, 3, 4, 51) 12) (it, 2, 3, 513 bby sorryst 1- by Should pais
22
unfold removeElenert, ptcpentttttpptptopeeeee
simp_ Icontig
IeUnhhande 1- taloe
simp (config :- Fitchaaaeee =- false [+]
25
plausible Iconfig = ountra: =- 1000, naxiice :- 180, munletrria =- 20, candondeed :- sone 421) corryl)
(removeElenert. nerooveleeeet_ppsstcoe
(t11,
(2) 18t2, 2,
Should tail
peteeelleeet__psttccrr postcord
28
simp_alll icontig z- Eeichhuueddd :- talse })
29
samp (contig p- tailItUhetanged :- ralse [+]
ausio lconfig
nomfrst 1000, naxSize 189, nunRetries 20, randenSeed
4233
Figure 12: Example (verina.basical Evaluating post-condition soundness and completeness using unit tests in Lean 4.
26

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL EVALUATION RESULTS
Based on the construction methodology of VERINA datasets in Section 3.2 we categorize the problems translated from human-written Dafny datasets as VERINA-A and the problems written from scratch as VERINA-B.
VERINA-B is much more challenging than VERINA-A. The comparison between VERINA-A and VERINA-B in Figure 13 reveals substantial difficulty gaps on all three tasks. This demonstrates that problem complexity significantly impacts all aspects of verifiable code generation, and VERINA-B provides a valuable challenge for advancing future research in this domain.
Apmii Cludd Samst377
C7T Gamin Rahh
Cauce Opua 4.2
DanpSenc 93
Ownn 2350-222--55
(VERINA.A)
SoundtSCopplete
Proof
100
100
100
80-
80
R25
60
4..
32-8
(VERINA
Sound&Complete Score (VERINA
Proof Score (VERINA
100
100
100
Enorter
800
03
80
60-
40
323
264
20
184
Figure 13: p performance on three foundational tasks for VERINA-A and VERINA-B.
Achieving simultaneous soundness and completeness poses great challenge, particularly for post-conditions. As shown in Figure 14 the substantial performance gap between preconditions and postconditions confirms that generating complex input-output relationships remains signifi- cantly more challenging than input validation constraints. Furthermore, the drop performance when requiring both soundness and completeness simultancously- compared to achieving either individually demonstrates that partial correctness insufficient and justifies our comprehensive evaluation framework for specification quality.
Precond Sound
Precond Complete Score
Precond Sound&CComplet
XB
Inter
80
60-
60
475
40
Postcond Sound Score
Postcond Complete Score
Postcond Sound&Complete Score
Crei 2om
Enorbery
Drenbey
80
60
925
$0.5 X32
40-
332
20
323
Figure 14: Detailed performance of LLMSs on VERINA'S SpecGen task.
Code summarization does not consistently benefit SpecGen. We conducted an ablation study where replaced the full code reference with behavior-only summaries (generated by prompting LLMs to extract high-level contracts and behavior descriptions without implementation details). The results in Table show highly model-dependent effects: o4-mini improves slightly with summaries (51.0%
27

---

Published as a conference paper at ICLR 2026
- 53.2%) GPT 4.1 shows smaller difference (~42-44%) while Gemini 2.5 Flash performance is hurt by summaries (45.0% 37.3%). We note that LLM --M-eenerated code summaries can themselves be detail-heavy and potentially more confusing than the original code, especially when attempting to describe low-level implementation logic in natural language, which may explain why summaries do not consistently improve performance and can even hurt it.
Table 7: Ablation study on SpecGen performance using code summaries versus full code references. "Ref" indicates the reference provided to the model; "GT" is Ground Truth, "Gen" is Generated.
Model
No Ref (%)
GT Code (%)
Gen Code (%)
GT Summary (%)
Gen Summary (%)
14-mini
51.0
50.6
50.7
53.2
52.6
GPT 4,1
42.6
43.9 9
41.6
43.9 13.9
41.8
Gemini 2.5 Flash
45.0
43.3
4,588
37.3
37.4
Naive proof refinement gains diminish when problem is difficult. As shown in Figure 15| iterative proof refinement yields substantial improvements on simpler problems but only modest gains on more complex ones. For example, 04-mini improves from 7.41% 411 22.22% on VERINA-A after 64 iterations, while on VERINA-B B the success rate rises only from 1.23% to 6.17%. Specialized provers like Goedel Prover V2 and DeepSeek Prover V2 generally outperform general- purpose models, yet o4- mini remains surprisingly competitive on difficult instances, achieving stronger iterative refinement gains on VERINA-B. This suggests that while verifier feedback erucial, naive refinement strategies struggle to overcome the inherent complexity of challenging proofs, and that general purppoe LLMs can still contribute meaningfully in difficult settings.
Cidc SCOMO 3
kerolive Refrements (1ERNNFFLL))
Derad (VERNNN-
beralive Sedinements (E4NN6-I
Dioci Senesibi NEVNA RULL)
Di Senertic dee IVERINA
Dieci Generiion MENNN
Figure 15: Breakdown of iterative refinement versus direct generation across different subsets. Refinement yields large gains on VERINA-A but limited improvements on VERINA-B.
Iterative refinement increases proof verbosity. We manually inspected and analyzed the structural differences between the 46 human-written ground truth proofs and successful proofs generated by models. As shown in Figure 16. human proofs are highly concise, averaging 169.6 characters, as experts effectively utilize automation tactics (e.g., s simp, aesop) and standard library lemmas. In contrast, while single-pass LLM generation produces similarly short proofs, iterative refinement leads monotonic increase in verbosity, with the highest model (Gemini 2.5 Flash) reaching > 1200 characters after 64 iterations. Moreover, manual inspection reveals that L.MMs often cannot correctly identify or use relevant lemmas from the standard library, leading them explicitly prove tedious intermediate goals that humans would automate. This suggests improving LLMs' understanding of proof automation and lemma usage is a critical direction for future work.
Iterative refinement is more cost-effective than COPRA. We conducted budget-normalized analysis to compare the marginal utility of iterative refinement against the agentic COPRA framework using 04-mini (Figure 17). Iterative refinement proves significantly more cost-effective, achieving an 8,99% overall success rate at a 50k token budget compared to COPRA's 4. 4.76%. With the budget extended 350k tokens, iterative refinement scales to 14.29% while Copra saturates at 7 7.94%.
28

---

Published as a conference paper at ICLR 2026
D4mrki
GFT 8.L.
Claads Sereer
anoff
1260
Longth
660
200
20
Figure 16: Comparison of average proof length between hrittewritten ground truth and successful proofs.
with a substantially lower average cost per successful proof (57,594 tokens vs. 84,027). Stratified analysis reveals that returns diminish sharply with problem difficulty: while iterative refinement achieves steady gains on VERINA-A (reaching 22.22%) performance on the harder VERINA-B subset saturated early 3.70% (compared Copra's 's 1.23%), indicating that increased inference budgets alone cannot overcome fundamental reasoning gaps in complex verification tasks.
Sucuem 2.3
29100 20000
1ie Rudger
Figure 17: Budget-normalize comparison of proof success rates between iterative refinement and COPRA (using o4-mini) on the ProofGen task. Iterative refinement demonstrates higher marginal utility across all token budgets compared to the agentic COPRA framework, though performance gains on the harder VERINA-B subset remain limited for both approaches.
Detailed performance breakdown. Tables 8 16 provide detailed breakdowns of model per- formance across the three foundational tasks. They reveal that syntax incorrectness and use of non-existent library functions (as demonstrated in Appendix F represent the major problems, espe- cially for less capable models. Specifically, after manual inspection the evaluation result, Qwen 235B-A22B-FPE suffers from instruction following ability, failing output the desired format specified in our prompts (cf. Appendix CC.). The relatively low unknown percentages across most evaluations demonstrate that our specification evaluation metric reliable. Pre-conditions are gener- ally simpler than post-conditions, resulting in lower unknown rates during evaluation. More capable models often generate specifications with more complicated logical structures. leading to higher unknown percentages in post-condition evaluation. We present case study in Appendix F on the challenge of automatically evaluating L.M- -eMerreedt specifications.
our main results, we report the uncertainty from unknown cases using error bars, where the lower bound represents the Pass% in the table and the upper bound represents Pass UnknownS in the table.
ProofGen failure analysis. To systematically diagnose bottlenecks in proof generation, catego- rized failure modes across four top performmnnnng models up to 64 refinements, as detailed in Tables [7 to 19 Our analysis reveals distinct failure signatures: Gemini 2.5 Flash and GPT 4.1 primarily struggle with Incomplete Proofs (77.55% and 38. 38% respectively). often leaving goals unsolved after exhausting initial tactics. In contrast, Claude Sonnet 3.7 and o4-mini frequently resort Cheat
29

---

Published as a conference paper at ICLR 2026
Codes (c.g., sorry), which account for 48.53% and 28.30% of their failures, suggesting tendency to explicitly acknowledge their inability to complete the proof. Additionally, errors stemming from Unknown Identifiers, Tactics, and Constants consistently account for 15-30% of failures across all models, underscoring a pervasive lack of familiarity with the specific syntax and standard library of the Lean ecosystem.
30

---

Published as a conference paper at ICLR 2026
Table 8: Detailed performance of CodeGen.
Model
Cannot Compile %
Fail Unit Test
Pass %
GPT 4a-mini
70.1
28.6
GPT 4o
51.6
2.8
45.7
GPT 4.1
40.5
3.1
56.4
o4-mini
34.1
4.5
61.4
o3
25.8
1.6
72.6
Claude Sonnet
54.1
44.2
Claude Opus 4.0
30.6
2.7
66.7
Gemini 2.5 Flash
62.9
0.6
36.5
DeepSeek V3
62.3
1.7
36.0
Qwen 3 335B-A22B-FPS
80.0
0.0
20.0
Table Detailed performance of CodeGen on VERINA-A.
Model
Cannot Compile%
Fail Unit Test%
Pass
4o-mini
54.9
44.0
GPT 4o
35.7
2.6
61.7
GPT 4.1
27.4
2.8
69.8
o4-mini
28.3
3.0
68.7
o3
22.8
1.3
75.9
Claude Sonnet
0.4
54.0
Claude Opus 4.0
21.1
0.4
78.5
Gemini 2.5 Flash
49.3
0.6
50.2
DeepSeek V3
48.7
1.5
49.8
Qwen3 235B-A22B-FPS
67.2
0.0
32.8
Table 10: Detailed performance of CodeGen on VERINA-B.
Model
Cannot Compile %
Fail Unit Test%
Pass %
GPT 40-mini
89.9
1.7
8.4
GPT 4o
72.4
3.0
24.7
GPT 4.1
57.8
3.5
38.8
o4-mini
41.7
6.4
51.9
2.0
68.4
Claude Sonnet 3.7
65.2
3,5
31.4
Claude Opus 4.0
43.0
5.7
51.4
Gemini 2.5 Flash
80.7
0.7
18.5
DeepSeek V3
80.0
2.0
18.0
Qwen3 235B-A22B-FPS
96.8
0.0
3.2
31

---

Published as a conference paper at ICLR 2026
Table 11: Detailed performance of SpecGen for pre-condition.
Soundness
Completeness
Model
Cannot Compile %
Pass %
Fail
Unknown %
Pass %
Fail
Unknown %
4o-mini
40.8
58.2
0.0
47.5
11.8
0.0
GPT 4o
19.8
77.7
1.8
0.8
71.1
8.7
0,4
GPT 4.1
24.3
70.7
4.0
69.1
3.5
3.1
04-mini
5.4
91.0
3.0
82.1
1.8
03
6.2
88.7
1.6
3.5
82.0
9.6
2.1
Claude Sonnet 3.7
4,9
84,4
2.3
8.5
84,5
3.7
6.8
Claude Opus 4.0
4.5
86.6
1.7
7.2
87.1
3.3
5.1
Gemini 2.5 Flash
81.4
2.5
79.4
1.0
DeepSeck V3
43.7
54.3
0.8
1.2
52.1
3.1
1.1
Qwen 235B-A22B-FP3
80.4
19.6
0.0
0.0
18.8
0.8
0.0
Table 12: Detailed performance of SpecGen for pre-condition on VERINA-A.
Soundness
Completeness
Model
Cannot Compile%
Pass %
Fail%
Unknown %
Pass %
Fail%
Unknown
GPT 4o-mini
20.8
78.1
1.1
0.0
65.1
14.2
0.0
GPT 4o
10.8
0.2
4.1
9.8
85.7
0.8
3.8
82.3
4.7
3.2
o3
3.2
93.4
0.9
2.5
87.2
7.6
2.1
o4-mini
4.0
92.1
0.9
3.0
88.1
6.4
1.5
Claude Sonnet
0.0
0.9
90.6
4.7
4.7
Claude Opus 4.0
1.3
92.1
0.9
5.7
90.9
4.7
3.0
Gemini 2.5 Flash
5.5
91.5
0.8
2.3
88.9
5.3
0.4
DeepSeck V3
26.4
71.1
0.8
1.7
67.9
4.0
1.7
Qwen3 235B-A22B-FPS
69.4
30.6
0.0
0.0
29.4
0.0
Table 13: Detailed performance of SpecGen for pre-condition on VERINA-B.
Soundness
Completeness
Model
Cannot Compile%
Pass %
Fail%
Unknown %
Pass %
Fail%
Unknown
4o-mini
66.9
24.4
0.0
GPT 4o
31.6
64.0
3.5
1.0
56.3
11.4
0.7
GPT 4.1 04-mini
43.2
51.1
1.5
4.2
51.9
2.0
3,0
7.2
89.6
0.3
3.0
74.3
16.3
2.2
10.1
2.5
4.9
75.3
2.2
Claude Sonnet
11.4
72.6
4.0
12.1
76.5
2.5
9.6
Claude Opus 4.0
8.6
79,5
2.7
9.1
82.0
1.5
7.9
Gemini 2.5 Flash
26.7
68.2
2.5
2.7
66,9
4.7
1.7
DeepSeck V3
66.4
32.4
0.7
31.4
0.3
Qwen3 235B-A22B-BR
94.8
5.2
0.0
0.0
4.9
0.3
0.0
32

---

Published as a conference paper at ICLR 2026
Table 14: Detailed performance of SpecGen for post-condition.
Soundness
Completeness
Model
Cannot Compile %
Pass %
Fail
Unknown %
Pass %
Fail
Unknown %
4o-mini
68.3
27.1
4.2
0.4
28.2
2.6
0.9
GPT 4o
49.1
41.7
4,6
4.6
41.0
1.8
8.1
GPT 4.1
41.8
49.2
7.2
43.1
14.3
04-mini
22.7
58.5
15.7
55.6
19.0
03
23.1
60.2
1.8
14.9
57.1
2.3
17.5
Claude Sonnet 3.7
30.6
53,9
3.2
12.3
48.2
1.6
19.6
Claude Opus 4.0
27.3
54.9
15.4
50.1
21.3
Gemini 2.5 Flash
40.6
50.4
7.5
47.5
10.9
DeepSeck V3
53.9
39.9
2.6
3.6
37.5
3.6
4.9
Qwen 235B-A22B-FP3
83.0
16.4
0.6
0.0
17.0
0.0
0.0
Table 15: Detailed performance of SpecGien for post-condition on VERINA-A.
Soundness
Completeness
Model
Cannot Compile%
Pass %
Fail%
Unknown %
Pass %
Fail%
Unknown
GPT 4o-mini
51,5
41.9
5.9
0.8
44.9
2.3
1.3
GPT 4o
30.8
4.3
3.6
60.6
7.2
GPT 4.1
27.4
65.9
1.9
4.9
60.4
0.8
11.5
04-mini
16.8
73.8
1.3
8.1
70.0
0.8
12.5
03
14.3
73.4
0.9
11.3
70.6
1.3
13.8
Claude Sonnet
22.6
2.5
64.0
Claude Opus 4.0
19.4
69.4
1.9
65.7
0.2
14.7
Gemini 2.5 Flash
24.0
69.4
1.1
5.5
63.6
0.8
11.7
DeepSeck V3
39.4
56.6
1.5
2.5
54.0
2.5
4.2
Qwen3 235B-A22B-FPS
72.8
26.2
0.9
0.0
27.2
0.0
Table 16: Detailed performance of SpecGen for post-condition on VERINA-B.
Soundness
Completeness
Model
Cannot Compile%
Pass %
Fail%
Unknown %
Pass %
Fail%
Unknown
4o-mini
90.4
0.3
GPT 4o
73.1
16.1
4.9
5.9
15.3
2.2
9.4
GPT 4.1 04-mini
60.7
27.4
1.7
10.1
20.5
0.7
18.0
30.4
38.5
5.4
25.7
36.8
5.2
27.7
34.6
43.0
19.5
39.5
22.5
Claude Sonnet
41.0
35.3
4.2
19.5
27.7
2.2
29.1
Claude Opus 4.0
37.5
35.8
3.2
23.5
29,6
3,0
29.9
Gemini 2.5 Flash
62.5
25.4
2.0
10.1
26.4
1.2
9.9
DeepSeck V3
72.8
18.0
4.0
5.2
16.1
5.9
Qwen3 235B-A22B-BR
96.3
0.3
0.0
3.7
0.0
0.0
33

---

Published as a conference paper at ICLR 2026
Table 17: Proof failure category distribution across models.
Category
Claude Sonnet-3.7
Gemini 2.5 Flash
GPT 4.1
04-mini
Incomplete Proof (unsolved goals)
0.55%
77.55%
38.24%
13.10%
Cheat Code Usage
48.53%
1.20%
20.55%
28.30%
Unknown Identifier
15.38%
5.48%
9.01%
12.32%
Unknown Tactic
3.05%
3.50%
6.04%
17.76%
Tactic Failed
9.92%
3.72%
2.75%
6.55%
Unknown Constant
4. 4.14%
3.94%
5.05%
5.55%
Syntax Error
3.16%
0.77%
9.56%
3.66%
Type Mismatch
2.62%
0.22%
0.99%
1.66%
Unknown Import
0.98%
0.22%
0.22%
1.89%
Other
11.67%
3.40%
7.58%
9.21%
Table 18: Proof failure category distribution on VERINA-A across models.
Category
Claude Sonnet-3.7
Gemini 2.5 Flash
GPT 4.1
04-mini
Incomplete Proof (unsolved goals)
0.98%
70.67%
33.47%
13.91%
Cheat Code Usage
30.47%
0.39%
16.24%
18.55%
Unknown Identifier
18.55%
4.13%
8.51%
12.30%
Unknown Tactic
4.30%
6.10%
7.13%
20.77%
Tactic Failed
14.45%
6.10%
4.55%
8.27%
Unknown Constant
6.84%
5.91%
7.92%
7.46%
Syntax Error
4.69%
1.18%
9.11%
3.23%
Type Mismatch
4.30%
0.20%
1.39%
1.61%
Unknown Import
1.76%
0.20%
0.40%
2.82%
Other
13.67%
5.12%
11.29%
11.09%
Table 19: Proof failure category distribution on VERINA-B across models.
Category
Claude Sonnet-3.7
Gemini 2.5 Flash
GPT 4.1
04-mini
Incomplete Proof (unsolved goals)
0.00%
86.17%
44.20%
12.10%
Cheat Code Usage
71.36%
2.22%
25.93%
40.25%
Unknown Identifier
11.36%
7.16%
9.63%
12.35%
Unknown Tactic
1.48%
0.25%
4.69%
14.07%
Tactic Failed
4.20%
0.74%
0.49%
4.44%
Unknown Constant
0.74%
1.48%
1.48%
3.21%
Syntax Error
1.23%
0.25%
10.12%
4.20%
Type Mismatch
0.49%
0.25%
0.49%
1.73%
Unknown Import
0.00%
0.25%
0.00%
0.74%
Other
9.14%
1.23%
2.96%
6.91%
34

---

Published as a conference paper at ICLR 2026
E CASE STUDIES OF MODEL FAILURES AND EVALUATION METRICS
In this appendix section, we provide a detailed qualitative analysis of common model failure patterns across the three foundational tasks and illustrate how LLMs struggle with different aspects of verifiable code generation through concrete examples. We also discuss how our evaluation metrics flag these failures, highlighting both their effectiveness and limitations.
Code generation failure: hallucinated method usage. Figure 18 demonstrates common LLM failure mode where 04-mini generates code that appears syntactically correct but contains non-existent methods. While the model correctly identifies the XOR-based algorithmic approach and provides accurate comments, it hallucinates the Int xor method that does not exist in Lean 4's standard library. This shows that current LLMs fall short in understanding Lean 4's language features.
dee FandBingl (cuns List InL) (upresoond Pinds wag.llinggellleo LeNuneess pewwordd [naaat)) Int. code
fbenchnark
wlll elecentes dupl Fostes canpe] opt, Teaving the unique mumber
miaaaa foldi (Cun acc x Int.x0x aed x3
(benchmark Bend code
Figure 18: Example (verina_b.l) 04-mini generates code with hallucinated Int. xor method that does not exist in Lean 4.
Code generation failures: unit test rejections. Figure 19 illustrates how subtle logical errors in LLM-generater code can lead to unit test failures. The task requires implementing a function that finds the next greater element for each number in nums within the array nums2, or outputs -1 if there is none. 04-mini generates rextGreaterOns helper function with bug in the state management logic. After finding the target element, the function incorrectly calls aux tl false instead of aux t1 true in Line 22, causing it to lose track of having found the target and fail to identify subsequent greater elements. This results in incorrect outputs for the test case where [3, 1, 4] should return [4, 4, 4].
numsl [1, 2, 31 and nums2
2 Jbanchmark 8start code_aux
/-
Seiper cunction
rext greater elenent of "target" 'numo2
5 No travarse the list, fiest locating "tazost', ', then searching foz the dizat elanen: > "tazost".
If not found, returns -1.
nextGreatarCne (nuns2 let r2c aux 11 List Intl (founsTarget Bool) Int 1"
Itarget : Int) à
10
match 1 with
[]
ba:st.
13
-f FoundTargat shaa
24
if hd mn tareaz then
27
aux falsa
18
wlew
hd target ther.
21
22
aux tl falso
lbwnchauck Send coske_aox
25
def next eeent (nunsl Tist Tot (nuns2 Tist Tot (ippoesnd new.lrowatsseeeeen... (bonchaazk Racazt code
(muas))) (muna231
29
For each elanent in 'ounsl 1 comput lex next greater dn "rume?"
30
nunal .nag Tun => Gowaaewwa nums? (benchnazk Bend cods
Figure Example (verina.b_57). o4-mini generates code with logical error state management, causing unit test failure.
Specification generation failures: unsound pre-conditions. Figure 20 demonstrates how LLMs can generate specifications that are too restrictive, leading to unsound pre-conditions. The task description states "Assuming k < number of distinct elements nums". The ground truth pre-condition correctly uses nums erasebups. hength allow k to equal the number of distinct elements. However,
35

---

Prop Published as a conference paper at ICLR 2026
the LLM- -LM--enerated version uses strict inequality rectly excludes valid cases where k equals the total number distinct elements. This makes the pre-condition unsound it rejects legitimate inputs that should be accepted by the specification. In our test suites, we have positive test case with nuns [5] and k 1. Since the L.M--generaed pre-condition rejects this test case, our evaluation metric determines that it is unsound.
(distinct nums), Length, which incor-
[reducible,, simp!
det topkFrequent precord Inuns
List Intl
(K Nat) 1 Prop
fbencherrk rt: Secart preesnd
www.eeupee.eeeeeoo.. length
Ibenchnack Bend presond
(a) Ground truth prenconiii
Jbcachnark presond_aux
[reducible, simp!
3 def containa (xa List Int] (x Ins) : Bool :-
xs.fold2 (fun 2
on ralse
Breduciill,,
distings (Ka list Int! a List xs. to.1old: (fun acc y "> if conrains ase then aec clse sec +t (y3) II
8 Ibenclaeck Send pavconddaux
@[redacible, _preccnttppccttttt Inuns
topkFsoquant_pccom
Intl (R Not) Pzop
fbmnclateck Sacazs preesed
12
(diatinet nuns) .1eacta
(benchnerk Bond
(b) Unsound pre-condition generated by o4-mini.
Figure 20: Example (verina.b_77 b_)): 04-mini generates unsound pre-condition using strict inequal- ity instead of allowing k to equal the number of distinct elements.
Specification generation failures: incomplete pre-conditions. Figure 21 demonstrates how LLMs can generate overly permissive preconditions that fail capture essential constraints. The task description specifies that "All integers in both arrays are unique" and that "numsl: A list of integers, which is subset of nums2". The ground truth precondition correctly enforces three critical require- ments: List. Nodup numsl ensures uniqueness in the first array, List List.Nodup nums2 ensures uniqueness in the second array, and nums alll (fun x => x E nums2) verifies that nums indeed subset of nums2. However, the LLM-generatie precondition simply uses True, completely ignoring all stated constraints. This makes the precondition incomplete as accepts invalid inputs that violate the problem's 's fundamental assumptions, potentially leading incorrect behavior in the implementation and proof generation phases. In our test suites, we have a negative test case with numsl [1, and nums2 [1, 21. Since the LLM-generated pre-condition accepts this negative test case, our evaluation metric determines that the LLM ---geeerrttedd pre-condition is incomplete.
Ground azo-condition
2. [[reducible, sinpl
3 der nextGreatert: www.rreeeeetttteeeeeeeeeeee pementtpreccodd (numsl List Int) (nums2
List
Int)
fbenchnazk #scart precond
List.Nocup numel List, Nocup nuns2 A aunsl.all (Cun e> x nuno2) (benchnark sond
(a) Ground truth pre-condition.
1. 8 Treducible, sinpl nextGreatert? precord fbenchnerk Éscarc precond
Tee)
Prop
Tsue
(benchmark Bend precond
(b) Incomplete pre-condition generated by o4-mini.
Figure 21: Example (verins everina-advane advanced.557 o4-mini generates incomplete pre-condition using True instead of enforcing uniqueness and subset constraints.
Specification generation failures: unsound post-conditions. Figure 22 illustrates how LLMs
generate post-conditions that miss critical constraints, leading unsound specifications. The
36

---

Published as a conference paper at ICLR 2026
task involves adding two numbers represented digit lists in reverse order. The ground truth post- condition correctly enforces three essential properties: arithmetic correctness, digit validity (all digits should be less than 10), and prohibition of leading zeros except for the special case where the result zero. However, the L.M--generaee post-condition omits the leading zero constraint entirely, only checking that the result non-empty and digits valid. This unsound specification would accept invalid outputs like [2, 1, 0J representing 012 (which is one of the negative test case), which should be rejected favor of [2, 11. The missing constraint demonstrates how LLMs may capture the primary functional requirements while overlooking more subtle constraints that are crucial for correctness.
Joenchmerk poeroond
2 det listTeleet : List Net + Nat
3 11 >>
d
10 listTcMat
Joenchnark postesnd_asx
Bireducible, simpl
7. der sddiwesiumoorsrrooteeea (11 List Natt (12 : List Nat! (result: List Nett (buprocond (121)
addtwokumbers__p
(benchnark 5 ListTaNat resuit 1is:toNst 11 ListIoliat 12
post
10 (V d E resplt, d 10) A
No leading traselz.gotiass & (i1 [0] (0] sesuit (011)
lbenehmark #end posccona
(a) Ground truth post-condition.
genorated post-condition 2 Jbenchmack Sotast poncoond
3 @treducible, sinpl listTeMatRev List
6 d da => 10 listToMatRev da Townchmark gend postesnd mux [reducibble, singl
Hatt (12 Llat Mat! (cesult: List Nat! (Lprrccond ca addrsolumbers, prepand (11) (121) Prop :- (bencloteck Bacazs cand
10
13 V d, d E sesult -a ToacToNat 13 Toat Rev 12.
lis: rexult
(benchnark Bend pastoond
(b) Unsound post-condition generated by 04-mini.
Figure 22: Example (verina b.5): 04-mini generates unsound postcondition that fails rule out leading zeros in the result.
Specification generation failures: unsound and incomplete post-conditions. Figure 23 demon- strates how LLMs can generate post-conditions that are both unsound and incomplete by failing to handle edge cases properly. The task involves finding the smallest single-digit prime factor of a natural number. The ground truth post-condition correctly handles all cases including the edge case where n 0, specifying that the result should be 0 when the input 0 or when no single-digit prime divides n. However, the -LM-generated post-condition fails to consider n 0 entirely. When O, the condition & P * is false for any prime p (since 0 p 0), making the first disjunct impossible to satisfy. This renders the specification both unsound (accepts incorrect outputs) and incomplete (rejects valid cases where n = 0). The missing edge case handling demonstrates how LLMs may overlook comner cases that are crucial for specification completeness (and soundness). We have a positive test case where n = 0 and result = and a corresponding negative test case where and result 2 that capture this edge case. The LLM -eMeeeerrteed post-condition rejects the positive test case and accepts the negative test case, therefore our evaluation metric determines that this generated post-condition is both unsound and incomplete.
37

---

Published as a conference paper at ICLR 2026
Ground tzuth cost-conditin (redaaci,,,, simp]
2.
(seaull: (Bprewoodd singlaDigilPa
n)1 Frop 2-
féenchawwk Sscare post cond
result C [D, 2, 3, 5,
(rasul:
* 0)1)
treauls
seault A (aist reauls) .all (fun x 12, 3, 70
cu of 0))
B
(benchnazk Bend posscond
(a) Ground truth post-condition.
1. LM generated post condition Jon Joenchnark 9start postoona_aud B[reducible, sinpl
det 1ssinglofigtterim (p Nat) : Prop
p 5 V 5 p
loenchnark gend postesnd mux
[reesacibl,, simp!
det singled wtt..eeelltttteimefcoorooo postcons in Natl tresuit: Nat) precend stoggeeeiggttrrineecctoo..ooo + n)1 Prop =*
ibenchmark #start postzond
no smail prime
(rapci: A P, isSingiaDioitPrrmm + a $
12
V
result
ansllese ssingledigge prime divisor a, LssingisDigitPrime q rosuls
(LsSinglaDigitPrrin reseit
15
tbenehmark #end posccona
(b) Unsound incomplete post-condition generated by o4-mini.
Figure 23: Example (veri na b_72): 04-mini generates unsound and incomplete post-condition that fails handle the edge case n 0.
Untestable post-conditions. Figure 24 demonstrates the limitations of our testing-based evaluation framework when encountering specifications with quantifiers nite domains. The LLM-generated post-condition for finding the length of the longest increasing subsequence contains a universal quantifier lists, making it impossible to evaluate even with plausible testing. Our evaluation framework returns unknown for such cases, as neither decidable testing nor plausible exploration can adequately handle the unbounded quantification. This example highlights fundamental challenge in automatically evaluating I.I. L.L.-egnerteed formal specifications: while our framework successfully handles most practical cases, very complicated specifications require more comprehensive approaches such as automated theorem provers or LLM-based proof generation, which we leave to future work.
complicated structures infi-
List Int that ranges over all possible integer
Tnenchmark Setart postoond @[reducible, singl det IsSubeequence Liet Int + List Int + Prop
5 II, => True
False
x5, 2 11 TaSuhaeguercee x8 ya
y then
clse IsSuscequence (x 11 X5) ye
@[reducible, det strictiyyncreassing List Int + Frop
12
I1l =5 True
13
[_] = True
ixily
x < striotlytnnceassing (y
loenchnark Send postesnd aux
16 @[resacible, simpl
lengshöttis
Inans
Intl (result: Nat) it_preoend lengthött ppreecnd inuns)
Prop
postrond
(V 5 : List Iat, Issunsequenes s nune A sessetiyIncreasiig s List.icngth S sesult)
20
A 3 List Irt, IeSubsequence nuns A strieriyttnceeasing lenghh result
(benchnark Bend poscoond
Figure 24: Example (verina 25): 04-mini generates post-condition with quantifiers over lists that cannot be evaluated by plausible testing.
26 27 and 28 demonstrate 04- mini's 's iterative proof refinement process over 24 attempts. The task involves proving that list element removal function satisfies its specification. Initially, o4-mini generates overly simple proof using direct induction and simp tactics, which fails to handle the complex logical structure. After 23 failed refinement attempts, the model makes useful improvements based on Lean compiler error
Proof generation success with iterative refinement. Figure 25
38

---

Published as a conference paper at ICLR 2026
messages, significantly restructuring its approach. The error message from the 23rd attempt reveals that the proof got stuck on conditional expressions involving decide predicates. In the successful 24th attempt, 04-mini improves its strategy by introducing auxiliary lemma helper_ eq ______________ filter and using explicit case analysis with by_cases. This demonstrates how iterative refinement with compiler feedback guide I.LMs toward substantially different and correct proof strategies, though it may require many attempts to achieve success.
(reducible,, simpl zonoveELcment_ ibonehmerk #scart procond
Hatl (carget Nat)
Pzop
Tzue (benchnark Bond precond
7 def zomoveElenent (2st List Hatl Icargat Nat) phepoooddd senovaElenat__prece (1ac) (tazget])
+ List Nat
(benclitwzk
helper (1st List Mat) (tazqet Hat)
10
motch 1st with
11
[] []
reat helger xa targes
12 x LazgeL then zeat elae x ceaL
15
helper 1st target Send
18
0[reducible;
19
def senove ementt postcood (1ass List Hat) Baaret Natl (results List Nall (Cprecond anoveEleneni lenentt_peocreed
(tazget)):
(benchnark Becare postrond
result fat.filies [Ean. (benchaack Band posccond
Lerget)
22
25
lbenchmark Sstazt proof_aux loenchmark gend proof_
theoren removeElenent spea_sstietied Clets List Nat) (target: : Nat) (hupreoond 1 renoveE, perooveeleeent_ppeeene
<a 1st] Itarge:)::
29
AemoveBlement poweeeeeen__ppsooo (1a.). [LazgeL) (rmnoveBlemen. (191) [Largel) luprecsad) hupswoond =- by
proot
Unfold the definit/oms reaoveßllneet and poetooodd
32
daiag [sumuvelleenll, senove Leveelleeenn..oo pusteedd)
Proceed Induction ion the Hast induct 1st
nill
36
-- Raze case: bothn sides reduce to I7
cone xe
Enduct five sceps use the Induct fan hes simp [ih]
40
lbenchmazk Bend prost
Figure 25: Example (verina.b.64). 04-mini generates incorrect proof on the first attempt.
2. Jocnehmark Istart prost_sux
Liary Lentar the 'heiper" tunction produces exactily "ist.riiter (tun x carget) theoren helper filter f1at rist Nat) (target Nat) consess..cen..noipee
target ist.fiiter (tun target) 1"
5 induction 1st with
nil m>
samp ItenoveSlenent nelpe
cons x xs ih -- unroid one step of 'heiper' IremoveElenert .helpet
10
solve simp [1h]
statenent using induction hypothosis
14
Jbaachmazk fend proof_aux
cheozen removeElement stwpentt_poo_ttttiiieeee (let: Nat) (target: Mat) (hupresond zenoveElement pprpoontpppoootttto
+ 1st) itarge:)):
17
Sanoce_cccnt__ootte.. (1sz) (target) (ponoveBlencat (1s2) itarget) h_proccad) h_precand :" by
#start
posteondition
dofinition of 'sanovoElonant"
20
drinp
rsmoveElement) the preconaition is trivia!
21
h_orecond
apply suxiliary Joma to csnciude
24
exact helper_eq_fittee lsz Lazqet
25
(benchnark Rond proo?
Figure 26: Example (verina o4-mini generates incorrect proof after 23 refinement attempts.
39

---

Published as a conference paper at ICLR 2026
1. unsolved goals
2. case cons
x5 List Mat
1h : renovefleeentt.in helper xe Larget Cilte (fun x wo dedide (8 o target))
+ (ir Larget then filtez (Dun x w> Idecide Go tazget)] xs List. (Fun Idecida (x target)) xs)
filt. Mun Idecide Larget))
Figure 27: Example (.erinabb_64) Lean 4 compiler error message from the 23rd refinement attempt.
Jocnchnark
Auxiliazy 'heipez" function produces exactly filcer (Cun & cargety 3 theozen halper_sg__ttee list List Net) (tarqet Hat) wonoves-cment. heiper 1st target ist.filter (tun x >> x # target) 17 by induction
nil
both sidea seduce to YJ sinp [renoveBlenent [hllpe]]
ane siep "halpes"
dainp [xenoveElenart, halpar]
12
-- caze-spiit on whother 'x target"
by_owsee casa pas =>
Lazget
15
If 'x = starget", "helper' drops 16, and WFllter' also drope de simp [h, ib]
18
of target', ' thelper" keepy it, Milleer' aleo keeps siop Th, ihl Jbanchmark Send proof_aux
20
Lhwosen Seatoeeleenttsspeeaette
(latz Liat Nat) (Lazget: Mat) @eprewood semoveElement pewceeeeeeetttpcooddd
c lsti itarge:)):
23
amoveelleeen poxtccodd (1as). [target) (rmnoveBlament (193) Itareet) huprecend) hupswoord =- by
(benchaazk Sacaze
unfold the postcondition
the definition of 'removeRienent"
dsinp [senove www.eelleennntaaleeeeoe posteendd, tenoveElenent))
the precondition in trivial precond
apply auxillary Jeans to conplade
30 exact bpere__ si_itteer 1st target
(benchmerk Bend prosf
Figure 28: Example (verina.b.6)); 04-mini generates correct proof on the 24th attempt.
F USE OF LLM
LLMs used in limited and supervised manner during the construction of VERINA. Specifically, we employed OpenAl o3-mini with few-shot prompting to assist in translating 59 Dafny instances from CloverBench (Sun et al. 2024) into Lean, as discussed in Section 3.2 All such translations were subsequently inspected. corrected, and verified by the authors to ensure accuracy. In addition, LLMs were used as assistive tools for editing and polishing the presentation of the paper. LLMSs were not involved in research ideation, discovery of related work, experimental design, dataset selection, or analysis.
40


Reviews:
[{"review_id": "RoUiS6l85X", "reviewer": "Reviewer_sru5", "summary": "The paper introduces VERINA, a benchmark for verifiable code generation built on the Lean 4 proof assistant. VERINA provides a framework for evaluating three core tasks: code generation (CodeGen), formal specification generation (SpecGen), and proof generation (ProofGen), as well as their modular compositions. The benchmark consists of 189 manually curated problems, each with a natural-language description, reference implementation, formal specification, and a test suite. The authors propose a hybrid evaluation pipeline for SpecGen that combines automated proof attempts with property-based testing. An extensive evaluation of general-purpose LLMs and specialized theorem provers reveals that ProofGen is the primary bottleneck in the end-to-end pipeline, with even iterative refinement showing limited success.", "questions": "Please see the Weaknesses above. In addition:\n\n1. The paper states the SpecGen evaluator compares outcomes for R and ~R to select the \"more accurate\" result. Can you please clarify what \"more accurate\" means here?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper makes a well-scoped and valuable contribution by introducing a unified benchmark for code, specification, and proof generation.\n2. The proposed hybrid evaluation metric for SpecGen is a practical approach to a difficult problem, leveraging both formal proving and extensive testing to assess specification quality.\n3. The paper provides a thorough experimental evaluation across a diverse set of models, including general-purpose LLMs, specialized provers, and an agentic baseline, offering a snapshot of current capabilities.", "weaknesses": "1. Gaps in Evaluation Metrics: The SpecGen evaluation logic is potentially not perfect. It uses negative test cases to evaluate both the soundness and completeness of different specification components, but does not appear to distinguish between tests that violate the pre-condition versus those that violate the post-condition. This can lead to noisy or incorrect soundness/completeness judgments.\n2. The paper's claims of test suite adequacy rely on coverage metrics from Python reference implementations, as Lean lacks mature coverage tooling. This is a significant gap, as coverage in one language does not guarantee equivalent coverage in a translated Lean implementation. Can authors discuss more about it?\n3. The evaluation lacks a publicly available frontier reasoning model (e.g., OpenAI's o3 or GPT-5, Google's gemini-2.5-pro or Claude-4/4.1 opus), while it currently claims o4-mini to be a frontier model. While the conclusions may not be wildly different, given this is a benchmark paper, evaluating frontier models remains essential.\n4. While LLM-based provers are part of the SpecGen evaluator, their extremely low success rates (<4%) and inherent non-determinism make their inclusion in the primary metric questionable. Some manual evaluations may either strengthen the claim or clarify the limitations.\n\n\n**Other Minor Weaknesses**\n1. Despite the emphasis on compositionality, the paper omits results for the full end-to-end task (Description → Code + Spec + Proof). Reporting this, along with the success rate for generated code satisfying generated specifications, wuld further strengthen the paper.\n2. Insufficient Failure Analysis: The qualitative case studies of failure modes are useful, but the paper would be significantly strengthened by a quantitative breakdown of error categories (e.g., syntax errors, API hallucinations, tactic misuse, logical errors). Correlating these failure types with the dataset origin (VERINA-A vs. VERINA-B) would provide more actionable insights for future research.\n3. Limited Scope and Generalizability: The benchmark's reliance on Lean and the Interactive Theorem Proving (ITP) paradigm necessarily limits the external validity of its findings to other verification ecosystems. Furthermore, the paper could characterize the classes of real-world programs (e.g., those with effects, IO, or concurrency) that are out of scope for the current benchmark design.\n4. Presentation and Framing Improvements: a.) > \"The high-quality samples and robust metrics\"  in the introduction. Robust metrics have not been defined yet, or clarified why exactly they are robust. b.) \"top-performing general-purpose LLM\" is referred to o4-mini. c.) \"ITPs support constructing proofs with explicit intermediate steps. This visibility enables LLMs to diagnose errors, learn from unsuccessful steps, and iteratively refine their proofs,\" in related work. However, iterative refinement is also possible with ATP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KFocKYbhDX", "reviewer": "Reviewer_6DqP", "summary": "This paper introduces VERINA (Verifiable Code Generation Arena), a new benchmark designed to evaluate the ability of Large Language Models (LLMs) to perform verifiable code generation. The authors define this as the joint task of generating code, formal specifications, and proofs that the code aligns with the specifications. The benchmark consists of 189 manually curated programming tasks in the Lean language, each with detailed descriptions, reference implementations, formal specifications, and comprehensive test suites. The authors evaluated state-of-the-art LLMs on VERINA and found that these models face significant challenges, especially in proof generation. The paper concludes that VERINA provides a rigorous framework for measuring and advancing LLM capabilities in producing formally verified software.", "questions": "1). How sensitive are “R might hold” outcomes to the property-based testing budget and random seeds? Please report the split of “proved vs. tested vs. unknown,” and provide seed-averaged results.\n\n2). Beyond Python, did you attempt Lean-level mutation testing or adversarial test synthesis to ensure that the Lean implementations are exercised comparably to the Python references?\n\n3). Can you provide n-gram or signature overlap checks between VERINA items and public code corpora likely used for pretraining? A filtered subset with low overlap would help calibrate headline metrics.\n\n4). What is the marginal utility curve of iterative refinement (cost vs. accuracy) and how does Copra’s benefit scale across difficulty strata? A budget-normalized comparison would help practitioners.\n\n5). Were prompts fully identical across models for each task? If not, please quantify the effect of template differences on pass@1 and pass@k.\n\n6). Could you report a control where the code is summarized into behavior-only comments or contracts before SpecGen to test whether verbosity or implementation detail confounds the result?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1). VERINA covers CodeGen, SpecGen, and ProofGen and allows their compositions, which matches realistic verification workflows rather than isolated tasks.\n\n2). The formal treatment of soundness and completeness for pre- and post-conditions is crisp. The two-stage evaluator (prover first, then property-based testing) gives a practical way to score imperfectly decidable relationships and communicates uncertainty via “R might hold” and error bars.", "weaknesses": "1). Modest scale and representativeness are not convincing. With 189 single-file tasks, the benchmark remains small for broad performance claims. The focus on stand-alone snippets under-represents the multi-module, dependency-heavy settings that matter for real systems. The paper acknowledges this limitation but it still constrains the practical impact.\n\n2). The “100% coverage” is established on Python references rather than the Lean artifacts under evaluation, due to missing Lean tooling. This creates a fidelity gap between what is instrumented and what is graded. Mutation analysis or Lean-side coverage proxies would reduce this gap.\n\n3). Sources include MBPP, LeetCode, and LiveCodeBench. The paper notes risk but does not present a systematic decontamination audit or overlap analysis against model training corpora. This could inflate performance on familiar patterns. Besides, these sources from popular platforms are almost certainly present in the training corpora of the state-of-the-art LLMs evaluated in the study.\n\n4). This contamination risk in  3) may directly undermine the evaluation of the \"CodeGen\" task. The paper reports that the best model, 04-mini, achieves a 61.4% code correctness rate. This relatively high score—especially when contrasted with the dismal 3.6% proof success rate—is ambiguous. It may not reflect the model's genuine ability to generate correct Lean code from a description, but rather its ability to recall a memorized algorithm and \"translate\" it into the required Lean syntax.\n\n5). The central premise of VERINA is to provide a \"holistic evaluation framework\" for the joint task of generating code, specifications, and proofs. However, if the CodeGen task is compromised by memorization, the benchmark is no longer evaluating a \"generation\" pipeline. Instead, it may be evaluating a disjointed task: recalling code, and then attempting to generate specifications and proofs for that recalled code. This is a fundamentally different and less challenging task than true end-to-end generation from a novel problem description, which is what the paper claims to benchmark. Also, the contamination risk makes the paper's main findings difficult to interpret. The key result:  ProofGen is the primary bottleneck, is called into question. Is proof generation difficult in general, or is it difficult when the associated code is recalled from memory? The latter is a much weaker claim and provides less insight. The benchmark fails to isolate whether the models are failing at reasoning or simply failing to formally verify a solution they do not understand but have memorized.\n\n6). The evaluator may return “R might hold” when proofs are inconclusive and tests pass. This is reasonable, yet it weakens the claim of verifiable benchmarking; readers would benefit from a quantitative breakdown of proof-decided vs. test-backed cases and sensitivity to the testing budget.\n\nIn general, the proposed SpecGen metric and the compositional framing are valuable. However, the dataset scale is modest, coverage is measured on proxies, “R might hold” weakens formal guarantees without a thorough sensitivity analysis, and contamination controls are limited. The high probability of data contamination is critical. It strikes at the construct validity of the benchmark, making it unclear what is actually being measured. For a paper whose primary contribution is a new evaluation benchmark, this ambiguity regarding what it validly measures renders its conclusions unreliable and its utility to the community questionable. Addressing these issues may lift the work to a acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Un0Re2smcs", "reviewer": "Reviewer_a4Ez", "summary": "This paper introduces VERINA (Verifiable Code Generation Arena), a benchmark comprising 189 manually curated programming tasks in Lean for evaluating end-to-end verifiable code generation. The benchmark assesses three foundational tasks—code generation (CodeGen), specification generation (SpecGen), and proof generation (ProofGen)—along with their flexible combinations. The best model (o4-mini) achieves 61.4% code correctness, 51.0% specification soundness/completeness, and only 3.6% proof success.", "questions": "- What evidence demonstrates Lean is used for production code verification (not mathematics)? How do you justify that insights from ITP-based verification in Lean generalize to ATP-based systems (Dafny, Verus, Frama-C) that dominate production use?\n- Why not include at least one ATP-based system for comparison to validate generalizability?\n- Can you provide systematic breakdown of proof failure types (wrong strategy, missing lemmas, syntax errors, etc.) with percentages?\n- How do LLM-generated proof structures differ from the 46 human-written ground truth proofs? Are the remaining tasks really provable in Lean?\n- What is the actual end-to-end success rate (NL → verified code) for each model?\n- What are the average branches of the task?\n\n\n\nComments:\n- Incorrect citations: some paper references are incorrect, like SAFE, AutoVerus, etc.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear Presentation: The presentation is well-executed.\n- Extensive Task Support: VERINA comprehensively supports all three core tasks: CodeGen, SpecGen, and ProofGen, utilizing a modular and compositional approach.\n- Thorough Experimental Evaluation: The evaluation is extensive, involving 8 general-purpose and 3 specialized LLMs. It includes detailed ablations on the impact of contextual information, iterative refinement, and a breakdown of difficulty levels.", "weaknesses": "- Limited Language/Tool Coverage: The reliance on Lean significantly restricts the generalizability of the findings. The paper fails to justify how Lean-specific results would translate to other widely used verification ecosystems like Dafny, Verus, or Frama-C, which often feature different proof automation capabilities and specification styles.\n- Unclear Justification for Lean in Code Verification: Lean is primarily known for mathematical verification rather than production code verification. The paper does not adequately explain its choice of Lean for this context.\n- Insufficient Analysis of Proof Generation Failures: Despite proof generation being a major hurdle (with a mere 3.6% success rate, the lowest among all tasks), the paper offers minimal systematic analysis of the underlying causes of failure or actionable insights.\n- Task Simplicity: The tasks appear to be overly simplistic. The paper should provide information on the average number of branches per task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)