VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code
Authors: Lingfei Zeng
Fengdi Che
Xuhan Huang
Fei Ye
Xu Xu
Binhang Yuan
Jie Fu
Venue: ICLR 2026 Poster (2026)
Keywords: Formal verification
verifiable coding agents
code generation
large language models
benchmark
Primary area: datasets and benchmarks

Abstract: Formal verification is the next frontier for ensuring the correctness of code generated by Large Language Models (LLMs). 
While methods that co-generate code and formal specifications in formal languages, like Dafny, can, in principle, prove alignment with user intent, progress is bottlenecked by specification quality evaluation. 
Current benchmarks rely on matching against ground-truth specifications, a manual and expertise-intensive process that has limited existing datasets to a few hundred simple problems and also suffers from a reliability issue.
To address this, we introduce VeriEquivBench, a new benchmark with $2,389$ complex algorithmic problems that probe the limitations of current models in both code generation and formal reasoning. 
Our evaluation framework replaces ground-truth matching with a formally grounded metric, the equivalence score, and rigorously verifies the quality of generated specifications and code.
Our results show that generating formally verifiable code remains a profound challenge for state-of-the-art LLMs. This underscores both the difficulty of the task and the need for benchmarks like VeriEquivBench to drive progress toward scalable and reliable coding agents.

Published as a conference paper at ICLR 2026
VERIEQUIVBENCC:: AN EQUIVALENCE SCORE FOR GROUND-TRUTH-ERR EVALUATION OF FORMALLY
VERIFIABLE CODE
Lingfei Zeng'; ; Fengdi Che²; Xuhan Huang,, Fei Ye', Xu Xu',
Binhang Yuan Sy Jie Fu
'Huazhong University of Science and Technology, "University of Alberta
TThe Chinese University of Hong Kong, Shenzhen, *Jilin University
HHong Kong University of Science and Technology, "Shanghai Artificial Intelligence Laboratory
ABSTRACT
Formal verification the next frontier for ensuring the correctness of code generated by Large Language Models (LLMs). While methods that co- generate code and formal specifications in formal languages, like Dafny, can, in principle, prove alignment with user intent, progress is bottlenecked by specification quality evaluation. Current benchmarks rely on matching against ground-truth specifications, mamual and cess that has limited existing datasets to a few hundred simple problems and also suffers from reliability issue. To address this, we introduce Ver- "RquivBenche a new benchmark with 2, 389 complex algorithmic problems that probe the limitations of current models in both code generation and formal reasoning. Our evaluation framework replaces ground-trun ttrth match- ing with formally grounded metric, the equivalence score, and rigorously verifies the quality of generated specifications and code. Our results show that generating formally verifiable code remains a profound challenge for stale-of-Che-art LLMs. This underscores both the difficulty of the lask and the need for benchmarks like VeriEquivBench ward scalable and reliable coding agents. The datasets are available at https:/ https//ggtttub.oom com. htn////unyooooo BaouuyGGooo//Viiiiiiiieeeee VeriEquivBBnch.
pro-
drive progress to-
INTRODUCTION
Large language models (LLMs) already possess substantial capacity for following natural- language instructions and executing a wide range of coding tasks (Li et al., 2022a; Jain et al., 2024; Zhao et al., 2025). At the same time, the correctness of the generated code remains concern (Cotroneo et al., 2024; Wang et al., 2025b), where functional errors cost users extra effort to debug and also pose significant risks in the safety-critical domain (Dalrymple et al., 2024). A common solution is to evaluate generated code through unit tests (Jimenez et al., 2024; Wang et al., 2025c). However, this process offers no provable guarantee of correct as insufficient unit test coverage can fail to detect critical errors (Yu et al., 2025). On the contrary, a verifiable system resolves the issue by co-generating formal specifications and code to formally verify the alignment with the natural language query intention (Sun et al., 2024). Our work focuses on building an end-to-end agent for formal verification, for which adopt Dafny (Leino, 2010). It is an ideal choice as Dafny's automatic theorem prover (De Moura & Bjørner, 2008) climinates the need for manual proof writing. Furthermore, its similarity to common languages like Python and C simplifies code transformation.
While several benchmarks (Ye et al., 2025b; Thakur et al., 2025) target at building reliable reasoning system by formally ensuring the exact code generation (György et al., 2025), their progress is constrained by the reliance on hanually-written ground-truti truth specifications for
"Equal contribution.
I *Crresponniig author.

---

Published as a conference paper at ICLR 2026
Query
Code Spec
Dafny Verifier
lec Airarpancce th amaaytinnyy tays int eecuree pias
Binary scarch:
3-c succeeeee was kay 2ee scteege ciiee
Given a sorted arry and target. return the index if found -I.
teen
Hardl-Codal
Dafny Verifier
pacezd sbece agec ScoeEgs b. ACERyTAZEN, lag LOG ERTuOra 1880) SEST
Baffy verified
Equiv. Code & Spee
sidtrectional Proofs
cleett wnd alsl al211 mnkays
the saace sasaggenedan Aight
Statement that Spee - Code
Generated Spee
Natural Language Query
The method searches for Aey in sorted, non-null army a. key found, returns negutive one guarantees the not the army.. it finds the key.
receires a In soll
aoaee alk) key
cenppre oe tae fenech any (inag
it retumss the index of an element equal to the key.
validate The Solation
Figure I: An end-to-end verifiable coding agent first generates both the implementation and its formal specification, and uses the Dafny verifier to prove their mutual equivalence. It then interacts with the user to validate that the formalized intent matches the desired behaviour. The specification-to
implication can be verified directly by submitting the generated Dafny program to the verifier. To establish the reverse implication, verification script is constructed from the generated Dafny program to prove that the code does not exhibit behaviours beyond those described by the specification. Together, these two directions of implication establish mutual equivalence, ensuring both the soundness and completeness of the solution.Thh verified formal specification subsequently translated back into natural language, enabling the user to confirm that it faithfully captures their original intent.
and requires deep expertise (Misu et al., 2024), which sets a barrier to broadly applying formal verification and scaling benchmarks in both size and complexity. As result, prominent Dafny benchmarks, including Synthess (Misu al., 2024) and CloverBench (Sun et al., 2024), contain only 215 simple examples combined, insufficient for evaluating current LLMs' advanced reasoning abilities. Moreover, the reliance on expert annotation is not only a scaling bottleneck; it also leads to a reliability issue. An analysis (Sun et al., 2024) has figured that 10% of expert-written specifications in DafnySyuthesis are wrongly claimed as ground-trutha. and our own review finds another 18%, containing errors ambiguities. Such flaws undermine the validity of any benchmark that depends on a ground-truth solution. Therefore, our pipeline requires models to perform automatic formalization, rather than assuning that specifications are given solely for evaluating code quality. This raises a critical question: How can we reliably evaluate specifications' quality without depending on the ground- truth? To answer this question, we make the following concrete technical contributions:
evaluation. This formal annotation process is incredibly
Contribution 1. We propose a novel formally- grounded metric, named the equivalence score, that measures the quality of generate specifications as well as the code by verify- ing their mutual equivalence. The score confirms whether a specification unambiguously describes the code's 's behaviour by using the Dafny verifier to check for bidirectional impli- cation. This automated process has no false positives, ensuring that only correctly matched code-specification pairs are accepted. In order to validate the alignment with the query in- tention, we further include a sccond evaluation step: translating formal specifications back to natural language, tions with users to fill in ambiguity and clarify their intentions. The complete pipeline is illustrated in Figure 1.
used by Ying et al. and Sun et al.. This allows further interac-
2

---

Published as a conference paper at ICLR 2026
Contribution 2. Equipped with our automated evaluation metric, we introduce VeriEquiv- Bench, a benchmark of 2, 389 examples with natural language problem descriptions, code and specifications, and additionally, 678 synthetic algorithmic problems. VeriEquiv- Bench significantly expands on prior work in both dataset size and problem complexity, leap demonstrated by the average Cyclomatic Complexity score, which rises from 2.44 in DafnySyuthesis to 5.63. Our benchmark supports evaluating verifiable coding ability given user query, as well as auxiliary tasks introduced in prior work, such as specification gen- eration from code (Yan et al., 2025) and specification refinement (Loughridge et al., 2025). The core of our dataset converted from the LeetCode corpus (Xia et al., 2025), a large and yatnuunnnty--llddaaeee collection of algorithmmic problems well-suited for evaluating a model's reasoning abilities. To supplement this data, we also introduce a synthesis pipeline that uses a structured tagging system to generate novel queries by randomly combining lags for different domains, data structures, and algorithms, introduced in Section 2.3. This provides scalable method for creating large training datasets of new problem descriptions that are fully compatible with our automated evaluation signal.
Contribution 3. We conduct concrete evaluation of state-of-the-art LLMs, where Ver- iEquivBench Bench serves as a testbed for these models to explore and extend the reasoning abilities on complex problems, beyond human-annofaled data (Silver el. al., 2021; Ye et al., 2025a; Shojaee et al., 2025). Instead of requiring users to validate the intention satisfaction, we use Claude-4 as a judge (Wang et al., 2025a). Our evaluation highlights the profound difficulty of this task and the effectiveness of our benchmark. The best-performing model, Claude-4- sonnet, which solves 75.81% of the problems on CloverBench, fails on all instances in our datasct, even under the pass04 evaluation metric. This highlights the limited capacity of current models to carry out reliable formal reasoning on complex algorithmic tasks.
2 BENCHMARK OVERVIEW AND CONSTRUCTION PIPELINE
In this section, we first present aggregate data statistics for VeriEquiivBench. Subsequently, we introduce the two curated subsets released with the benchmark: (i) the LeetCode- transformed datasct, and (ii) tag-composition datasct, called TagComp, the latter being explicitly constructed to evaluate verifiable agents on novel data without contamination (Tu et al., 2024; Riddell et al., 2024).
Each problem in our benchmark provides comprehensive set of artifacts: natural language query, implementations in both Python and Dafny, unit tests and two versions of formal specifications: a strong auto-formalized baseline explained in Section 2.1 and a weaker, verifiable but incomplete version explained in Section 2.2. The strong version the specification is derived from the natural-languaae query without omitting any information, so that Claude-4 can generate the correct implementation solely from the specification, without access to the original query. However, this fully detailed version does not pass the verifier. Therefore, we introduce an incomplete but verifiable alternative, referred to as the weak baseline. Additionally, each problem is annotated with metadata, including its difficulty level and descriptive tags for the relevant algorithm, data structure, and domain. Unlike LeetCode, our benchmark uses a more detailed and structured set of tags to categorize problems. This new tagging system is described in Section 2.3 for future query synthesis.
Starting from the original Leetcode split of 2, 641 training and 228 test instances, we first curate 2, 174 cases successfully transformed to Dafny. Then we compose new problems by merging tags, producing 1, 893 additional items; the full tag-composition procedure is described in Section 2.3. For new problems, we ask Claude-4-sonnet to generate pairs of Python code and corresponding unit tests. For only 300 of new problems, Claude-generated code passes at least 85% of their corresponding unit tests, forming the cleaned TagComp pipeline, giving us 2,389 problems
dataset. Of these, 215 samples clear the in total that pair natural-language queries with formally annotated code.
Table 1 presents key metrics for our annotated Dafny code, which uses the weaker, verifiably correct specifications. Our problems are significantly more complex than those in Clover- Bench, often involving multiple methods rather than a single one. Furthermore, the corre- sponding specifications, while incomplete, contain substantial number of formal clauses.
3

---

Published as a conference paper at ICLR 2026
Table 1: The table overviews several attributes of our annotated code.
Dataset.
Metric
function
method
invariant
ensures
decreases
LeetCode
mean
0.78
1.33
5.12
1.71
0.46
TagComp
mean
0.96
3.18
7.34
3.14
0.70
2.1 LEETCODE AUTOFORMALIZATION
Past formal-language sets such as DafnyBench (Loughridge et al., 2025) are still small and narrow, because hand-written specifications are too costly scale (Misu al., 2024). To oblain large, varied training data without extra human cost, mine the classic Leetcode pool, convert problems to formal specifications, stated in Figure 2 Pipeline 1, while keeping query and specification aligned with two short tightening evaluation protocols (Sun et al., 2024), shown in Figure Pipeline 2.
Specification Generation We feed the problem description to Claude-4-sonnet to obtain an initial Dafny specification, yet even the initial drafts often contain syntax errors. Thus, revise and resubmit up to ten times until the file has parse resolution errors. We find that supplying two simple examples exploits the model's in-context Icarning (Dong et al., 2023) and sharply lowers the error rate (prompt template in Appendix C).
Furthermore, we constrain the model to generate specifications using only first-order logic, prohibiting recursive or dynamic programming-styke definitions. This ensures the specifica- tion describes the problem's declarative properties without leaking the implementation.
Equivalence to NL The equivalence check follows the protocol proposed by Clover (2024)
and contains two steps: (1) A model (we use Grok4 here) rewrites the description so that it cleanly mirrors the specification, then another model (Claudo-4) judges the equivalence between the original description and the rewritten one, yiclding score; (2) The specification alone is translated into Python and executed against the ground-truth Leet Code unit tests. The unit test passing rate is reported in Appendix G.I.
Pipline
Pipline
natural language query
New description
Ask jugde +
Prospt
available dafny_spes
pythos code
refine-10
SSver ass-aagetion teights bara with nith of canguta fow with mater reirdng.
Pipline
Data
Quaat-ona
dafny_spac
rollowtx2
Heliable dafny,
probtem description
Generate Complete Code
Insver
refinexd
Secheight
Ground Truthloyl
Figure 2: The figure outlines our autoformalization and code generation workflow: Pipeline produces comprchensive and syntax-free specifications; Pipeline 2 checks consistency be- tween the NL query and the specifications; Pipeline 3 emits fully annotated code that passes the verifier.

---

Published as a conference paper at ICLR 2026
2.2 LEETCODE VERIFIABLE CODE GENERATION
Owing to the limited performance of state-of-the-art LLMs on challenging formal-language coding, we adopt the multi-stage pipeline (pipeline 3 of Figure 2): prompled by the previ- ously generated specifications, the problem description, and a reference Python solution, the stronger model (Claude-4) produces annotated Dafuy code, while a lighter model (Claude- 3.5) then polishes this output, iterating up to six times to eliminate any syntax errors.
In practice, the vast majority of problems converge within three refinement rounds, while residual subset still fails to yield a well-formed artifact even after the sixth attempt, with the success transformation rate reported in Table 2, and data statistics reported in Table 1.
The table shows the number of examples without syntax errors in autoformalization and verifiable examples in code generation.
Table 2:
Spec Autoformalization
Dafny Code Genetation
Dataset
Number
Rate (%)
Number
Rate (%)
LeetCode
2584 296
90.1 98.7
2174 215
75.8 71.7
TagComp
2.3 DATA SYNTHESIS THROUGH TAG COMPOSITION
We propose constructing a fine-grained, "template-level" taxonomy to provide an abstract description of algorithmic problems via tags (Wang et al., 2025c). In system, every task is labelled with three orthogonal categories: domain, data-structure, and algorithm class((Chollet et al., 2025)).
To obtain these labels, we (i) harvest a high-quality seed pool from the Luogu online judge (luo, 2025), and (ii) manually prune hallucinated or off-topic tags. Our ontology defines over 500 fine-grained tags, offering more than seven times the descriptive granulariyy of the 69 tags usod by LoctCode (sce Appendix A for comparison). The tag set is designed so that, taken together, the tags collectively reflect the complete programming knowledge entailed by each individual problem, while retaining a modest level of abstraction.
The three categories of tags capture complementary aspects of programming knowledge. First, the domain category encompasses the overarching problem space or application con- text. in which an algorithm operates, such as graph theory. Second, the data structure category pertains to the foundational mechanisms for manipulating data that underpin the algorithm's functionality and efficiency, like arrays. Third, the algorithm category refers to the core strategic paradigm employed, such as sorting, focusing on the decision-making logic. These algorithm tags directly shape the overall control flow of solution, they orchestrate the program logic and structure.
However, not all problems conform to highly standardized patterns. In contemporary algo- rithmic competition problems, for instance, many challenges necessitate solvers to discern the underlying mathematical structures, an approach commonly termed "constructive meth- ods". From a coding perspective, these constructive methods typically appear as compact code blocks that rely solely on fundamental loops arithmetic operations. Consequently, it is difficult to categorize them beyond a general "constructive method" tag. From the problem setter's viewpoint, such problems and their solutions stem from empirically ob- served mathematical structural properties, which inherently resist exhaustive coverage by conventional tags.
To synthesize novel queries, we sclect tags in the following workflow: first, we randomly pull 12 tags from each of three pools, and then let Claude-4 pick any 3 tags in total. This short list is fed back to the model that Claude can create one clear algorithm question with roughly 40 unit tests (Xu et al., 2025). Initially, we create approximately 1, 900 problems, but only retain the 300 that pass at least 85%of their tests (Xu et al., 2025), and call this clean set, TagComp. The detailed pipeline and prompt templates used can be found in A Appendix B and Appendix C.
5

---

Published as a conference paper at ICLR 2026
method Check Max_ (aec int, b: int) returns (max: int)
max := *;
assume max >= a; var value ;= Max ( a, b); assert value max;
Figure 3: We show an example where the equivalence score proves the given specifications are underspecified for returning the maximum between two integers. The code presents the statement. to verify whether the specification implies the code.
EVALUATION METRICS AND TASKS
A verifiable coding agent reduces hallucinations and provides trustworthy solutions aligned with users' intentions. As shown in Figure 1, our solution evaluation includes two steps, which are
verifying the equivalence between generated code and specifications, and validating the solution by translating formal specifications back to problem descrip- tions in natural langnages.
To understand the need for verifying the equivalence between the code and the specifica- tions, consider a simple binary search algorithm. The goal is to return the index of a key a sorted array a, or negative one the key not found. A weak but verifiable post- condition might only state that the output, idz, is within valid range: ensures -1 <= idx Length. While this specification passes the verifier, it fails to exactly describe the code. This creates dangerous loophole: incorrect implementation that doesn't actually find the key could still satisfy this weak condition, and the verifier would not catch the error.
Existing benchmarks do not offer metric to formally validate the quality of specifications. Without one, there is no way to guarantee that the verified code truly aligns with its intended behaviour. Instead, building up equivalence examines whether the specification is complete without ambiguities. Our equivalence score accomplishes the task by proving the bidirectional implication relationship:
whether the code falls into the latticos described by the specifications, and whether specifications tightly describe the code behaviour for any inputs.
Both proofs can be aulomatically completed by the Dafny verifier. The first direction can be verified by passing the annotation to the verifier. The second direction requires creating a statement that the specification implies the code for the verifier to check.
Figure 3 presents counterexample to illustrate how our equivalence score identifies an underspecified function. The Max method correctly returns the maximum of two integers and b, but its post-condition (ensures max a) is weak; doesn't guarantee that the output is also greater than or equal to b. To test if the specification fully implics the code's behaviour, we use the theck_Max, Spec method. This method creates an arbitrary value max, assumed to satisly all provided pre-conditions and post-conditions. Our equivalence score then tests the assertion that. variables described by the specifications are equal the method outputs. The Dafny verifier is guaranteed to find this assertion to be false without any false positives. Because the specification not strong enough to imply the code, this program would not reccive equivalence score.
As mentioned in the introduction, end-to-end formally verifiable code generation still challenging for current proprictary LLMs. Dafny has its programming logic, such as claiming the invariance old elements in arrays to support the proof. Therefore, we re-emphasize the importance of two auxiliary tasks to facilitate understanding of specific

---

Published as a conference paper at ICLR 2026
mances of Dafny, introduced in DafnyBench (Loughridge et al, 2025) and Veri-Code Scries I (2025):
Verifiable Code Refinement: Given annotated but unverified Dafny code, the model's goal is to add the necessary intermediate clauses, such as invariants and lemmas, to make the code pass the verifier. Succoss determined by successful verification.
Code-To-Spec Generation: Given a Dalny implementation, the model attempts to generate the strongest formal specification. The quality of the output is evaluated by measuring its strength improvement over a baseline, using the spec-superior-soon (Yan et al., 2025).
Our two sets of formal specifications map onto these auxiliary tasks. For the Verifiable Code Refinement task, models are challenged to fix our strong auto-formalized specifications. For the Sopecco-Speiiiicttin Generation task, models improve upon our weaker, but already verified, specifications.
EMPIRICAL EVALUATION
This section validates the quality of our benchmark and the reliability of our evaluation metric. We then present the performance of several state-of-the-art LLMs the end-to- end verifiable code generation task, followed by an analysis of our baselines on the two auxiliary tasks.
4.1 QUALITY METRICS
Specification Quality Our strong specification baseline, generated via auto- formalization, contains the ground-truth specification for 7.14% of the LeetCode-derived problems and 7.87% the synthetic TagComp problems, shown in Figure 5. In total, this process yields 161 complex algorithmic data with rigorously verified specifications. This significantly enriches the publicly available dataset of ground-truth specifications.
Code Transformation Quality To evaluate the quality of our Pychon-to-Dafny code transformation, we attempt to validate 1, 011 Dafny programs from the Code set against the corresponding unit tests. Due to the mismatch between Python and Dafny unit test formats, we only successfully execute 648 transformed files. However, the transformation is proven highly reliable, with 81.79% of the translated Dafny programs passing all tests.
Data Complexity The average Cyclomatic Complexity (MeCabe, 1976) quantitatively manifests the increasing complexity of our data, which counts the number linearly inde- pendent paths in the control flow graph. It is computed using the Radon software package for Python, listed Table 3.
We list the score for MBPP (Austin et al., 2021), since 50 manually annotated data in DafnySynthesis are based on MBPP-50 and the other 103 are also transformed from it. Thus, the analysis represents comparison to DafnySynthesis. We skip the analysis of CloverBench due to a lack Python implementations. Our benchmark's s average score of 5.63 is significantly higher than the 2.44 for DafnySynthesis, indicating more complicated control flows. Notably, our synthetically generated data is even slightly more complex than the Loeeeodeeer portion, with a score that is 0.25 points higher. This complexity is further validated by a manual rating from Claude-4, which classified the majority of our synthetic problems either medium or hard.
The table compares the code complexity of a previous benchmark and VeriEquii- Bench, indicating a more intricate control flow of our data.
Table 3:
Dataset
MBPP-50
MBPP
LeetCode
TagComp
Average Cyclomatic Complexity
2.44
2.78
5.38
5.63

---

Published as a conference paper at ICLR 2026
4.2 VALIDATION OF THE EVALUATION METRICS
We first validate our equivalence score on 50 expert-written verifiable code provided in DafnySyuthesis. CloverBench has reviewed their data and reported that 10% of the data does not give the ground-truth specification. After testing on our evaluation metric, the equivalence score, we figure out another nine examples where the formal specification con- tains ambiguities or the original code has errors. An example is shown in Figure 4, where the formal specification does not specify the invariance array length and leaves a logic gap. However, only eight examples out of 14 failures are successfully fixed by us, demonstrating the hardness in manual annotation. All examples with wrongly claimed ground-truth are listed in Appendix G.4 with the issues stated.
method SxapFirstandlast (a: array<int>) requires a. Length >
modifies
***н**** | The added post-condition ensures L.Lengh == old(a) Length
********
ensures a[o] old(a(a. Length 1])
ensures ala. Length 1] (1](
ensures forall k :: 1 <- k < Length - --> a akk]
(a](a(k))
var tmp := [0]]
a[0] : a(a. Length 1]; a(a Length 1] := tmp;
Figure 4: An example of a weak specification in sample #625 that fails equivalence scoring- The formal specification is ambiguous it omits post-condition the invariance of the array's length.
Next, we evaluate all previous benchmarks and observe serious quality issue in previously provided ground-truth formal specifications, shown Table 4. It has been discussed that the equivalence check relying natural language provided in Clover has limitations, and turns out that a large number specifications do not establish the equivalence with the code. Meanwhile, Dafny Bench is not designed for checking the completeness of specifications and thus, gives the lowest score.
Furthermore, evaluate Grok-4's translation ability, using Claude-4-somna (Wang et al., 2025a). We test on our filtered auto-formalized specifications derived from LeetCode and observe high success rate of 82.98%, validating its functionality to translate.
judge
Table 4: The percentage of data gaining the equivalence score in previous benchmarks.
Datasct
DafnySynthesis
CloverBench
DafnyBench
Equivalence Score
76.22%
61.29%
43.09%
4.3 VERIFIABLE CODE GENERATION
Figure 5 (b) and (c) present the pass@4 results of three proprietary LLMs end-to-end formally verifiable code generation, tested on CloverBench and our contamination-free syn- thetic set, TagComp. We also evaluate three open-source model with complete results presented in Figure 23 in Appendix G.2. On the previous CloverBench benchmark, a ca- pable model like Claude achieves 75.81% success rate, with most errors stemming from
8

---

Published as a conference paper at ICLR 2026
issues in specification writing rather than fundamental code generation flaws. However, our more challenging TagComp dataset, this performance collapses dramatically.
A closer look at benchmark results reveals the challenge of verifiable code generation. In our rigorous two-step evaluation, the equivalence score measures the formal alignment of code and specifications, while the exact matching score further validates against the original natural language intent. While Claude is the most capable model producing syntactically correct Dafny code, achieving success rate of 73.79%, all three models struggle substan- tially to generate verifiable solutions. More critically, they perform poorly at producing mutually equivalent code and specifications that are properly aligned with the user's query intent. GPT achieves the highest autual-cquivalene rate at 2.65%; however, further analy- sis reveals that these mutually equivalent solutions often provide simplified implementation, hacking the reward and fail to satisfy the intended user requirements. This result under- scores the difficulty of formally verifiable code generation on complex algorithmic problems, requiring strong coding and formal reasoning abilities.
Multiple failure cases are provided in Appendix E, where we also illustrate how Claude-4, acting judge, identifies incorrect specifications, even in difficult scenarios where only subset of post-conditions does not fully satisfy the query requirements.
[ol: Dotaset Quality Analysis
(b) CloverBench Performance
(c) VentEiuiv Bench Performance
32.08
22.2
3.7
60
5.8
Ledtooe
TepComp
GPT
Claude
Iicc Matching
ligivolent
Venified
Synto-Comee
Figure 5: Exact matching score measures the percentage of data passing our two-step eval- uation framework, giving solutions aligned with the query intention. Part (a) gives the amount of verified ground-truth solutions in our benchmark. Part (b) shows that the pre- vious CloverBench benchmark is too simple to properly evaluate the advanced reasoning abilitics of capable models, as evidenced by a high success rate. Part (c) presents the pass04 performance of gemini-2.0 gemini-2.5-flash, gpt-5, and claude-4-sonnet our end-to- end verifiable code generation task.
4.4 AUXILIARY TASKS
For the two auxiliary tasks mentioned in Section 3, we provide two Rl-trained baselines, with the SFT model provided in Veri-Code Series (2025). As stated, the verifiable code refinement task uses passing the verifier or not as the reward to infill intermediate clauses, while the spec generation task uses the spec superior score introduced by Yan et al. (2025). Spee superior score measures whether the generation specifications described the code better than our weak baseline. The choice of the RI. algorithm and hyperparameto meterss follows their implementation as well. We split our LeetCode transformed data into three parts with 1770 training data, 200 validation data and 204 out-of-domain test data, using tags uncovered by the training data.
Our baseline scores 17.68% for the refinement task and 54% for the spec generation task on the validation set. However, in the spec generation task, almost no data generates a complete specification, resulting in equivalence score. A possible reason is that the SFT model provided is trained on overly simple problems and does not have enough exploration ability. The training curve and results on the test set are presented in Appendix G.3.
9

---

Published as a conference paper at ICLR 2026
5 RELATED WORKS
central challenge in advancing LLMs is developing metrics that not ouly assess perfor- mance but also provide clear signal for improvement we desire. Outcome-based metrics, such as final-answer accuracy in mathematical reasoning (Cobbe et al., 2021) or pass rates on unit. tests in code generation (Austin et al., 2021), are prevalent but limited. They disregard the fidelity of the reasoning process and remain susceptible to false positives, a limitation shared by methods employing external solvers for verification (Huang al., 2025; Feng et al., 2025).
Formal verification offers a more rigorous evaluation alternative, using proof checkers like Dafny (Leino, 2010) or Lean (De Moura et al., 2015) to provide an unambiguous correctness signal without roquiring a ground-trute solution. However, in verifiable code generation, this signal is fundamentally undiirccional:: it validates that the code satisfies specification but offers no guarantees about the specification's 's quality. This vulnerability allows models to pass verification ion using trivial or flawed specifications (Yan et al., 2025). While Yan et al. (2025) attempt to address this by comparing generated specifications against ground-truthh specifications using a partial order, their method remains dependent on the quality and availability of trusted ground-truth. In contrast, our work introduces a formal equivalence metric that verifies the bidirectional correspondence between code and specification. This approach ensures the specification fully captures the program's 's behavior without relying on ground-truth specification.
The absence of such a metric has hampered the creation of high-quality benchmarks for aut- oformalization. Existing datasets often lack the tripartite alignment. of natural language, code, and formal specifications (Lohn & Welleck, 2024; Loughridge et al., 2025; Dougherty & Mehtaa 2025; Yan et al., 2025) or are small-scale due to the high cost of manual annota- tion (Misu et al., 2024; Sun et al., 2024; Miranda et al., 2025; Ye ct al., 2025b). Attempts to automate equivalence checking have proven unreliable; for instance, Clover (Sun al., 2024) relies LLM-based judgments that suffer from high error rates. Addressing these deficiencies, we present VeriEquiv Bench, a benchnark an order of magnilude larger than prior work. Enabled by our robust equivalence metric, it provides large-scale, trustworthy resource for developing and evaluating models for verifiable code generation.
6 CONCLUSION
In this paper, we confront a foundational challenge hindering the development of reliable verifiable systems: the dependence small, manualy--annttaeed benchmarks for formal ver- ification. This issue limits the scale and complexity of evaluation and has also introduced a ceiling by human knowledge. Our work breaks the dependency and introduces VeriEquiv- Bench, a large-scale end-to-end formally verifiable code generation benchmark. Our novel automated equivalence score provides a rigorous evaluation signal without any need for human-writte
ground-truth specifications. Second, our structured tagging system enables the scalable, automated synthesis of novel and complex problems, directly addressing the data generation bottleneck. By using VeriEquivBBech to evaluate state-of-the-art. L.MSs, we have demonstratod that end-to-end verifiable code generation remains an open challenge, fact obscured by the inflated success rates simpler, older benchmarks. Following the cent discussions on self-evolving agents, our benchmark provides a scalable data generation engine and reliable auto-ovaluation metric, setting the groundwork to foster trustworthy AI agents with exact solution generation and sustainably supervise snper-intelligence agents.
7 REPRODUCIBILITY STATEMENT
The code and our dataset are included in the supplementary material and will be publicly available after the double-blind review process for reprodueibility.
10

---

Published as a conference paper at ICLR 2026
8 ETHICS STATEMENT
This work does not present any foresccable ethical concerns. The research involves only publicly available datasets and does not use analyze sensitive personally identifiable information.
REFERENCES
Luogu online judge, 2025. URL https://ww..wwete luogu com. cn/, Accessed: 2025-03-28
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint Xrxie:2108.00 07732, 2021.
Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Are- agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXi::2505.1188 2025.
Karl Cobbe, Vineet. Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL 14168.
nttps://arxie wwwpiii.org/bbs.o. arggabbs/2100... 14168.
Domenico Cotroneo, Cristina Improta, Pietro Liguori, and Roberto Natella. Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks. In Proceedings of the 32nd IEEE/ACM International Conference Program Comprehension, pp. 280 292, 2024.
David Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, et al. Towards guaranteed safe al: A framework for ensuring robust and reliable ai systems. arXiv preprint arXiv:245..0660 2024.
Leonardo De Moura and Nikolaj Bjørner. Z3: An efficient smi solver. In International conference Tools and Algorithms for the Construction and Analysis of Systems, pp. 337-340. Springer, 2008.
Leonardo De Moura, Soonho Kong, Jeremy Avigad, Floris Van Doorn, and Jakob Von Raumer. The lean theorem prover (system description). In International Conference Automated Deduction, pp. 3888 Springer, 2015.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey in-context learning. 2023. doi: (LA8550/ARXTI 20..2220..00220 .00234. URL https: //arxiv. orgg abs https///3000000 00234.
Quinn Dougherty and Ronak Mehta. Proving the coding interview: A benchmark for for- mally verified code generation. In 2025 IEEE/ACE ACM International Workshop on Large Language Models for Code. (LLM4Codde), pp. 72-79. IEFF, 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong, Retool: Reinforcement learning for strategic tool use in Ilms, 2025. URL https: htmps://araee //trxv//..iivva/ org/ aba/abbs///00 11536.
András György, Tor Lattimore, Nevena Lazić, and Csaba Szepesvári. Beyond statis- tical learning: Exact learning is essential for general intelligence. arviv preprint arXim:2500.220 223008, 2025.
Xuhan Huang, Qingning Shen, Yan Hu, Anningzhe Gao, and Benyou Wang. LLMs for mathematical modeling: Towards bridging the gap between natural and mathematical languages. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Associ- alion. for Compulational Länguistics: NAACL 2025, pp. 2678 2710, Albuquerque, New
11

---

Published as a conference paper at ICLR 2026
Mexico, April 2025. Association for Computational Linguisties. ISBN 977-8-89176-195 7. doi: 10.18653 / /0225. findings- naacl.16.. URL httpa////ccaantoooooorrog/////// findings-nnacl. 146/ 146/.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sem, and Ion Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code. arviv preprinl ar Xiv:2403.07974, 2024.
Carlos E Jimencz, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In 12th Intervational Conference on Learning Representations, ICLR 2024, 2024.
K Rustan M Leino. Dafny: An automatic program verifier for functional correctness. In International conference on logic for programming artificial intelligence and reasoning, pp. 348-370. Springer, 2010.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092 1097, 2022a.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmect Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-lewe code generation with alphacode. Science, 378(6624): 1092 1097, December 2022b. ISSN 1095-9203. doi: 10. 00.112//siienceeaayl155 URL http: //dx. org/ org/oo///// 10.
Evan Lohn and Scan Welleck. minicodeprops: a minimal benchmark for proving code propertics. arXiv preprint arXix:2466. 11915, 2024.
Chloe R Loughridge, Qinyi Sun, Seth Ahrenbach, Federico Cassano, Chuyue Sun, Ying Sheng, Anish Mudide, Md Rakib Hossain Misu, Nada Amin, and Max Tegmark. Dafnybench: A benchmark for formal software verification. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL yBgTVWccIx.
Thomas J. McCabe. A complexity mcasure. JEEE Transactions on Software Engineering, (2::344):08-32) 1976. doi: 1.11099TSE.1976... 233837.
Brando Miranda, Zhanke Zhou, Allen Nie, Flyas Obbad, Leni Aniva, Kai Fronsdal, Weston Kirk, Dilara Soylu, Andrea Yu, Ying Li, et al. Veribench: End-to-end formal verification benchmark for ai code generation in lean 4. In 2nd Al for Math Workshop @ ICML 2025, 2025.
Md Rakib Hossaín Misu, Cristina V Lopes, Tris Ma, and James Noble. Towards ai-assisted synthesis of verified dafny methods. Proceedings of the ACM on Software Engineering, (FSE))812 835, 2024.
Martin Riddell, Ansong Ni, and Arman Cohan. Quantifying contamination in evaluat- ing code generation capabilities of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Popers), pp- 14116-14137, 2024.
Parshin Shojace, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and lim- itations of reasoning models via the lens of problem complexity. arXiv preprint arXim:2506.069) 2025.
David Silver, Satinder Singh, Doina Precup, and Richard Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021.
12

---

Published as a conference paper at ICLR 2026
Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett. Clover: Clo sed-loop ver ifiable code generation. In International Symposium on AI Verification, pp. 134-155. Springer, 2024.
Amitayush Thakur, Jasper Lee, George Tsoukalas, Meghana Sistla, Matthew Zhao, Stefan Zetzsche, Greg Durrett, Yisong Yue, and Swarat Chaudhuri. Clever: A curated bench- mark for formally verified code generation. arXiv preprint arXiv:250,. 18938, 2025.
Shangqing Tu, Kejian Zhu, Yushi Bai, Zijun Yao, Lei Hou, and Juanzi Li. Dice: Detecting in-distribution contamination in Ilm's 's fine-tuning phase for math reasoning. arXiv preprint arXiv:2400 04177, 2024.
Yutong Wang, Pengliang Ji, Chaoqun Yang, Kaixin Li, Ming Hu, Jiaoyang Li, and Guil- laume Sartoretti. Mcts-judge: Tost-time scaling in llm-as-a-judge for code correctness evaluation. CoRR, 2025a.
Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. Towards understanding the charactoristics of code generation crrors made by large language models. In 2025 TEEE/ACM 47th International Conference on Software Engineering (ICSE), pp. 717 717. IEEE Computer Society, 2025b.
Zihan Wang, Jiaze Chen, Zhicheng Liu, Markus Mak, Yidi Du, Geonsik Moon, Luoqi Xu, Aaron Tua, Kunshuo Peng, Jiayi Lu, Mingfei Xia, Boqian Zou, Chenyang Ran, Guang Tian, Shoutai Zhu, Ycheng Duan, Zhenghui Kang, Zhenxing Lin, Shangshu Li, Qiang Luo, Qingshen Long, Zhiyong Chen, Yihan Xiao, Yurong Wu, Daoguang Zan, Yuyi Fu, Mingxuan Wang, and Ming Ding. Aethercode: Evaluating lmss' ability to win in premier programming competitions, 2025c. URL https: //txxvvvvv//////////////////aa// org/ aa//a/bbs//..../////// 16402.
Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifong Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leelcodedataset: A temporal dataset for robust evaluation and efficient training of code Ilmss 2025. URL attps://arxivv wrrg/abs/2555 14655.
Zhangehen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025. URL https: //arxiv.org abs abaaabs//000 250.0.2951........
Chuanhao Yan, Fengdi Che, Xuhan Huang, Xu Xu, Xin Li, Yizhi Li, Xingwei Qu, Jingzhe Shi, Zhuangzhuang He, Chenghua Lin, et al. Re: Form reducing human priors in scalable formal software verification with rl in 1lms: A preliminary study on dafny. arXiv preprint arXiv:2507. 16331, 2025.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. In Proceedings of the 13th International Conference on Learning Representations, ICLR '25, April 2025a. Full version available at attpps:/ssstt
comm/abstract-5550225
Zhe Ye, Zhengxu Yan, Jingxuan He, Timothe Kasriel, Kaiyu Yang, and Dawn Song. Verina: Benchmarking verifiable code generation. arXiv preprint arXie:2505.2911 23135, 2025b.
Huaiyuan Ying, Zijian Wu, Yihan Geng, Jiayu Wang, Dahua Lin, and Kai Chen. Lean work- book: large-scale lean problem set formalized from natural language math problems. Advances in Neural Information Processing Systems, 37:105848 105863, 2024.
Box Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. Utboost: Rigorous evaluation of coding agents on swe-bench. aixie preprint ar arxi::506. Nix:2506.09280, 2025.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. aix preprint arxiv:250 23335555.003335, 2025.
13

---

Published as a conference paper at ICLR 2026
A DETAILS ABOUT ALGORITHM TAGS
To assemble a suitable tag vocabulary, we first collect high-quality, high-frequency labels from Luogu-a compective-programmigg platform with millions of users and unusually fine-grained tag taxonomy-and and treat them as a seed set. For cach LectCode problem, the model is prompted to pick the most relevant domain, data-structure, and algorithm tags from this pool, and is allowed to introduce new tags only when no suitable match exists. All model-sclected tags are pooled, automatically partitioned into the three coarse categories, and then manually filtered in single pass: hallucinated labels are removed, uear-duplicates merged, and overly broad or overly narrow tags discarded. The resulting inventory contains over 500 clean triples that serve as the controlled vocabulary for subsequent tag-composition.
nilting moudon
Nesbalie entrovti
Pretters
Table Statistics algorithm tags
Oyaam hingrensiing
Avay
Mertt
Stoing morrpsattnn
Sorting
Tag category
Numbers
Domain
53
Hattermecc
Basb Salet
Data Structure
68
Algorithm
480
Figure 6: The fifteen most frequently used tags in our dataset.
14

---

Published as a conference paper at ICLR 2026
The complete curated tag set is listed below, grouped under the three top-level categories: domain, data-structure, and algorithm.
Table 6: Domain tags
Category
Tags
Mathematics, Number Theory, Probability Theory, Combinatorial Mathematics, Linear Algebra, Computational Geometry, Plane geome- try, ctree-dimensional computational geometry, Graph Theory, Simple Graph Theory, Game Theory Information Theory, Dynamic Connec- tivity, expectation, Set Cover Problem, allocation problem, Extremum problem, path problem, Chess Board Problem, Stock Problem, Island Problem, Maze Problem, Josephus problem, Frobenius problem, N- Queens Problem, Kuight's Tour Problem, Two-dimensional partial or- der problem, matching problem, Pairing problem, Interval problems, Knapsack problem, Subset Sum Problem, Jump Game, Maximum Sub- array Problem, Maximum Subsequence Problem, Largest Rectangle in Histogram, longest chain, Path counting, Path Statistics, Connectivity, Reachability analysis, periodic, Discrete Event Simulation, Time con- straint, Pormutations and Combinations, Counting Principles, Inclusion- Exclusion Principle, Pigeonhole principle, Catalan number, Stirling numbers of the second kind, Combinatorial counting, Combinatorial Op- timization, Mathematical Techniques
Domain
Table Data Structure tags
Category
Tags
array, Two-dimensional array, Multidimennsionll array, sortod array, Cir- cular array, tagged array, Difference Array, rolling array, Linked List, doubly linked list, Circular Linked List, Queue, deque, Priority Queue, Stack, monotonic stack, monotonic queue, tree, undirected tree, un- rooted tree, Ring tree, Binary Tree, Complete Binary Tree, Perfect Binary Tree, Balanced Binary Tree, Binary Search Tree, Tree data structure, Trie, Segment Tree, Binary Indexed Tree, Heap, heap min heap, Huffman tree, Set, Hash Table, Adjacency List, Adjacency Matrix, weight graph, Bipartite graph, Complete graph, Undirected graph, di- rected graph, Reverse graph, Star graph, Directed Acyelic Graph (DAG), Balanced tree, sparse matrix, Disjoint Set Union (DSU), Red-Black Tree, AVL Tree, B-Tree, B+ Tree, Skip List, Bloom Filter, LRU Cache, Prefix Tree, Suffix Tree, Suffix Array, Cartesian Tree, Splay Tree, Scapegoat Tree, Persistent Data Structure, Linear List, Sparse Table, Mo's Algo- rithm Structure, Leftist Tree, Fibonacci Heap, Pairing Heap
Data Structure
15

---

Published as a conference paper at ICLR 2026
Table 8: Algorithm tags
Category
Tags
Compression algorithm,-ynnmiic Programming..yynan Programming
Linear DP,Dynamic Programming- DinggLLL,,DDmmmiie Programming-Pres Prefix
Programming State Compression, Dynamic Programming Interval DP,Dynamic Programming 2D Dy,Dynamic Programming Prefix Sun Opti- Inraation,ynamit Programming Top-Down, Dynamic Programming Iterative, Programming, Compression algorithm,yynaate Pro- gramming, Dynamic Programming Linear Dy,Dynamic Programming: Programming- ceremiiinnPeefixx Sum, Bymaminiicc Programming - 0/1 Knapeack.Dynamic Programming State Cuupression.Dynantee Pro- gramming Interval Dynamiic Programming 2D DP Dynamic Programming Prefix Sun Optimization,
Symammmmiic Programming 0/1
Dynnmiiin,Dnmmicc
Algorithm-1
Programming Dynamic Program-
Top-Down.Dynamic Programming ming, State Compression DP.Dynamic Programming Mathemati- cal Optimization, Digital DP. UP.Count Dr,Tree DP. DP.knapsack St,State Compression DP,Dynamic Programming (DP), DP,Bidirectional DP,Sequence DP. Dt.State Machine DP, DP.Bottom-up Dynamic Seerraminng..ddieettion
Mulliiuutssouure BFS,0-1 BeS,Depth-Finn Search (DFS), Breadthhiitt Search (BFS)
Memorziiinn Statt space search,state scarch, scarch,Path Finding.Binary scarch,Binary Scarch Answer, Binaryy Scarch Right Honndry,,Binary Search Left Boundary,Binary Search Count,BBiaa Search Peak Pindiig,Binary Search Maximum Value, Search-Prefix Sum,Binar Search Middle Element,Binary Search Line Search
Table 9: Algorithm tags
Category
Tags
Sorting,Merge Quik, , Three-way Inncksorr,Insertioo
Stable Sort,Sorting Lexicographical Drdeer,Diffeence Sorting. colticottiiono sorting, Wiggle Sort, oo-t.in-place sorting, Topological sorting, Quick Se- algorithm,suffix
Sort,Counting Sort,Bucke Sort.Soot
Custom Sort,
lcct,KMPP algorithm, array,suffix tree, free,,ffunttio,prefff function,string pattern match- Binetrrackingg.Buuuaattiiooo Enumeration, Binary
ing,string wildcard Cauumeration,Comiialotott Chumeration,Subsee Enumeration, Combinatorial Enumeration, Two- Interval Opera-
dimensional enumeration, Simiuaatiinn,iimmlatioon.n Simmulatin,,Greedy..Gii Greedy tivide and conquer, Divide and Conquer String Splitting, and Conquer Closest Pair of Points in a Plane, Central Expansion Method, Stainin method, Pointers,Two Pointers - Sliding Window Fast and slow point- ers, Three Pointers,par path compression,Pate Tracing, reconstrue- tith, Planning, Planning,Single-SS Single-Source Shortest Path.,Mlti-Source Shortest Peth.Second shortest circuit, Conntt,Cn Shortest. Path.shortest path, Heppooeep-pp Dijksttaa,iijkstra algorithm,,ijjkstrassss Algo- sycle-finding algo-
Divide
Algorithm-2
method, window, Two
rithm Beliaant,Bllllaa-aeee Ford
Serortth,,Pri''' Kruskal' algorithm, Prim's ligorithn,Minmmmn Spanning Tree,
Bipartite MatchinggMMxximun Matching in Bipartite algorithm, Minimum Cost Maximum Flow, Graham scan, Welzl's algo- sithm,lincal sieve, sieve,Euler Euler sicve, Eratosthenes sieve,Prime Sieve, Euclidean Aigorithm.Bézout' Bézout's 's identity.Bézoutss 's thcorem, Greatest Common Divisor (GaD),Least Common Multiple (LCM), Prime Number Chock
Hungarran
algorithm,
16

---

Published as a conference paper at ICLR 2026
Table 10: Algorithm tags
Category
Tags
Euclidean algorithm, Bézout's identity, Bézout's theorem,Greatest Greatest Common Divisor (riisorGCCD,LLeast Common Multiple(T.CMM) Prime Number Phimk,Prime factorization, Vactorization,Intee factorization, Cannri Cantor expansion,Fass exponentiation, Matrix Fast Exponentiation,Matrr aultiplication, rotation,matrix Transposition,Mattii opera- rotasooot matrix, flood fill algorithm,A" algorithm, Tarjan's algo- rithm,Morris traversr Traversal, Inorder Traversal, Postordde traversal, order Lraversal,Level Order Traversal, Reverse inorder Vertical Order Booudaryl, traversal.Diagonal Traversal,2D matrix traversal,Travvrsee of 2D Array, Graph Traversal,Linkes list traver- sal, Tree traversal traversal,Bicirectionn traversal, traversal, Path Validation, Path compari-
traversal traversal.zigzag traversal,spiral, traversal, Vertical L
Sevversal,Reeerss Traversal,nee-ar Construction,/exioorrappii Jon.Lexicograpphccllly smallest, path, Maximum Value Search,Maximun
counting. Statistics,
Value SaaintenaneeRanng Maximum,Maximut Column Value,prefit maximum,sufiix minimum,sutii
product,prefix product, Prefix Sum,Prefix Sum Difference, Sum Modular Arithmetic, Sum Binary Search Optimization, ,DD prefix sum,suffix sum,partial sum, subarray sum, submatrix sum, Area Sum,Area Calculation,ASCA code haapppllattin,,araattio Mapping, Character Count,character
Groodingg,igitee Parsing,Data Extraction Integer Reversal, Square Inot,I Division, FraisioonFrattion Addition and Subtraction Fractional Arithmetic, Fraction simpli- Score
Frequency,iigita
Algorithm-3
fication, Score calculation.percenttin Circular shift,Loop Detec- tion,Ring Detection, Strings.String comparison, Case Conversion concate- nation,string concatenation, String manipulation,Strine search,string matching,Strinnn Substring
Assessment,Braddee Bracket Matching, Isomoorhii
replacement,
soammingg,ttiingg slicing,string splitting,String pression, decoding.stio string matching,prefit matching, Prefix Check,Longest Common Pre- Longess Common Suffix,Longer Common Subsequence,
parsing.string continuity,substring
Common Subarray,Longest Repeating Belistring,Longeee Palindromic Subsequence, Non-decreasing Subarray.Longest Consecutive Sequence,Jongest consecutive characters, Word Chain,Zigzas, Conver- parsing, Exprossion Evaluation, Reverss precodence, Lexical Polish Notation, Postfix expression, decoodiigdccoooing decoding,RRun Run-
length encoding,Sel Operations,Sel
Intersection,
oper- State
alion,Bitwise operation optinization, Bitwise Operations Compression, bittrrsssioonbbbmms.k..k... bitwask,Bitwise Bitwise OR, AND operation, Monttion,,O,,bbbie biDary,inry,Bnaar Addition,binary Baliitiinn,Biaaat counting, bit count, Hamming dis- tance, Two's Complemmnt,,Mddube Congruunn,,,,,, Congruence theorem, diivisible, erty,divisor, perfect equare,square number, Perffct number,Ugly num- ber,trailing trailing digit reparation,Digiie code, Pede,,eetttaiin ring, Cyclic permutation, Pascall Sttangle.Fermat' theorem on sums of two, squares, Pythagorean theorem, Triangle incquality.absooute value,absolute value inequality,Bip Big Integer Addition, High precision
arithmetti,moduulo 3 opera-
prop-
biocessing,Diiitaal Sum,Gray Permutation
Next
17

---

Published as a conference paper at ICLR 2026
Table 11: Algorithm tags
Category
Tags
Floating-poinn Piaeessinn.lootingg--o. precision,Lineer
htapparisoonnoainngggno Number Opera- wttpppoollllloooaa..... factoaa, factorial, Sum Rational number representation, recurring decimal,
armmula,rithmmtte sequence, Arithmece sequence sunmation,pate sum,Maximum Sun Path,,nx spac- ing.Neiighbr Count,Adjacee elements, Aanaaett,Adjcentt Element Differ- ence, Global Lnversion,Loca inversion
of Suraare,,S,,S,mmmme
pair, aaaraam, vowel coordinates, coordiiate system, coordiate comparison,coordinats translation, cogolaaioon compression.2) off- set,2D plane,3D space,collinear points, Collinearity detection,couvex hull, minimuum bounding rectangle,Triangle Area,Rectangle Area Cal- culation,Overraapiinn
Overlapping Area Caleulation,,eectnggee Intersection, Citrr Circle- Rectangle Interscction Detection.Minmmin Enclosing Circlc,Spatial segmentation,21 cutting.Spatial optimization,
complex- space complexity,Lineee Linear space
ity oplimizaation,Cnss
plexity,Time complexity analysis,Linear time complexity,Linear
www.oeeoorroooossippeeeeee Offire Offline processing.Dynaae Dynamic uydate,DDuamic Laintenanne,Dynannnn Maintenance Interval,Dynamic modification, Range query, Intervall computation, Interval Statistics, Range untete,Interval Interval Merging.Interve Scheduling, Range extrema, Path Intersection De- calculation, dist projecccooetiionncooo cross Construction, Tree Reconstruction,
Range Saaintenance,Singl--aa
coverage,Interrss Interval tection, Distance tance, tance, Chebyshev ing, construct, Binary construction,
Calcuulatin,,uullidann Euclidean
distance, Bestance,Manhattann Manhattan dis-
product, Polar sort- Construction, Tree Re- Conerrrction,Cnsttuutttnnng Constructing the answer
Algorithm-4
in reverse order,reverse, Reverse Linked List, Lint,Linked Tist Rever- sal,String Reversal, Array Rearrangementt Linked List Reorder- Node and
Array,split string,Split ma- ID Array to 2D Array, trix,sparse matrix matrix, Matrix, Tree delction operation, Tree depth, Tree Centroid, Tree Di- ameter, suutter,subtee,Suubtte Sum,leaf nodeiintermediate node,dummn node,sentinel Mode,Middie of the Linked List,indegree indreree and monooniic array, Decision MonooonnnMit. operations, Logical Polaiito, Coolarity,Parity dling,Egge case
Goverage.Addjceeee
Monooooniiiityyoooooooiiii Monotonicity sequence,
Operations,Conditoonna Conditional statement, Filter Cri- han-
Bouuddryy check, Boundary handling, Status Check,Status Log.State tran- sition,State Machine, Finite State iirtomaton,Priorigg
han- dling, Query Processing, processing, Overflow handling, Carry han- dling, Recurguon Inciursiive,IIuuliiee dethood dravvee Araverse,,rray search,State
traversal, Grid traversal, directional traversal,directionall
compres- sion, Handling Duplicate Elements, deduplication, Emumeration optimiza- tion,Sequence comparison,
CnccppliaationEEEmmaatiioo
ceaaparrson.coppariin comparison function, Comparator, Regular chaining,Swerr
Expression, Pointer manipulation, Method Methoo cration, Displacmenn operation, Roor and Column Opera- tions, pronuc.oodut,MMnnn Multiplication
op-
Principle, Exponentiaiion, Basee Base,Base Conversion,Clos Clock losu section,IP addresssreordeer wevorddeing.Prriiaae Partial Ordering, Equation Equuring.Equation RallinngRanddomization..
thinking, Horse Rac- ing Strategy, Connected component, Coneetttdd Component,Stroom Connected Component, Common Ancestor (Lulei,,ullrian circuit, Hamiltonian path
18

---

Published as a conference paper at ICLR 2026
B PIPELINE OF TAG COMPOSITION
Figure 7 illustrates our pipeline for generating new programming problems through tag composition. The process begins by creating candidate pool of 36 tags, randomly selecting 12 from cach of our three categorics: domain, algorithm, and data structure. This pool is provided to LLM, which is prompted to select coherent subset of three to eight tags that form & promising basis for a new problem. Using this selected combination, we then instruct the LLM to generate a complete task, comprising problem description, corresponding unit tests, and a Python solution. As final quality control step, we filter these generations by executing the unit tests. We retain only those instances where the generated Python code passes all tests, ultimately yielding dataset of 300 validated programs.
Domains
Rondomly
Thers....
Algorithms
Domaine go
Dramic
Randamly
12 Data Structures
12 Algorithms
Tag Set:
LLM
Programming,
Data Structures
Kandouly
Suppose.. caves Iaput Python_Code_
Unit Pass Rate 88%
300 Samples Pythom
Figure 7: The pipeline for the tag compositon process.
19

---

Published as a conference paper at ICLR 2026
C PROMPT TEMPLATES
C.1 NOVEL TAG COMBINATION
[Task]
You have three categories of lags: domain, algorithm, and data structure, each containing lags. Your task select a combination of 3-8 tags from these categories to form coherent programming problem. The problem should have specified difficulty level: easy, medium, or hard. Ensure the selected tags are compati- ble and can logically form single problem. Provide the chosen tags, the difficulty level.
Requirements]]
1.The task is clearly defined, specifying the need to select 8 lags from three categories (domain, algorithm, data_structuree to form a coherent programming problem with specified difficulty level.
2. R.Rquirements outline the tag selection process, ensuring compatibility and reason- able tag collection, the need for a difficulty level. 3.e selected tags must be compatible and form reasonable tag collection that results in a practical and solvable programming problem.
4.The problem must be assigned one of three difficulty levels: easy, medium, hard, reflecting the complexity of the problem based on the selected tags.
Domain tags] {l domain tags }} [Algorithm lags]
{l algo tags }}
Data Structure tags) {f data_tags }}
Output Format
This is the ouput format, You must respond in this specified output format:
"all taga": "Graph Theory, Depth Search, Union Union-Find, Graph, Diajoint Set",
"Domain": "Graph Theory",
"Algorithm"; "Depth-First Search, Union-Find",
"Data "taasttuucuuee"""raaa Disjoint Set" "Difficulty Level": "medium",
Problem End/>
Figure 8: The prompt is for selecting useful tags. We feed the model the 36 real tags from 3 categories randomly that will later drive new-problem generation, it returns the 3-8 tags that form the most promising combination.
20

---

Published as a conference paper at ICLR 2026
C.2 NOVEL PROBLEM SYNTHESIS
[Task]
You are an expert algorithm problem creator. Your task is create an easy or medium difficulty ranking original coding problem using the given algorithm inal.en the given tags gencrate a new problem. The problem should be completely original coding problem that is NOT from any existing platforms (LeetCode, Codeforces, etc.)
or texkbooks.
[Requirements]
1. Create a truly novel problem scenario with constraints
2. Combine the given tags in innovative ways
3. Ensure the problem is solvable but challenging
4. Provide clear problem statement, examples, and constraints
5. Rate the difficulty(ean) medium, hard) appropriately
[Algorithm lags]
tags
Output Format
This is the output format. You must respond in this specified output format: < <[Problem Begin|>
problem
<(Problem End/>
Figure 9:
The prompt uses the previously obtained real tags to generate a brand-new
problem.
C.3 SPEC-TO-NL
Prompt]]
Can you think of minimal code implementation satisfying the specification? For example, the spec just ensures true, then any code can work. If the specification ensures return values within range, then assigning any value within the range work. Please think of the minimum code implementation and then come up a problem description this minimal code satisfies. Below are the specifications:
Figure 10: The prompt asks the model to read the supplied Dafny specification and produce a concise summary that fully describes the coding problem defines.
21

---

Published as a conference paper at ICLR 2026
C.4 LLM-AS-A-JUDGE
[romptt
You expert in analyzing algorithm problem descriptions. You need carefully analyze the equivalence of two algorithm descriptions based on the following dimensions: Core Problem Equivalence: Is the essence of the problem identical? Are the solution objectives consistent?
1.
2. Constraint Comparison: Input constraints Boundary case handling Special case requirements
3.
Complexity Requirements: Time complexity requirements Space complexity
requirements
4. Detail Completeness:
Information loss check Additional information analysis Please provide an equivalence score from 0-100 and give detailed analysis of your reasoning.
Please analyze the equivalence between the following two algorithm descriptions:
Original Description:
New Description:
Please analyze according to the dimonsions above and provide a score with detailed explanation.Only put the score in code block surrounded by triple backticks ("t)"""
Figure 11: The prompt instructs the model to determine whether the two given programming problems semantically oquivalent.
22

---

Published as a conference paper at ICLR 2026
C.5 NL QUERY To VERIFIABLE CODE
Prompp]
You will get a problem description. Your task is to give a fully verified Dafny program.
Refer to the Dafny examples guidance:
Fewshot Examples:
Problem description:
Please write the Dafny code that implements the functionality while ensuring:
1. Reference the Python implementation for algorithmic insights;
2. Add appropriate loop invariants with brief explanations;
3. Ensure full verification your code must pass the Dafny verifier.
Output the complete Dafny program, including both the specification and implemen- tation.
Figure 12: The prompt turns natural-lanuuage query into fully formal, verifiable speci- fication together with correct-by-consstuution code.
D MODEL ARGUMENT SETTINGS
Throughout all experiments, we retained the default values for every hyperparameter except temperature and top-p. To balance creativity with reliability, we employed a two-level sampling strategy: during the initial proeffiationngeeerattin Figure 2, temperature was set to 0.7 and top-p to 0.9 to encourage diversity for generating high-quality formal specifications equivalent to the NL query(Li et al., 2022b).
stage shown in Pipeline in
In all other phases, including annotated code generation in Pipeline 3 in Figure 2 and model evaluation, temperature was reduced to 0.5 and top-p to 0.8 promote deterministic and consistent outputs.
The prompts provided in Section C.
For model evaluation, the coding agent is provided with the problem in natural language and is asked to generate four rollouts of annotated Dafny code. The equivalence score is then evaluated for each rollout. Next, those rollouts that gain the equivalent score are passed to Grok-4 to translate specifications back into NL. Finally, Claude-4 judges the equivalence between the translated new description and the original query.
23

---

Published as a conference paper at ICLR 2026
E QUALITATIVE ANALYSIS
E.1 EXAMPLES OF VERIFICATION FAILURES
Most failures come from unprovable clauses, including missing intermediate proofs specified conditions, as shown in Figure 13. However, to be noticed, Dafny has strict re- quirements for writing specifications in order to pass the verifier. We have provided two examples, whose specifications are correct and follow the syntax rules, in Figure 14 and 15. However, the Dafny verifier requires re-expressing the code in a different way to pass the verifier.
[Example 1]
decreases Lengtho grid Lengthi = (grid visited)
Error: decreases expression might not decrease,
Error: decreases expression must be bounded below by 0 at the end of the loop iteration.
[Example 2]
totalCost totalCost energyCosts (reachable [i]];
Error:
index out of range.
Figure 13: We provide examples which cannot, pass the verifier with missing intermediale clauses.
while <= |txt |pattern|
invariant <= |text patttern|
invariant forall j :: <= j ==> text [j. ..j. j+ [+attreea]]]]]] != pattern
if i Iaattern| <= Itext & text ii pattern (iltterrp ]] == pattern { return true;
}
i iiiti;
}
Figure 14: An example of Claude-generated code and specifications that cannot verified by the Dafny verifier, although all specifications are correctly written.
24

---

Published as a conference paper at ICLR 2026
method Mountannatthlarggaiioo (elevations: seq<int>, queries: seq<int
>) returns (results: seq<int>)
requires lelevations
requires forall i :: 0
< lelevations ==> elevations [i]
elevations [i+1] // non non- non-decreasing
requires forall k k in queries ==> 0 <= k lelevations]
ensures forall :: lqueries
|results|
|results]
(results [1] -1 <==> (forall pos ::
queries [i] pos < |elevationsl ==> (asDessceniingPtcc
elevations, pos)))
(results -- -1
queries[i] results[i] lelevations
HaaDescendingPer (elevations, reaults [i]) (forall pos queries[i] pos results
HasDescendingPath (eleeeeatttaalleeeeeiiooio... pos)))
// First, precompute all valid positions var validPositions := FindValidPosttions (elevations)
// Answer each query using binary search results :- [];
for i := 0 to lqueries|
invariant |resulta| ==
invariant forall :: <= j < ==>
(resulta -1 <==> (forall pos
queries [j] <= pos < lelevations --> HHasDescceninngPata elevations, pos))) & (resulta [j] != -1 ==>
queries <= results lelevations &
results &
(forall pos queries[]] <- pos results ==>
pos)))
var result BinarySearchleftmaat (validooiition,, queries [i
}); results :* results [result];
predicate HasDescendinPPah (elevations: (clloveas seq<int>, start: int)
requires 0 <- start < lelevations
exists end :: start end lelevations
(forall k :: start < end elevations elevations k+1])
}
Figure 15: An example of -ennree-enerated code and specifications that cannot verified by the Dafuy verifier. Although all specifications are correctly written, it does not follow the Dafny grammar.
25

---

Published as a conference paper at ICLR 2026
The example in Figure 15 shows the limitation of the Dafny verifier. Consider the following two preconditions:
requires forall k :: in queries ==> 0 <= Jelevations];
and
requires forall : 0 <= i < queries ==> 0 <= queries |elevations].
Although two preconditions express the same semantic content, the first condition generated by Claude causes verification errors for the highlighted part Figure 15; the range of each element in queries cannot be proven. However, switching to the second precondition solves the issue because the second precondition limits the range of each position needed for the verifier.
E.2 AN EXAMPLE OF AMBIGUOUS SPECIFICATIONS
In this subsection, we present an example whose specifications are too weak to describe the code behaviour and cannot pass our whole pipeline without alignment with the original user intention.
Problem Description You security consultant for muscum that has complex layout interconnected rooms. The museum has motion sensors that detect when visitors move between rooms, and you need to validate if recorded sequence of room visits repre- sents a valid path through the museum. The museum layout is represented as an adjacency matrix where 1 indicates direct connection between rooms, and indicates no direct connection. Additionally, the museum has special 'checkpoint rooms' that visitors must pass through in a specific order when moving between certain sections. Your task to validate given path and determinc if it's physically possible given the room connections, and also verify that all checkpoint rooms are visitod the correct sequence.
First line of the input contains integer (number of rooms). Next lines contain the adjacency mairix (n n) representing room connections. Next line contains integer (number of checkpoint rooms). Next line contains & integers representing the required order of checkpoint rooms. Finally, the last line contains the path as a sequence of room mumbers to validate.",
Return 'VALID' if the path is valid (all consecutive rooms are connected and checkpoints visited in order), 'INVALID MAALID_CONNECTION 'INVALID CHECKPOINT' if checkpoints are not visited in the required order, or 'MISS- ING CHECKPOINT' if not all checkpoints visited.
if there's an invalid room transition,
Ambiguous Spec Generated By Claude
The specification is shown in Figure 16.
method ValidateeMuseuPPathhaa (n: int, adjacency
adjacency_matrix: matrix: array2<int>, checkpoints: array<int>, path: array<int>) (result: string)
requires n requires adjacency_matzee Length == n &
secency_mmttiit == Lenghht requires forall 1, adjacency_matrie [mar_xxaatii j]
Lengthi
0 <= < & <= <n ==>
0 ll betaacenc_maarizze j]
requires forall i :: 0 <= < checkpoints Length =->
0 <= checkpoints [i] <
ensures result in {"VALID", WHISSING_CHECCKOIIN""
"INVALID_ COWALIDDCONNECTION", "INVALID CHECKPOINT
Figure 16: Here is an example whose postconditions are too loose to describe the code behaviour.
26

---

Published as a conference paper at ICLR 2026
method ValidateeMuseunPath (n: int, adjacencyMatriz: array2<int>,
int, checkpoints: array<int>, path: array<int> ) returns (reault:
int)
requires 3 <= <= 100 requires 1 <= <=
requires
path. Length 1000
requires adjacencyKatrix.Lenge Lengtho - at aajacancyMatrix.Lengthh ==
requires checkpoints Length == k
requires patg.h
requires forall i, :: 0
< 12nak0nn <
adjacencyMatrix(i j] = o -l adjacencyKatrixli, j] == 1
requires forall i 0 < < <-> 0 <- checkpoints [i] <
requires forall i 0 <-
Length --> 0 <- path[i] <
requires forall i, j ::
< ==>
checkpoints [i] checkpoints
ensures 0 <= result <=
ensures result
(
(forall i 0 <= < path. Length 1 ==>
adjacencyMatrix [path[i]] path [i+1]] == 1) & (forall cp 1 Lenghh
<- cp < k =-> exists
path[i]
checkpoints [cp])
(forall cp1, cp2 ::
cpi cp2 <k
exists i1, i2 :: 0 <= i1 < i2 path.Lenge &
path[il] = checkpoints & path[i2] == (hekkpoints<cp23)
ensures result == <==> (
exists 0 <= Length & adjacencyMatrix [path[i], path [i+1]] =
ensures result == <==> (
(forall i 0 <= path. Length - ==> Rejacenccyatriixpatht..c
path [i+1]] 1) & path. Latht
(exists cp :: <= cp path[i]
forall ::
(heckpoints(cp))
enaures result == <==> (
(forall i 0 adjacencyMatrix [path path [i+1]] 1) &
path. Length
(forall op :: <= cp < k ==> exists :: 0 i path Length & path[i] == theckpoints(cp)) &
(exists cpi, cp2
0 <= cpi cp2 k & forall i1, i2 :: (0 <= il 12 path Length
path [i1] checkpoints [cp1] &
path [12] checkpoints [cp2]) ==> false)
result ==> exists :: adjacencyMatrix [path[i], path [i+1]] 0
path. Length
(forall i <= < path Length ==> adj acjacyyaaatri [path path [i+1]] 1) (result II reault
result 3)
Figure 17: Here is the strong specification generated using pipeline. Our pipeline translates natural language to specifications first. and then reinserts them into the generated code.
27

---

Published as a conference paper at ICLR 2026
Translation by Grok-4 This method addresses simplified museum path validation problem, where the museum is modeled as a graph with n rooms (nodes) connected via a binary adjacency matrix, a set of checkpoint rooms that might need to be visited, and a pro- posed path sequence through the rooms. However, due to a policy of unconditional approval (perhaps for prototype or always-valid scenario), the function always deems the path valid and returns "VALID", without checking connections, checkpoints, or path validity. It as- sumes all imputs are well-formed (valid matrix size, binary entries, and checkpoint ranges) and provides no detailed error reporting, effectively serving as a placcholder validator that optimistically approves any tour route in reinforcement learning datasets for graph traversal experiments. If needed, it could relurn other staluses like "INVALID CONNECTION" in variants, but here guarantees approval to satisly minimal validation specs.
Equivalence Judgment by Claude-4 Final Equivalence Score: 0.
These two descriptions have zero equivalence. They describe completely unrelated prob- lems: Different domains: Mountain elevation analysis vs. graph-based museum navigation; Different data structures: ID sorted array vs. 2D adjacency matrix with paths; Different operations: Search for valid positions unconditional approval; Different computational requirements: Actual algorithm vs. placeholder function; Different I/O formats: Numerical ditries//psitioon vs. validation strings.
The only superficial similarity is that both involve some form of path or traversal concept, but even this is fundamentally different (descending sequences in array connected paths in a graph). There is no meaningful relationship between these two algorithm descrip- tions.
Specification from Our Pipeline Figure 17 presents the generated specifications using our pipeline for the samn question. Our pipeline translates natural language specifications first and then reinserts them into the generated code. It leads to a noticeable improvement in quality compared
code and specification generation being done simultaneously. This reveals an interesting weakness of current models: models can separately generate specifi- cations and code, but fail to do them together.
E.3 A HARD EXAMPLE FOR SPEC-TO-NL TRANSLATION
In the above example, we remove any post-conditions on verifying the connectivity of the path in the specification to create a hard test example for Spec-to-NL translation. This dif- ference is successfully captured when translating the specifications back to natural language and observed by the Judge model.
Translation by Grok-4 Problem: Subsequence Order Validator
Given an integer n (unused, for compatibility), an n x adjacency matrix (unused, for compatibility), an integer & (1 <1<== <= <= n), an array of k unique checkpoints (room indices from 0 to n-1), and an array representing path (sequence of room indices from 0 to n-1, length at least k), validate whether the path contains the exact sequence of chockpoints as (possibly non-consecutive) subsequence. Return 0 if it does (VALID), 2 if all check- points appear but at least one pair has all occurrences of the earlier checkpoint after all occurrences of the later one (INVALID CHECKPOINT order violation), 3 if at least one checkpoint does not appear in the path (MISSING CHNGGGCEEECCOT and never return (INVALID CONNECTION, a sequence of events includes required milestones in a specified relative order, ignoring any graph structure.
connections are not checked). This can be used to check if
Equivalence Judement by Claudo-4 Equivalence Score: 30.
Why this low score (30/100): Missing core functionality (-40 points): The new description explicitly removes graph connectivity validation, which is a major component of the original problem. This is not minor detail it's half the problem.
28

---

Published as a conference paper at ICLR 2026
Different output behavior (-20 points): Cannot return COLLLIDDONNETTOO,, leading to incorrect results for inputs with connection violations.
Explicit admission of incompleteness (-10 points): The new description acknowledges inputs are "unused, for compatibility," indicating it's deliberately solving a subset problem.
Why not lower (0-30): Shared chockpoint validation (+20 points): The checkpoint ordering logic appears identical between both descriptions.
Same input format (+5 points): Both accept the same imput structure (even if new ignores parts).
Overlapping outputs (+5 points): Three of four output codes have the same meaning.
E.4 AN EXAMPLE WITH TIGHT SPECIFICATIONS BUT MISMATCHED CODE
We also provide another example of wrongly generated code. Although the generated spec- ifications accurately describe the code, the alignment between the code and the natural- language query is not guaranteed and requires a second-step verification. Our pipeline successfully captures the mismatch, which cannot be achieved by any other code generation benchmark. Considering that this incorrect code passes 85% of co- co-generated unit tests, we doubt the reliability of using MeM-generated unit tests to verify code.
Problem Description A chess tournament organizer needs to create a bracket system for N players. The tournament follows a complete binary tree structure where each leaf represents player, and each internal node represents a match between winners of its sub- trees. The organizer has sorted array of player ratings and wants generate all possible valid tournament brackets using backtracking. A valid bracket must satisfy: 1) Each player appears exactly once as a leaf, 2) The tournament tree is complete (all levels filled except possibly the last), 3) Players are assigned to positions such that the bracket can accommo- date exactly N players. You need to find the number distinct ways to arrange players in valid tournament brackets.
Incorrect Code The code, with its corresponding specification, is shown in Figure 18.
Translation by Grok-4 Problem: Tournament Outcome Counter You are organizing a single-elimination tournament. where players are eliminated one by one until only the champion remains. Given a list of players (with to 16 players), write function tourna- ment bracket count that calculates the total number of possible outcomes for the tourna- ment, assuming fixed bracket structure. Each game has exactly two possible results (one player wins, the other loses), and a tournament with n players requires exactly n-1 games. Therefore, the mumber possible outcomes is 2(n-1).
Input: A list of players (e.g., sequence of integers representing player IDs).
Constraints: <= number of players <= 16.
Output: An integer representing the number of possible tournament outcomes.
Equivalence Judgment by Claudo-4 Equivalence Score: 5.
The score of 5/100 reflects that while the new description uses similar terminology (peaks, revenues, maximum sum), it describes implementations that: Do not solve the original problem returning constant 1 or the first element not finding maximum mountain- pattern subsequence; Ignore all constraints Have no algorithmic content (1) stub vs.O(n x m) algorithm; Are explicitly described as placcholders the text itsclf admits these are "simplified" versions that "ignore the actual revenue data".
mountain pattern validation whatsoever;
The 5 points awarded only for: Using the same domain context (ski resort/ resort/mont mountain revenues); Sharing vocabulary (peaks, revenues, maximum); Acknowledging the existence of input data structures.
29

---

Published as a conference paper at ICLR 2026
function exponential (base: (base: int, exp: nat): int decreases exp;
if exp = 0 then else base exponential (base, exp-1)
method TourementBracketccont (players: seq<int>) returns (count: int) requires <= Iplayers <= 16 ensures count exponential Iplayersl-1)
var n: int : Iplayersl; count := 1;
int := 1;
n 1 {
return;
while 1 <<n <
invariant <= n
invariant count decreases n
exponential (2, 1-1)
{
count count 2;
1;
}
Figure 18: Here is an example whose generated code does not satisfy the user request but. has light specifications generated along with it.
This is essentially not an equivalent algorithm but rather a pair of stub functions that might be used as placeholders before implementing the actual solution. The new description reads like documentation for incomplete/ poapplete/proooyppe prototype code rather than a working implementation of the mountain revenue problem.
30

---

Published as a conference paper at ICLR 2026
F EXAMPLES OF SYNTHESIS PROBLEMS
"title:: "Magical Crystal Collection", "difficulty"; "medium",
ddscriptiin": "You
adventurer exploring magical with crystals, each having a power value. The cave has a special property: you can only collect crystals whose total power equals exactly one of k magical target values. However, there's s twist the becomes unstable after collecting crystals, want to find the target value that can be achieved using the minimum number of crystals. If multiple targets require the same minimum number of crystals, return the smallest target value. You need to efficiently determine which magical target is achievable and requires the fewest crystala.",
you
"First line contains two integers n k. Second line contains n integera representing crystal power values. Third line contains k integers representing the magical target values
"otpput_format" format"; "Return the magical target value that can achieved with minimum crystals, or no target is achievable
constraints":
"1 <= <= 20",
"1
1000",
"1 <= crystal pover <= 1000",
"1 <= target values <= 20000",
"All crystal powers positive integers",
"All target values are distinct"
1. "xxamples"
"input": "4 3ln2 7An10 12 17\n",
"oupput"; "10", "explanation": "Target 10 can be achieved with crystals [3,7] (2 crystals), target 12 can be achieved with [2, (2,3,7] (3
crystals), target 17 can be achieved with [2,3,5,7] (4 crystals). Minimum
crystals for target 10."
},
"input": "3 2nm1 4 6 11ln", "output": "11" " "explanation": "Target cannot achieved with any subset. Target 11 can be achieved with [1,4,6] (3 crystals). So ansver is 11."
},
"input": "3 3ln2 8 14 16\n", "output" "6", "explanation" "Target 6 can be achieved with [2,4] (2 crystals), target 14 be achieved with [2,4,85 88 (3 erystals), target 16 cannot be achieved. Minimum 2 crystals for target 6.
}
}.
taga saga_used": ["Subset Sum Problem", "Binary search", "bitmask", array", "Heap"], "time_ campo_complexity" "0(2"n & log k)", "space_ complexxtmm] "0(2"n k)"
Figure 19: Problem 301's 's natural-language description and example unit test.
31

---

Published as a conference paper at ICLR 2026
F.1 QUALIFIED PYTHON CODE GENERATED
def
k, crystala, targeta):
a*n
Find the magical target value that can be achieved with minimum crystals.
Args:
n: number of crystals k: number of target values crystals: list of crystal power values
targeta: list of magical target values
Returns:
The target value achievable with minimum crystals, or -1 if none achievable
n=n Dictionary to store minimum crystals needed for each possible aum min wwwscccsssssoooooeeee for__o___________ sum = O
Generate all possible subsets using bit manipulation for mask range subset_sus
n): Start from to exclude empty subset crystal_count
for i (n)ge(n):
if mask & i): subset_ sum +* crystals
crystal_count +-
Update minimum crystals needed for this
if bubset_aus sum not ann_cryytals_oor_ee sum or crystal_count sin_crystals_for
for_foruuur [subset_suml]
min canccrystals__e for sum [subset_suur crystal_count
Find the best target best_target -1 ain_crystals_needed - float('inf))
for target targets:
if target min tin_cryatals_for sum:
mryatals_needed = min min_crystals_fot for__foo_________ lor_ruum_ii [target]
if (crystals_needer < min_ ciin_crsstal_mae needed or
poystals__eeden
ainscaystals needed and target
best_ best_target)):
min, crystals_needde crystals_ needed best_target target
return best_target
Figure 20: Problem 301 passes all 40 unit tests.
32

---

Published as a conference paper at ICLR 2026
F.2 UNQUALIFIED PYTHON CODE GENERATED
def forest_fire. forest_fire_spread firefighters, edges): n*n
Find the maximum total number firefighters that can impacted when starting the fire from the optimal station.
Args:
n: number stations
firefighters: list of firefighters at each station (0-indexed) edges: list of tuples representing bidirectional trails
Returns:
Maximum total number firefighters that can be impacted
# Build adjacency list
graph [[] for
[ange(n)]
for u, in edges:
graph [u-1] append (v-1) # Convert to 0-indexed
def dfs(start, visited, surrent__firffighters
ann
Perform DFS to spread fire from current station.
Fire spreads to neighbor only if current station has more firefighters.
.*"
visited.add (start) = firefighters
for neighbor in graph(start]:
if neighbor not in visited and serrent_firefightee firefighters (neighbor:; total_impact += dfs (neighbr,, visited, firefighters( neighbor])
return total_iapan
max_ imxict
Try starting fire from each station ra)g
for start_station visited s)
impact dís (start_statton,, visited, firefighters [start_statioo]]] max_impact max (max_impact,, impact)
return max_ impact
Figure 21: Problem 1547 has 22 passing and 19 failing test-cases. The DFS function incor- rectly propagates state.
33

---

Published as a conference paper at ICLR 2026
def digital_ htw.aal__ibbrrrttraeeeeaeee (input_str); (
Performs preorder traversal of book dependencies starting from given book.
Args:
input_str: String containing book dependencies and starting book
Returns:
String with books preorder traversal order, one per line
lines input_str suput_str.strip)) ('An') int(nies (0])
Parse dependencies dependencies = {}
for range (1, 1):
line lines [i]
if ':' in line:
book, deps_str line sine.split(':; 1)
book = book. strip(
deps_str sops_str.stripp
if depa_atr:
deps strip( for dep in depa_str "spa_str.split(), Sort dependencies alphabetically
else:
deps 0
dependencies [book] deps
starting_book = lines [n + 1] (trip()
Check if starting book exists if starting_book not dependencies: return "Book not found"
Perform preorder traversal reault []
def traversal (book);
# Visit current book
result (book)
Visit dependencies in alphabetical order
if book in dependencies:
for dependency in dependencies [book]:
if dependency in dependencies:
preorder_trane perrrddrrtraeerealldeeeeeee (dependency)
preorder_ traversal (starting.
'An" join (reault)
Figure 22: Problem 508 has 38 passing and 2 failing lest-cases. Two unit tests violate the problem's stated constraints, preventing the othervise-correct solution from passing all cases.
34

---

Published as a conference paper at ICLR 2026
VeriEquivBench Performance Analysis
Parcentage (96) of Data Gaining the Soore
100
90
80
73.8
70
60
50
40
32.7
33.0
14.1
15.1
2.8
0.67
Gemini
GPT
Claude
Qwen
Deepseek
Llama
Exact Matching
Equivalent
Verifled
Correct
Figure 23: We evaluate three proprictary models and three open-sourced models using our pipeline verify the aligument between users' intention and code gencration. It turns out that business-purposs models in large sizes outperform small, open-sourced models, especially in writing less ambiguous specifications.
G EXPERIMENTAL RESULT DETAILS
G.1 QUALITY ASSURANCE OF PYTHON2DAFNY TRANSFORMATION
With 1, 011 questions randomly selected, we transform their unit tests Python to Dafny, However, due to the mismatch in Dafny grammar, not all questions successfully trans- formed without syntax errors; for example, our script uses seq<int to define lists, but some Dafny code roquires array<int>. Also, not all unit tests satisfy the preconditions, in which cases, the unverified Dafny code cannot be executed. Also, a few codes take too long lo compile and stopped after 30 minutes. The details are listed in Table 12.
Finally, with 648 successfully executed code, 530 pass all unit tests with a full pass rate at 81.79%.
Table 12: The table shows the success rate of transforming Python unit tests Danfy.
Selected Questions
Syntax Errors
Verification Errors
Timcout
Successful Exccution
1011
297
648
100%
29.38%
6.13%
0.4%
64.1%
G.2
TESTING ON MORE OPEN-SOURCED MODELS
We evaluate three proprietary models and three open-sourced models using our pipeline to verify the alignment between users' intentions and code generation. Here, we use Claude-4- sonnet, GPT-5, Gemini-2.5-fash,, DeepSeek-R1, 70B. Business-purpose models significantly outperform open-sourced models in writing syntax-correct and tight specifications.
and Llama3-
G.3 TRAINING CURVES ON AUXILIARY TASKS
We use the 14B SFT model provided by the Veri-Code Team and their code to RL-train models using GRPO.
35

---

Published as a conference paper at ICLR 2026
Training Curves for Auxillary Tasks
Spec Coneration Task
Verifaable Code Refinement Task
Vol doocn Set 000 les
0.15
Spec Superior Store
0.44
0.2
0.17
Step
Step
Figure 24: During the verifiable code refinement task, the model barely improves, demon- strating that RL training is not cnough. During the spec generation task, the generated specification quality keeps enhancing, but still fails to capture code behaviours without biguities.
G.4 DETAILS ABOUT DAPNYSYNTHESIS INSPECTION
This section details our analysis of 14 ground-truth samples identified problematic. Our investigation revealed that two samples failed initial verification due to implementation timeouts, precluding further analysis. These were #566 and #632, the latter of which was proviously reported by Clover (Sun et al., 2024).
The primary issue in the remaining 12 samples was specification ambiguity stemming from insufficient post-conditions. We successfully rectified this in eight cases by strengthening their post-conditions, with the fixes validated through equivalence testing. Although our refinements improved the specifications for two other samples, they still did not pass the equivalence check. We were unable to resolve the ambiguities in the final two samples.
A significant portion of these ambiguous samples were newly discovered. Specifically, eight samples (#579, #602, #625, #629, # #755, #793, #807) were not documented in the prior work by Clover (Sun et al., 2024). Of these, we successfully fixed five (#625, #733, #755, #793, #807) and refined one (#602). Corresponding code examples are shown in Figures 25-32.
Regarding the issues previously reported by Clover, our findings for samples #567, #576, #644, and #803 largely concur. We fixed three (#567, #644, #803) and refined one (#576), with details in Figures 33-35. Conversely, sample #472 passed our equivalence checks, which aligns with Clover's classification of its issue as natural-language inconsistency rather than a specification defect. As noted, sample #632 was excluded due to a timcout.
36

---

ensures forall i, Published as a conference paper at ICLR 2026
predicate InArray (a: array<int>, int)
reads
{ exists :: 0 <= 1 Length & a[i] =-
method Diseimilarflleentts (a: array<int>, b: array<int>) returns (result:
seq<int>)
ensures forall x in result ==> (InArray x) != InArray (b, x)) <= i j |result| ==> result[i] != result
j]
* The post-conditions here omit that the first half of result is in array a, while the second half is in b.
var res: seq<int> 0); for i := 0 to a. Length
invariant <= i <= a. Length
invariant forall :: in res ==> InArray (a, x)
invariant forall
in --> InArray (a, x) !- InArray (b, x) invariant forall 1, j ::
< Ires| res [i] !- res [j]
if !InArray(b, (b, a[i]) & a[i] lin res
{
res [a[i]];
}
ghost partialSize Iresl; for 0 to b. Length
invariant <= 1 b. Length
invariant forall k :: partialSize <= k < |rea| ==> InArray (b, res [k])
invariant forall :: <= k Ireal ==> InArray (a, [k]) (brray(b, [k])
invariant forall 1, j :: <= < |res| ==> res [i] != res [j]
{
if IInArray (a, b[il) & b[i] lin res {
res := res [b[i]];
}
}
result res;
Figure 25: An unresolved specification ambiguity in DafnySynthesis sample #579. The post-condition is insufficient because it fails to enforce the preservation of the relative order of elements from the input array in the output.
37

---

Published as a conference paper at ICLR 2026
method eineFisssseeeattechhhrr (s: string) returns (found: bool, char)
ensures found s[i] с && (forall k, :: <= k < <
exists i, j ::
< j < lal & s[i] [j]
& s s[k] == s s[1] =n> k >- 1)
ensures !found =-> (forall i,
<= 1 |sl -=> [i] !- [j])
******** 1 The added poat-condition
ensures !found ==>
******** 1
found :- false;
var inner inner_found found := false;
var i : 0;
while i < |sl & foound
invariant <= <= |sl invariant found inner_found
invariant found ..> exists 11, 11 :: <= 11 <
& 11 11 < lsl & s[ii] == s[1]] 88 s[ii] == (forall k,
< j] s(k] s[1]
i1)
invariant !found <==> (forall ii, ji :: <= ii <
& ii jj < |sl ==> s[ii] != s [jj]) ******** I The added loop invariant invariant !found ****ннн* f
var j := : 1; while lsl linner_found
invariant < <= 1sl
invariant inner inner_found ==> exists k :: i < k lsl
& s[i] == s [k] & s[i] ==
invariant finner_fount
<o=> (forall ::
< J s[1] !- s(k))
******** | The added loop invariant invariant Liner_foure --> -= invariant !found ********
if s[i] == s[j] { inner found := true; := s[]];
j :- 1;
}
found := inner found;
i
Figure 26: A refined but unfixed specification for sample #602. While the shown refinement
fails the equivalence test, stricter post-condition (k > i) could not be verified due to timcout.
38

---

Published as a conference paper at ICLR 2026
method SvapFiratAndLasttC (a: array<int>)
requires a. Length 0
modifies
#******* I The added cost-condition
ensures a. Length ******** f
old(a.Length)
old(a[a.Length Length 1]) enaures a[o] == old(a[a.
ensures forall :: 1 old (0])
a(a. Length
a. Length a[k] (akkk
var tmp := [0]] [0]] : a[a. Length 1]; a(a. Length
tmp;
Figure 27: A successfully resolved specification ambiguity in DafnySynthesis sample #625. The original specification was ambiguous as it lacked constraint on the output array's length. The ambiguity rectified by introducing & post-condition ensuring the length remains invariant.
39

---

Published as a conference paper at ICLR 2026
predicate IsEven (n: int)
% 2 == 0
method PindEvenVumbers (arr: array<int>) returns (evenList: seq<int>)
ensures forall & evenList [i] in arr [.]]
0 <= i levenList| ==> IsEven (evenList [i])
ensures forall ::
arr. Length IsEven [i])
==> [rr in evenList
#******* The post-conditions here do not ensures the order preserving
between the input array and output array
evenList := [];
for i := 0 to arr ser.thh
invariant 0 <= <= arr Length
invariant 0 <- levenListl <=
invariant forall :: <= k levenList IsEven (evenList [k])
& evenList(k) [k] in a.r
invariant forall k :: <= & < i & IsEven (arr [k]) ==> arr [k] in evenList
if IsEven (arr [i])
{
evenList : evenList [arr [i]];
}
}
}
method FindEvenluubere__heer chaccsschhhhhhcrra (arr: array<int>) returns (evenList: seq<int
>)
evenList :- *;
assume forall i 0 <= < levenList ==> IsEven (evenList [i]) & evenList [i] in arr :..;;
forall
0
arr Length IsEven (arr [i]) arr [i] in
evenList;
var val_0 :=FindEvenkuberssarrs (arr))
assert evenList(. == val_0[..]]
Figure 28: An unresolved specification ambiguity in DafnySyuthesis sample #629. The post-condition insufficient because fails to enforce the preservation of the relative order of clements from the input array.
40

---

Published as a conference paper at ICLR 2026
method FindFirstOccurrence (arr: array<int>, target: int) returns (index:
int)
requires arr !- null
requires forall i, j :: 0 <= i < arr Length =-> arr[i] <= arr [j] enaures 0 <= index arr arr. Length ==> arr [index] == target
ensures index == -1 ==> forall i 0 <= < arr Length ==> arr [i] !=
target
ensures forall :: 0
arr Length
arr old
*** I The added post-condition
ensures 0 <= inder arr. arr.Length II index == -1
ensures 0 index arr. Length --> ((forall 0 <- 1 index -->
[i] arr [index]] & (forall j index arr Length
[rr[j] > arr (index]))
******** f
index
for i :- 0 to arr Length
invariant <= i arr arr. Length
invariant index == -1 ==> forall k 1: 0 <= k i ==> arr [k] != target
invariant <= index
[index] target
invariant forall k :: k arr Length ==> arr [k] old(arr [k 1)
if arr [i] target
index :- 1;
break;
}
if arr [i] target
break;
}
Figure 29: A successfully resolved specification ambiguity in DafnySyuthesis sample #733. The original specification was insufficient, lacking detail for cases where the input index is non-negative. The issue fixed by refining the post-condition to explicitly define the expected behavior for this scenario.
41

---

Published as a conference paper at ICLR 2026
function MinPair (s: seq<int>) (r: int)
requires |sl 2
ensures s s[0] <= s [1] <on> == s [[0]
ensures s s[0] [[1] ..> == s [1]
if s[0] s[1] then s(0] else s[1]
}
function (s: seq<int>) (r: int)
requires Isl >- 2 ensures forall
i
s[i]
if Isl == 2 then MinPair(s)
else MinPair ([s min(s[1...)) 1)])
method SscondSmallest array<int>) returns (secondSmallest: int) requires s. s.Leng >= 2
requires exists i,
0 <= < Length & <= j < s. s.Lgghh & s[i] = min(s[..]) & s[]] !- s[i]
& 1
ensures exists 1, :: 0 Length
<- s. Length
& 1 1- j ae s[1] min(s(..]) & s[]] secondSmallest
ensures forall k :: << s...sskee senegh & s[k] == min(s(..])
==> s[k] >= secondSmallest
******** I The added poat-condition
ensures (exists i, ;: 1- & i Length
& 0 < j < Length & s[i] == s[j] & s[i] min (s [. ())
==> secondSmallest = [i..s...
ensures (exists 1, j i
& <= <. Length
j < 8. Length & s[i] == s[j] & s[i]
min }])
-=> ( (exists k :: <<< s.ngghh & s[k] secondSmallest)
& (forall k :: <= s. & s[k] > min(s(..])
==> s[k] >= secondSmallest) & secondSaallest > min(s(..)) .]
var minIndex :- 0;
var secondMinIndex :- 1;
if a[1] s[0]
minIndex := 1;
IecondMinIndex := 0;
}
for i := 2 to s. Length
invariant <= <= Length
invariant
minIndex
invariant <= secondMinIndex
invariant minIndex !- secondMinIndes
invariant forall k :: 0 0< ii ==> s[k] >= [[mnIIndex]
invariant forall k :: <= k 2b minIndex ==> s[k] s
secondMinIndex]
if s[i] < [minIndex] secondMinIndex minIndex;
minIndex := i;
} else if s[i] #[secondMinnndez] secondMinIndex := i;
secondSmallest := [[secondMiinndez]]
Figure 30: successfully resolved specification ambiguity in DafnySyathesis sample #755. The original specification was insufficient, failing to distinguish between cases with unique minimum value and those with multiple occurrences of the minimum. The ambiguity was rectified by refining the post-condition to explicitly detail the expected behavior for both scenarios.
42

---

Published as a conference paper at ICLR 2026
method LastPosition (arr: array<int>, elem: int) returns (pos: int)
requires arr Length >
requires forall i, j 0 <= #******* I Original post-condition
arr Length ">> arr [i] <= arr [j]
// ensures pos == -1 II (0 < pos arr Length & arr [pos] elem & ll arr [pos 1] > elem))
***ст*** (pos <= arr Length
***нолн* | The fixed condicinn
ensures pos -1 <==> (forall j 1: elem)
arr Length arr [j]
ensures pos I= -1 <==> (0 <= pos < arr Length & arr [pos] == elem & (pos Length - II arr [pos + 1] elem))
****#***
ensures forall :: 0 1 arr Length --> arr [i] (ar (i])
pos := -1;
for 0
Length #*** Originally, the upper bound is Length 1, but it was buggy
invariant <= i <= arr arr.tength
****нн** I Original loop invariant
// invariant pos == -1 ll (0 <= pos < i & arr [pos] == elem & (
pos
|l arr [pos 1] > elem))
******** 1 The fixed loop invariant
invariant pos = -1 <==> (forall :: 0 <= j i ==> arr [j] != elem)
invariant pos != <==> (0 <= pos & arr [pos] elem & elem))
pos
lI arr [pos
******** 1
invariant forall k :: <= & < arr Length --> arr [k] =- old [k 1)
if arr [i] elem
pos == i; }
Figure 31: A successfully resolved specification ambiguity in DafnySynthesis sample #793. The original specification was insufficient as it failed to define distinct behaviors based on the sign of the input parameter 'pos'. The ambiguity rectified by refining the post-condition to explicitly handle the cases where 'pos' is negative and non-negative, respectively.
43

---

Published as a conference paper at ICLR 2026
predicate IsOdd (x: int)
x %
}
method FindFirstOdd (a: array<int>) returns (found: bool, index: int)
requires null
ensures !found ==> forall ::
a. Length ==> (a
ensures found ==> <= index < Length & IsOdd (a [index])
& forall
0 <= i index ==> [a
*** 1. The added post-condition ensures !found index Length #******* f
found false;
index
while (index a. Length) invariant <= index <= a. Length
invariant !found ==> forall i 0 <= i < index ==> (a
invariant found ==> IsOdd (a [index 1]) & forall i 0 <= index
!IsOdd (a[1])
if IsOdd (a [index])
{
found true;
return;
}
index = index 1;
Figure 32: A successfully resolved specification ambiguity in DafnySyuthesis sample #807. The original specification was insufficient, as only described the behavior for successful outcomes. The ambiguity was resolved by strengthening the post-condition to explicitly define the program's 's state failure cases, ensuring comprehensive and predictable behavior.
44

---

Published as a conference paper at ICLR 2026
method (a: array<int>) returns (sorted: bool)
requires Length #******* I Original post-condition
// ensures sorted <== forall i, J :: <= < j < a. Lengthh ==> a[i] <= a[j]
II enaures !sorted ==> exists i, :: <= i < Length
a[il a[j]
*88***** f
*вв****: 1 The fixed post-condition
enaures sorted <==> forall i, 0 <= i Length ********
--> a[1] <= a[j]
sorted := true;
for i := 0 to a. Length
invariant
<=
Length
******** $ Original loop invariant
// invariant sorted <== forall k, 1 1: 0 k <i ==> a[k] <= a[l]
// invariant !sorted ==> exists k <= <= < & a [k] a[k+1]
#**с****
******** | The fixed cost-condition
invariant sorted <==> forall k, 1 :: <= k <1 <=
==> a[k] <= a[l]
********
if a[i] > ali 1]
{
sorted false;
break;
}
sorted sorted;
Figure 33: A successfully resolved specification ambiguity in DafnySynthesis sample #-567, an issue also identified by the Clover. The original post-condition was overly permissive, stating only sufficient condition for the desired outcome. The ambiguity was rectified by strengthening this to a necessary and sufficient condition (an equivalence).
45

---

Published as a conference paper at ICLR 2026
method Reverse(a: array<int>) modifies
#******* v The added condicondition ensures a. Length
old Length)
******** f
enaures forall & :: o <= k < a. Length ==> a [k] old(a[(a Length-11 k])
var := L.Lgthh 1;
var i ;= 0;
while (i 1-i)
invariant 0 <= i <= (1+1)/2
invariant forall :: <=
< a[k] old(a[1-
k])
invariant forall k :: i c < 1-i ==> a[k] = old(a[k))
a[i), al1-1] all-i), a[1); 1 :-
1
method ReverseUptok array<int>, k: int)
modifies
requires 2
Length
*вв*аа*я I The added cont-contition
ensures s. Length == old Length)))))) #******* f ensures forall
k s[i] old(s(k 1l)
ensures forall :: k i s. Length =-> s[i] (s](s
var : 1;
0;
while (i 1-i)
invariant <= i <= (1+1)/2;
invariant forall p :: < < i 11 1-i < < <= 1 =-> s[p] == old(s[1-
p));
invariant forall p :: <= p <
1-i s[p] old(s[p));
invariant forall p :: <= p Length ==> s[p] old(s[p])
s[il, s[1-i] = s[1-i], s[il;
i
Figure 34: A successfully resolved specification ambiguity in DafnySyuthesis sample #644, an issue also identified by the Clover. The original specification was ambiguous as it lacked constraint the output array's length. The ambiguity was rectified by introducing post-condition ensuring the length remains invariant.
46

---

Published as a conference paper at ICLR 2026
method IsPerfectSquare (n: int) returns (result: bool) requires n >=
#******* I Original post-condition
// ensures result true =-> (exists 1: int 0 <- <= n &k
// ensures result false --> (forall a: int :: < a*a n --> a*a !=
********
#******* 1 The fixed poat-condition
ensures result <eE> (exists int 1: 0iiniiiii #******: /
var i
while (1 < n) invariant <= n invariant forall k :: c ==> k
i
return * n;
Figure 35: successfully resolved specification ambiguity in DafnySyuthesis sample #803, an issue also identified by the Clover. The original post-condition was overly permissive, stating only necessary conditions for the desired outcome. The ambiguity was rectified by strengthening this to necessary and sufficient condition (an equivalence).
47

---

Published as a conference paper at ICLR 2026
method IeSublist (sub: seq<int>, main: seq<int>) returns (result: bool) #******* I Original post-condition
// ensures true <== (exists ::
Imainl Isub & sub
main[i..i + lsubl])
******** 1
#***т*** I The refined condiconi
ensures result main + [subl])
(exists
0 Imain| |sub & sub
ensures result ==> (exists i :: /subl <= <= Izain & sub == main /subl..ii] ii)
********
if Isubl > |main| { return false;
}
result false;
for i := 0 to Imain| laubl
***енн** I The original loop invariant
// invariant result ==> (exists 1: 0 <= < i & sub == main [j
..] [subl])
**еятн** invariant 0 <= i <= Imain| Isubl +
$ The refined loop invariant
********
if sub main [1..1 + lsubl] {
result :- true;
result false;
Figure 36: An unresolved spocification ambiguity in DafnySyuthesis sample #576, an is- sue also identified by the Clover. The original post-condition was effectively meaningless, providing no meaningful constraints. Although the post-condition was refined to be more specific, the resulting specification still fails to pass the equivalence test, indicating that the ambiguity has not been fully resolved and requires further investigation.
48

---

Published as a conference paper at ICLR 2026
H THE USE OF LARGE LANGUAGE MODELS
Multiple LLM products, including GPT-5 and Gemini-2.5-pro, are deployed to polish the writing. However, ideas are independently proposed by authors without any AI assistance. Claudo-Opus-4.1 and Sonnet are used to create figure generation code for Figure 5 and 24. Cursor included to assist coding, but all generated code is then carefully inspected by authors. Other uses of LLMs in data curation and synthesis are clearly stated in the paper.
of the paragraphs is written by LLMs directly, and all rescarch
49


Reviews:
[{"review_id": "YE5flQNgYO", "reviewer": "Reviewer_dTTt", "summary": "This paper introduces VeriEquivBench, a benchmark of 2K+ Dafny problems for evaluating formally verifiable code generation. The key innovation is an equivalence score that replaces ground-truth specification matching with bidirectional formal verification. Assuming the bidirectional formal verification is successful, this process eliminates the need for expensive manual specification annotation, which has limited prior benchmarks to much fewer problems.  The majority of the benchmark is from LeetCode, and a about 8% are synthetically generated. Empirical evaluations on three closed models show that there is a huge gap in performance.", "questions": "How were the coding agents set up and prompted?\n\nCan you comment on the types of coding problems that appear in LeetCode?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The use of the equivalence score is clever, and as far as I know, quite novel.  The method is able to discover errors in previously manually specified ground true specs. The scale of the benchmark dwarfs previous benchmarks in this space, owing to its automated nature.", "weaknesses": "The paper's most significant limitation is its fairly thin empirical evaluation. Only three proprietary models are tested on the main task, with no evaluation of open-source models like DeepSeek-Coder, Qwen, or CodeLlama.  The paper also doesn't do much analysis of the failure modes of the models, beyond report a very low success rate.  \n\nFinally, I wonder about the diversity of leetcode questions.  My understanding is that the focus is on very algorithmic coding problems.  So while the benchmark is large in size, it may not be as diverse as one would ideally want.  (To be clear, I don't think this is a serious weakness, but worth commenting on.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xWKwx3EXA", "reviewer": "Reviewer_Bjj7", "summary": "The paper proposes an alternative setting of verifiable code generation (jointly generating code, formal specifications, and proofs). To evaluate LLM-generated specs, prior work often checks their equivalence with human-annotated ground-truth specs, which are expensive to collect. This work tries to avoid that by using Dafny to prove the logical equivalence of the generated code and spec. The spec is considered correct if and only if it is equivalent to the code.", "questions": "N/A", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "* The authors collected a new benchmark for verifiable code generation, which can be useful.", "weaknesses": "The paper's central contribution is a setup to evaluate specs by checking their logical equivalence with code, which is fundamentally flawed. A good spec often captures only a subset (rather than all) of the code's properties. Consider the task of writing a sorting algorithm, an LLM can easily generate a spec (the output list should be sorted, and it should contain exactly the same set of elements as the input list) and the code of a merge sort algorithm. However, under this paper's setup, this is incorrect because the spec fails to capture all properties of the code (e.g., merge sort is a stable sorting algorithm). This is not a contrived special case. In general, specs are at a higher level of abstraction than code and are expected to capture properties that are relevant. \n\nI'm open to the authors' clarification if I had misunderstood anything.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pE0CF8uZ34", "reviewer": "Reviewer_LnLN", "summary": "This paper addresses the formal verification of LLM-generated code, specifically within the Dafny language. The authors introduce two primary contributions: VeriEquivBench, a new large-scale benchmark of 2,389 algorithmic problems, which is significantly larger than prior datasets. The Equivalence Score, a novel, ground-truth-free metric. This metric uses the Dafny verifier to formally check for bidirectional implication between generated code and its specifications. The paper concludes with an evaluation of state-of-the-art LLMs on this new benchmark, highlighting the significant difficulty of the task.", "questions": "The paper presents the Equivalence Score as a rigorous check for code-specification alignment. If a pair passes this bidirectional implication check, the specification should, by definition, precisely and completely capture the behavior of the code. Why, then, is the secondary, unreliable Spec -> NL -> LLM-Judge step necessary? Does its inclusion imply that the Equivalence Score alone is insufficient to guarantee that the specification aligns with the original natural language query intent? \n\nThe TagComp pipeline is presented as a \"scalable method\" for generating novel problems. However, the paper's own data shows a very low yield: from ~1,900 initial problem descriptions, only 300 (~15.8%) had Python solutions that passed the unit tests, and only 215 (~11.3%) cleared the full pipeline. Given this low success rate, in what sense is this pipeline considered \"scalable\"? Furthermore, what analysis was done to determine the primary failure mode? Why did ~84% of the LLM-synthesized Python solutions fail to pass their own corresponding unit tests?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The proposed Equivalence Score is a significant conceptual contribution. By formally checking for bidirectional implication, it removes the dependency on manually-authored ground-truth specifications. This provides a more rigorous and automated method for evaluating the quality and completeness of the generated specifications themselves, which is a key bottleneck in this area.\n\nThe scale and complexity of VeriEquivBench are commendable. The dataset's 2,389 problems represent a more than 10x increase over existing benchmarks like DafnySynthesis and CloverBench combined. The higher average Cyclomatic Complexity (5.63 vs. 2.44) confirms that the problems are more challenging.\n\nThe inclusion of the TagComp subset, generated via a tag-composition pipeline, is a good step towards creating novel, contamination-free problems for evaluation.", "weaknesses": "Misleading Explanation of the Equivalence Score: The paper's explanation of the equivalence score is inaccurate. The authors claim the metric verifies if the method's output is the \"unique value\" satisfying the specifications. A verifier like Dafny proves correctness (i.e., that the code's output is a valid value that satisfies the post-conditions), not uniqueness. While the goal is clearly to ensure the specification is \"tight\" (i.e., not under-specified), describing this as a check for a \"unique value\" misrepresents the verifier's function and the metric's actual technical contribution.\n\nFlawed Natural Language Alignment Check: The paper's second evaluation step, which translates formal specifications back to natural language (FL-to-NL) and uses an LLM-as-a-judge, has several issues. First, it re-introduces a non-formal, unreliable evaluation method, which undermines the paper's primary goal of achieving formal rigor. The reliability of this LLM judge is not validated against human assessments. Second, this FL-to-NL evaluation approach is not novel and is a known technique in related autoformalization communities (e.g., for Lean), which the paper fails to discuss [1].\n\nInsufficient Validation of Benchmark Data: The quality and correctness of the VeriEquivBench dataset are not adequately established. The authors report that the Python-to-Dafny code transformation was validated by running translated unit tests on only 20 random samples (out of 2,389). While 90% of these passed, this sample size (<1%) is far too small to provide statistical confidence in the correctness of the entire 2,389-problem benchmark. Citing \"long compilation times\" is not a sufficient justification for such a limited validation.\n\n[1] Ying H, Wu Z, Geng Y, et al. Lean workbook: A large-scale lean problem set formalized from natural language math problems[J]. Advances in Neural Information Processing Systems, 2024, 37: 105848-105863.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)