Local Success Does Not Compose: Benchmarking Large Language Models for Compositional Formal Verification
Authors: Xu Xu
Xin Li
Xingwei Qu
Jie Fu
Binhang Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Formal Language
benchmark
Primary area: datasets and benchmarks
TL;DR: We present DafnyCOMP, a benchmark showing that while LLMs excel at single-function verification, they catastrophically fail at compositional reasoning across multi-function programs, exposing a critical gap in verifiable code generation.

Abstract: Despite rapid advances in code generation, current Large Language Models (LLMs) still struggle with reliable and verifiable code synthesis in the presence of *compositional reasoning* requirements across multi-function programs. To study this systematically, we introduce DafnyCOMP, a benchmark for generating compositional Dafny specifications for programs consisting of multiple interacting functions with non-trivial data dependencies. Unlike prior benchmarks that focus primarily on single-function annotation, DafnyCOMP targets programs composed of 2-5 functions arranged in acyclic call graphs, requiring specifications that establish correctness across component boundaries.

DafnyCOMP contains 400 automatically synthesized programs: 300 chain-structured instances and 100 non-chain DAG instances generated from 10 topology templates. We evaluate frontier LLMs from major providers under a unified prompting and verification protocol. While these models achieve high syntactic well-formedness (>99%) and moderate end-to-end verification (>58%) on prior single-function Dafny verification benchmarks, they obtain near-zero end-to-end verification on DafnyCOMP. On the chain split, even the strongest evaluated model reaches only 2% verification at Pass@8, with most models below 1%; the difficulty persists under broader topologies and stronger test-time scaling.

Our analysis identifies three recurring failure modes that hinder cross-functional reasoning: *specification fragility*, *implementation--proof* misalignment, and *reasoning instability*. DafnyCOMP provides a diagnostic benchmark for tracking progress in verifiable code generation, highlighting that bridging local correctness to compositional verification remains a key open challenge.

Published as a conference paper at ICLR 2026
LOCAL SUCCESS DOES NOT COMPOSE: BENCHMARKING LARGE LANGUAGE MODELS FOR COMPOSITIONAL FORMAL VERIFICATION
Xu Xu
Xin Li
Xingwei Qu
Jie Fu"
Binhang Yuan
The Hong Kong University of Science and Technology
2 Nanyyng Technological University University of Manchester SSanghaa Artificial Intelligence Laboratory
ABSTRACT
Despite rapid advances in code generation, current Large Language Models (LLMs) still struggle with reliable and verifiable code synthesis in the presence of compositional reasoning requirements across multi-function programs. To study this systematically, we introduce DAFNYCOMP, benchmark for generating com- positional Dafny specifications for programs consisting of multiple interacting functions with non-trivial data dependencies. Unlike prior benchmarks that fo- cus primarily on single-function annotation, DAFNY COMP targets programs com- posed of 2-5 functions arranged in acyclic call graphs, requiring specifications that establish correctness across component boundaries. DAFNYCOME contains 400 automatically synthesized programs: 300 chain-structured instances and 100 non-chain DAG instances generated from 10 topology templates. We evaluate frontier LLMs from major providers under a unified prompting and verification protocol. While these models achieve high syntactic folleformedness (>99%) and moderate end-to-end verification (>58%) on prior single-functeo Dafny ver- ification benchmarks, they obtain near-zero end-to-end verification DAFNY- CОMP. On the chain split, even the strongest evaluated model reaches only 2% verification at Pass@8, with most models below 1 1%; the difficulty persists un- der broader topologies and stronger test-time scaling. Our analysis identifies three recurring failure modes that hinder cross-functional reasoning: specification misalignment, and reasoning instability. DAFNY- Y-
fragility, implementation-pin CоMP provides a diagnostic benchmark for tracking progress in verifiable code generation, highlighting that bridging local correctness to compositional verifica- tion remains key open challenge.
INTRODUCTION
Large language models (LLMs) have transformed software development through their remarkable code generation capabilities, enabling developers to produce complex programs from natural lan- guage descriptions (Chen et al., 2021 AAsstn et al. a (021). These advances have driven widespread adoption of programming assistants and development environments, fundamentally transforming how modern software developed. As L.L.M-geerrted code becomes increasingly integrated into production systems, critical question emerges: how to ensure the correctness of automatically syn- thesized programs. Unlike human-writte written code that can be manually reviewed and tested, the seale and complexity of LLM outputs demand systematic approaches to verification that go beyond tradi- tional debugging methods. On the other hand, conventional testing provides only partial confidence and cannot rule out corner cases subtle specification mismatches.
Formal verification provides a principled solution to this challenge by offering mathematical guar- antees of program correctness through rigorous specification and proof techniques. Programming languages like Dafny enable developers to express precise pratractt-prcconiitiiion postconditions,
"Equal contribution.
1 Corresponding author (bi yuan yuangust. @ustt .h)).

---

Published as a conference paper at ICLR 2026
(a) Benchmark Performance
(b) Syntax Correctness
(c) Verification Rate
100
Dihyylechhh
Syntax Correct Hatee (%)
75
Venified Palte (%)
5.
90-
1.5
2.5
50.4
NBPP-DFY
80
1.0
0.5
DAPMWCOMP
96.0
20
40
60
80
100
81
802
24
@
81
82
04
28
Rate (%)
Shots
Shots
OpenA)
Google
Doecbook
Albiba
Al (miar))
Figure 1: The formal verification gap: high syntax success versus low verified rates. (a) DAFNY- CОMP exhibits large Syntax-Verified gap. (b-c) Syntax improves quickly with sampling, while verification remains low; inset: multi-turn refinement helps but does not saturate.
and invariants that can be mechanically verified against implementations (Leino 2010b). How- ever, the adoption of formal verification has historically been constrained by what we often refer to as the "specification bottleneck": writing comprehensive annotations not only demands special- ized expertise but also produces specification code that is comparable in size to the implementation itself (Leino et al.) 2017] Poesia et al.) 2024). Recent research has explored the use of LLMs to automate this specification generation process, demonstrating promising results where models can complete missing annotations for individual functions and achieve moderate verification success rates (Loughridge et al. 2024 Yan al., 2025). However, current work in this area often suffers from a critical limitation: existing benchmarks, such as DAFNYBENCH, primarily evaluate annota- tion completion within isolated functions (Loughridge al. 2024), failing to address the composi- tional reasoning ability required for real-world, sophisticated software systems, where correctness emerges from complex interactions between multiple components (Keysers al. 2020).
To fill this gap, we introduce DAFNYCOMP, the first benchmark explicitly designed to evaluate the generation of compositional specifications for programming languages equipped with formal verification. Concretely, we make the following contributions:
Contribution To address the limitations of prior verification benchmarks, we present DAFNY- CоMP, new benchmark explicitly designed for compositional formal verification ($3) Unlike existing datasets such as DAFNYBENCH (Loughridge et al.) 2024) that focus on specification gen- eration for isolated single functions, DAFNYÇOMP evaluates LLMS programs composed of mul- tiple interacting functions with real data dependencies. The benchmark consists of 400 Dafny pro- grams synthesized by combining 2-5 independent functions into acyclic call graphs, including 300 chain-structured programs and 100-program non-chain DAG extension, forcing models to reason across function boundaries to ensure mod-to-end correctness. Note that our design the first re- quire actual compositional reasoning in specification generation, bridging critical gap left by prior benchmarks and reflecting the complexities real-world software systems.
Contribution 2. We comprehensively evaluate 11 state-of-the-art LLMs on the constructed DAFNY- CоMP (5 benchmark across major providers; Figure [egureeprra provides an overview. The results reveal a dramatic collapse in verification performance despite high syntactic accuracy: while models often produce syntactically well-formed outputs, verified success remains extremely low. With up to attempts per problem, the best model reaches only 2% verification success, with most models below 1%, indicating that increased sampling provides only limited gains. Even after stronger test-time scaling via ceiffier-in-teellooe multi-turn self-refinement (up 3 turns), verified rates improve but still remain far from saturation. Besides, on the non-chain DAG extension, we observe the same persistent Syntax-Verified Verified gap with overall lower verified rates than in the chain setting. reinforcing that the core difficulty lies in coordinating interdependent contracts rather than in surface formatting.
We conduct a systematic failure analysis on DAFNYCOMP to explain why verifica- tion collapses despite high syntactic well-formedness, and to move beyond aggregate success rates toward actionable diagnostics. Concretely, we manually inspect 900 representative failure cases from the chain split and distill three recurring failure modes that account for most breakdowns (55): specification fragility, implementation proof misalignment, and reasoning instability. We further provide illustrative examples and discussion for each category, showing how local-looking anno-
Contribution 3.
2

---

Published as a conference paper at ICLR 2026
tations can be compositionally insufficient internally inconsistent, thereby blocking end-to-end proofs. These findings highlight systemic obstacles to compositional formal verification beyond surface-level formatting errors, and position DAFNYCOMP as a diagnostic benchmark for tracking progress toward verifiable multi-component programs.
2 RELATED WORK
Formal Verification Benchmarks. Existing benchmarks for verifiable code generation be cat- egorized into two types. Single-function benchmarks, such as DAFNYBENCH (Loughridge et al.) 2024) and MBPP-DFY (Misu al. (2024), evaluate annotation completion within isolated methods, achieving moderate success rates (50-60%) ) but failing to capture inter- functfnnntion dependencies. Inter- active theorem proving benchmarks (miniCodeProps (Lohn& Welleck (024)), FVAPPS (Dougherty) & Mehta 2025 ) target proof synthesis in systems like Lean (De Moura et al. 2015) but require ex- tensive manual validation and remain disconnected from practical programmiing DAFNYCOMP bridges this gap by evaluating compositional specification generation- prerequisite for scaling verification beyond toy programs to production systems. Unlike prior work, we explicitly construct multi-function programs with data dependencies, exposing the compositional reasoning deficit in C for detailed comparison).
current models (see Appendix
Dynamic Benchmark Generation. Static benchmarks suffer from contamination and overfit- ting (Hu et al.) 2025 Zhang et al., 2024). Dynamic generation techniques creating new tasks, transforming problems, or perturbing reasoning structures (Zhu al., 2024] 2023) -show promise in mathematics and logic but neglect formal verification's 's unique demands: specifications must be syntactically valid, semantically precise, and correct across all execution paths. Our synthesis pipeline addresses this by generating verifiable multi-function Dafny programs through controlled composition, ensuring both novelty and correctness while maintaining the semantic complexity that exposes compositional reasoning gaps.
Compositional Reasoning in LLMs. The ability to systematically combine simpler units into cor- rect larger structures remains a frontier challenge eL al., 2024 Dziri et al. (023). While progress exists in natural language and symbolic domains, formal verification imposes stricter demands: specifications must preserve invariants across components and ensure global correctness. Existing training paradigms favor pattern matching over principled proof construction. By requiring models to generate specifications bridge function boundaries with explicit data dependencies, DAFNYCOMP provides the first diagnostic benchmark for compositional reasoning in formal verification.
BENCHMARK CONSTRUCTION
DAFNYCOMP synthesizes 400 verified multi-function programs through a two-stage pipeline (Fig- ure D. including program assembly ($3.1) and formal translation ($3.2), which bridges the gap between practical Python implementations and verification-orientee Dafny specifications. Within this pipeline, program assembly ensures the construction of compositional Python programs with functional correctness, while specification translation with refinement ensures the quality and reli- ability of the resulting data. We also provide format details of evaluation tasks ($3.3 and the key characteristics of the benchmark ($3.4). Unless stated otherwise, our primary Pass @k evalua- tion focuses on the chain split (300 programs), and we additionally include a non-chain DAG extension set (100 programs) to broaden topology diversity.
3.1 PROGRAM ASSEMBLY
We construct compositional programs by systematically combining functions from LEETCODE- DATASET (Xia et al. 2025), selected for their algorithmic depth and verification challenges.
Function Selection. We filter the corpus using McCabe's cyclomatic complexity (McCabe 1976)
as proxy for verification difficulty, retaining only functions with complexity >5 (around top 30%
of the dataset) and least 10 lines of code. This threshold ensures non-trivial control flow-loops with complex termination conditions, nested conditionals, recursive patterns-th stress specifica- tion generation. For tractability, we restrict to mangll--nputssigle-oous functions, yielding 1 1,847 candidate functions.
Compositional Strategy. Following Hu et al. 2025). we primarily employ chain-based composi- tion where each function's output feeds the next function's 's input, creating explicit data dependencies
3

---

Published as a conference paper at ICLR 2026
10101001
betted Esutple(a: areoy (1a63) (b: aErayS
requiras a Length > Precssdision
PY
Leleghh Postonstition
Complestty Fllleing
Call-grophh Asembly
Correctness
Hf inplesentation
Python
Syntox (auP(%)
PROGRAM ASSEMBLY
Input for
Mnn
Verty Paua(()
Metrics
Base dataset
nothed Exappia(s: array (ist>) raturne (b: azzay<inty
AST Analytt
Modutar Code
Iterative Ref inmeenn
Verfied Dainy Implementation
for
b.length Dostccndiiinn
Precendision
FORMAL TRANSLATION
Ieplesantation
Figure 2: Two-stage benchmark synthesis: (1) Assembly combines independent Python functions with controlled data flow, ensuring algorithmic complexity while maintaining tractability; (2) For- mal translation converts to verified Dafny through incremental AST-guided transformation.
while keeping the assembly procedure controlled. To broaden topology diversity beyond chains, we additionally instantiate 10 non-chain DAG topology templates and synthesize 100-program DAG extension set, covering common branching patterns in acyclic call graphs (Appendix D lists the templates, illustrating the extensibility of our assembly to broader acyclic call graphs). After com- position, further identify the minimal set of shared import dependencies across the combined Python functions. This step is essential because the original LEETCODEDATASET often relies on broad import statements, which obseure library ownership and names. Without explicit map- pings, Dafny cannot interpret external libraries, preventing the synthesis of intermediate functions to replace missing third-party features. We generate programs with 2-5 functions, exploring multiple permutations since function ordering affects both data flow and verification complexity.
Validation Pipeline. After composition, the resulting Python code is subjected to three-stage validation pipeline, which filters candidates before their use in Section 3.2
(i) Type checking via constraint propagation: We statically infer candidate types and shapes for each function's 's inputs and outputs and propagate these constraints along the composition chain. This pass rejects compositions with incompatible interfaces (e.g., scalar-sequence element-type mismatches) and flags violations of simple value constraints inferred from guards (such as non- negativity or length bounds). The result a set of compositions whose interfaces are consistent end-to-end, providing reliable basis for subsequent translation and verification.
(ii) Formatting standardization: We apply deterministic rewriter, implemented with tools such Black and isort, to normalize code style, including indentation, whitespace, line breaking, and import organization. Canonicalizing these incidental variations yields stable, diff-friendly ar- tifacts and reduces prompt variance in later stages. This step preserves semanties while producing uniform program layouts that are easier to parse, translate, and verify.
(iii) Test validation: We execute each composed program against the reference unit tests from LEETCODEDNTASET to confirm functional correctness and basic exccutability. We construct unit tests for composed functions by intersecting the onput-output constraints of their constituent functions. the intersection is empty, the sample discarded. Programs that raise runtime exceptions, fail assertions, or produce incorrect outputs are discarded, ensuring only behaviorally sound compositions advance. This filtering isolates verification challenges to specification and reasoning rather than implementation errors during the Dafny translation stage.
Following this procedure, we obtain 1,200 valid Python programs with 2-5 functions.
3.2 FORMAL TRANSLATION
We translate validated Python compositions into Dafny implementations with formal guarantees, focusing on the crificationnorieente aspects of the benchmark here.
Translation Challenges. Direct end-to-end translation from Python to Dafny proved largely in- effective, with empirical success rates below 5%. The core difficulty lies in Dafny's demand for
4

---

Published as a conference paper at ICLR 2026
explicit specifications, invariants, and termination arguments semantic elements absent in Python. This semantic gap makes single-pase pass translation infeasible for non-trite trivirivial programs.
Incremental Pipeline. Inspired by Wenn al. (2024). we adopt an incremental approach: the abstract syntax tree (AST) of each Python program is decomposed into function- or control- structure-leve fragments. Each fragment is translated into Dafny and immediately verified, lo- calizing errors to the smallest possible unit. Verified fragments are then progressively reassembled according to the AST hierarchy, culminating in a complete Dafny program. Importantly although translation proceeds incrementally, the Python program must be composed in its entirety before it can be executed. Whole-prograte composition in Python provides two benefits: (i) Python's ex- plicit AST nodes and mature tooling make program assembly more reliable and transparent: and (ii) having a coherent Python blueprint ensures that the incremental Dafny translation preserves global logical relationships, rather than producing isolated fragments that fail to compose. Thus, whole- program composition and incremental translation are complementary design choices. To further improve reliability, each candidate Dafny program undergoes up to 10 refinement iterations, where specifications are strengthened in response to verifier feedback (e.g., adding loop invariants, refining postconditions... ). We select CLAUDE-4- see Appendix E for success rate comparisons and Appendix | for the exact prompts. To mitigate potential information leakage arising from evaluating models from the same family used in synthesis, we do not include Claude-family models in the downstream comparative evaluation.
for the entire synthesis and refinement process;
In total, the pipeline ultimately yields 564 verified Dafny programs from 1,200 attempts (corre- sponding an overall 47% success rate). Translation synthesis errors primarily arise from incom- plete specifications (3 %). type inference errors (22%), timeouts (18%), and irreconcilable semantic gaps (29%). ) From these. we retain 300 chain-structured programs carefully balanced across com- plexity levels (100 each with 2-3, 3-4, and 4-5 functions) as our primary chain split. In addition, we apply the same assembly and tramalltranslation pipeline to non-chain DAG compositions and obtain 100 verified DAG-structured programs as an extension set, yielding 400 benchmarks in total. Given that DAFNYCOMP is automatically synthesized and intended to stress compositional verification, is crucial that the resulting instances are both novel and not inadvertently recoverable from exist- ing benchmarks or corpora. To ensure evaluation integrity. we conduct a thorough contamination analysis against the source Python dataset used by MBPP-DFY (Misu et al., (002)). The results in Appendix provide strong evidence that our test set is indeed free from bias due to data overlap.
3.3 EVALUATION TASK FORMAT
We adopt a specification reconstruction task inspired by Loughridge et al.) (2024). Still, with a crucial difference: rather than removing all annotation statements, we only strip away the con- tract clauses (requi res, ensures, reads, modi fodifiee fies, decreases) that appear before the opening brace of each het or function LLMs to be evaluated are then required to regen- erate these specifications to enable verification. This design isolates the challenge of reconstructing cross-function contracts from implementation concerns, focusing evaluation on whether models can generate specifications that capture emergent correctness properties across component boundaries. Unlike annotation completion tasks that permit purely local reasoning, our multi-function programs require understanding how data flows and invariants propagate through compositions. We use uni- fied prompt and standardized compute settings across all evaluations (Appendix Hnd Appendix [].
3.4 BENCIMARK STATISTICAL SUMMARY
The resulting benchmark comprises 400 mechanically verified Dafny test cases, including 300- program chain split and a 100-program non-chain DAG extension set. Unless stated otherwise, the statistics reported in this section summarize the chain split, and the DAG extension is described separately. The benchmark jointly captures three key dimensions: compositional complexity from fouunciotoo-uunction call dependencies, algorithmic diversity across multiple categories, and verifi- cation challenges arising from the increased specification burden.
Compositional Complexity. Each program contains 2-5 functions (mean = 3.2) with an average of 8.4 cross-function data dependencies, requiring models to reason about specification alignment across component boundaries. Unlike single-fune function benchmarks where specification generation is largely local, our programs demand that preconditions of called functions be implied by post- conditions of their callers-e requirement that introduces cascading verification challenges when specifications fail to propagate correctly.
5

---

Published as a conference paper at ICLR 2026
Algorithmic Diversity. The benchmark spans 15 algorithmic categories with balanced representa- tion: dynamic programming (18%), string manipulation (20%), number theory (15%), and graph algorithms (12%) constitute the primary categories, with the remainder distributed across sorting, searching, and combinatorial problems. Beyond the balance of individual categories, diverse per- mutations and combinations of these types yield composed programs with more intricate, layered structures. Consequently, the target of composing function calls is reinforced by the characteristics of the source dataset (LBETCODEDATASE) (Xia et al., 2025)), ), which in turn induces composition at the level of algorithmic logic. This design ensures models must develop general compositional reasoning rather than memorizing eetegorr-speccinne patterns.
Verification Challenges. Every program mechanically verified by Dafny 4.10.0, thereby provid- ing ground-truth correctness. The median program requires 7 loop invariants and 4 assertions for verification, with 23% demanding intricate termination arguments via decreases clauses-a 3.5x increase in annotation density compared to BENCI''' average of 2 per program. This added specification burden reflects the extra complexity of compositional verification, creating a graduated challenge that pinpoints where current models shift from local reasoning to compositional failure.
4
EXPERIMENTAL SETUP AND RESULTS
In this section, enumerate the evaluation metrics ($4.1 LLM model selection for the benchmark and the corresponding evaluation results ( 4.3 3
(.22).
4.1
METRICS
We evaluate two complementary aspects of model performance:
measures whether generated specifications parse successfully in Dafny. This baseline metric captures models' grasp of the formal language syntax.
Syntax Correctness:
measures the fraction of syntactically correct programs that pass Dafny's veri- fier-the ultimate test of semantic understanding. This metric is computed only over syntactically valid outputs, verification requires parseable code.
Verified Rate:
Independent sampling (Pass@k). Following Chen et al. (021), we report Pass@k for 21, 4, 8). measuring the overall probability of successfully solving problem within & attempts. @ass@1 1 provides a strict zero-shot baseline of immediate reasoning ability, whereas larger val- ues further exploit additional test-time compute to improve success compositional tasks (Snell al. 2024). In this setting, Pass provides a stable view performance under moderate test-time compute beyond single attempt.
Verifier-in-the-loop multi-turn refinement. In addition to independent sampling (Pass@ k), we evaluate stronger test-time scaling baseline based on verifier-in-the-lon multi-turn self- refinement. Starting from an initial generation (turn 1), if the output fails parsing or verification, we feed the model its previous output together with the verifier error message and request revision while keeping the method signatures and declared contracts unchanged. We repeat this process for up to T 3 turns, running the Dafny verifier once per turn under the same timeout, and report Syntax Correct Rate@tunn turn/ and Verified Rate @ turn/ for / E (1,2,3) 3). Due to the cost of repeated generation and verification, report multi-turn results representative subset of models.
4.2 MODEL SELECTION
We evaluate 11 frontier models spanning four major model families, chosen for their demonstrated strength in code generation and general reasoning:
OpenAI: GPT-40 (Hurst et al. 2024), GPT-4.1 (OpenAl 2024), 04-MINI OpenAl 2025
Google: GEMINI-2.5-PRG. GEMINI-2.5-FLAN (Google DeepMind 2025)
DeepSeek: DEEPSEEK-RI (Guo et al. 2025), DEEPSEEK-V V3 (Liu et al. 2024). DEEPSEEK-
V3.1 (DeepSeek-Al 2025
Alibaba: QWEN2.5- 1--.5-CCC--333 32B (Hui al. 2024 OWEN3-CODER-4800 (Qwen Team
2025a), QwQ-32B (Qwn Team 2025b)

---

Published as a conference paper at ICLR 2026
Table Main results on the chain split (300 problems) under independent sampling (Pass@k).
We report Syntax Correct Rate and Verified Rate (i.e., Pass@kk for k 4,8) under unified prompt and fixed verification procedure. Despite high syntactic well-formedness, verification success remains extremely low (best 2% at Pass@8).
Syntax Correct Rate (%)
Verified Rate (%)
Model
@
@2
@4
@
@ 1
@2
@4
@
OpenAl Models
GPT-40
94.33
98.67
99.33
99.67
0.33
0.33
0.33
0.33
04-MINI
80.00
92.67
98.00
99.00
0.00
0.00
0.67
0.67
GPT-4.1
59.00
69.67
79.33
86.33
0.00
0.00
Google Models
GEMINI-2.5-ELASH
54.00
64.00
81.00
89.67
0.00
0.00
0.00
0.00
GEMIN1-2.5-PRO
69.00
81.00
91.67
96.00
0.00
0.33
0.67
2.00
DeepSeek Models
DEEPSEEK-RI
85.67
95.33
98.33
99.00
0.33
0.33
0.33
0.33
DEEPSEEK-V3 DEEPSEEK-V3..
77.33
88.67
95.33
97.33
0.00
0.00
0.33
0.33
54.67
72.33
83.33
92.00
0.00
0.00
0.67
Alibaba Models
85.33
94.00
98.00
99.00
0.00
0.33
0.33
1.00
SEEEN25--CODER32B--ISS-- QWQ-32B
62.00
74.67
85.00
89.00
0.33
0.67
46.67
61.33
78.00
91.00
0.00
0.00
0.00
0.00
Table 2: Verifier-in-the-loop multi-turn self-refinement (representative models). We report rep- resentative models weett-perorming among evaluated multi-ture models per family under Verified Rate @ (T) on the chain split (300 problems) and the DAG extension (100 problems) for readability; full results are provided in Appendix
Syntax Correct Rate (%)
Verified Rate (%
Model
T1
T2
T3
T1
T2
T3
Chain split
DEEPSBEK-RI
85.67
94.33
95.00
0.33
5.67
9.67
QWEN3-CODER
SEESS-COERR-4800 480B---88---- --8--A35B--MMA INSTRUCTT
78.33
93.33
95.67
0.33
0.67
3.00
04-MINI
37.33
92.00
96.00
0.00
2.00
9.67
DAG extension
13.
59.00
94.00
98.00
1.00
3.00
7.00
QWEN3-CODER 100---888--333- A35B-INSTRUCT
69.00
95.00
97.00
0.00
1.00
4.00
GPT-4.1
26.00
77.00
96.00
1.00
2.00
4.00
4.3 RESULTS AND DISCUSSION
Table reports our main results on the chain split under independent sampling (Pass@ k). To eval- uate stronger test-time scaling beyond independent sampling, Table 2 reports a verifier-in-the-loo multi-turn self-refinement baseline both the chain split and the DAG extension for representative models, with full results Appendix J
Observation 1. Universal verification collapse despite syntactic mastery. The most striking result a large and consistent gap between syntax correctness and verification success across all evaluated models. At Pass @ 8, models achieve = 94.36% (SD 4.86%) Syntax Correct Rate but only = 0.55% (SD = 0.58%) Verified Rate, yielding a 93.82% Vertax-e gap. This mismatch not confined to particular model family scale: it appears across dense and MoE models, across general-purposs and code-specialized systems, and it persists even when increasing test- time compute via independent sampling (from @ass I to @ass@8) 8). The universality of this failure suggests a systematic limitation in current LLMs' ability to coordinate interdependent contracts, rather than issue that can be remedied by formatting fixes modest increases sampling.

---

Published as a conference paper at ICLR 2026
Observation 2. Compositionality induces disproportionate degradation from single-function ification. In prior single-functee verification benchmarks, frontier LLMs can achieve high syn- tax comrccness (> 99%) and moderate verified success (> 58%). In contrast, multi-function compositional tasks, verified success collapses to low single digits even under multi- ple attempts (average 0. 0.55%, best 2.00% at k=8). This degradation cannot be explained by additive difficulty alone: composing functions introduces interdependent contracts, where missing or slightly weakened postconditions in one component can fail establish a callee's precondition downstream, cascading into and-to-end proof failures.
DAFNYCOMP
Observation 3. Limited gains from sampling; multi-turn refinement helps but remains far from saturation. Under independent sampling, verification gains saturate quickly: the average improve- ment from k=4 to 8=8 is only +0.27%, and 7/11 models show no gain at all, whereas syntax correctness continues to improve with additional attempts. Under stronger verifier-in-the-lee multi-turn baseline, Verified Rate improves substantially on the chain split (e.g., representative mod- els reach up 9.67% turn 3; Table 2p. yet success remains far from saturation. Moreover, on the DAG extension, we observe a similarly persistent gap: even with refinement, Syntax Correct Rate @T3 remains high (up to 99%) while Verified Rate @T3 stays in the low single digits (1-7%), with verified rates on DAG topologies lower on average than in the chain setting (Appendix D).
Observation 4. Seasoning-speecializn models show no clear advantage. Models explicitly opti- mized for reasoning do not consistently outperform general-purpose models on DAFNYCOMP. For example, QwQ-32B still achieves 0% verification at Pass @ 8, and DEEPSECK-RI remains below 1% verification at @ despite its tion success is tightly clustered in the low single digits across diverse training objectives and model families. This null result informative: it suggests that current reasoning-tracee trace-basedd and reward- optimized approaches, as instantiated by today's frontier models, are not yet sufficient to reliably solve compositional verification on DAFNYCOMP.
peterreemeet-leamnng based post- post-training. Overall, verifica-
5 FAILURE CASE ANALYSIS AND DISCUSSION
Table 3: Distribution of verification failure modes across 900 analyzed cases (3 best-performing models under our primary chain Pass @8 setting x 300 chain benchmarks) from DAFNYCOMP.
Failure Mode
Frequency
% of Total
Primary Mechanism
Specification Fragility
353/900
39.2
Contract propagation failure
Implementation-PPonn Misalignment
195/900
21.7
Independent generation pathways
Reasoning Instability Other (syntax, timeout, misc.)
127/900
14.1
Inductive chain breakdown
225/900
25.0
Various
The significant gap between syntax correctness and ultimate verification success demands a clear mechanistic explanation. We therefore conduct systematic manual analysis of 900 failures drawn from the chain split (300 programs x the three best-performing models under our primary Pass @8 evaluation). Table summarizes the distribution of failure modes over this analyzed subset; our goal is to distill recurring breakdown patterns rather than to estimate a global failure distribution across all models. We discuss each mode below and provide additional concrete examples Appendix K
5.1 SPECIFICATION FRAGILITY: THE DOMINO EFFECT
Specification fragilit, the inability generate contracts that remain valid across function composi- tions, accounts for most failures. Consider a representative case from DAFNYCOMP: a d bigitSuun function correctly implemented but missing the postcondition ensures result >= In iso- lation, this omission appears minor. In composition, it siccades--heee BigitSun' output feeds a caller expecting non-negative input, verification fails globally despite both functions being locally correct. Note that this pattern recurs throughout our dataset: models generate specifications suffi- cient for local correctness but insufficient for compositional soundness. A requires n >= precondition absent from one function invalidates the entire pipeline's 's verification, even when each component individually passes most test cases. The fragility stems from a fundamental mismatch: LLMs learn specifications local patterns rather than global contracts. They lack the architectural machinery to reason about how data constraints propagate through function calls-e a capability es- sential for modular verification. More broadly, systems requiring compositional correctness from
8

---

Published as a conference paper at ICLR 2026
distributed systems protocols to smart contract verification may face similar failures until models can reason about specification flow across component boundaries.
The first key takeaway insight about specification fragility is summarized below:
Takeaways (i: LLMs often handle local specs but fail under composition. Missing contract propagation is the main cause of verification breakdowns.
5.2 IMPLEMENTATION-PROO MISALIGNMENT: THE INDEPENDENCE ASSUMPTION
The second failure mode points to a generation- mismatch: models often produce implementa- tions and specifications if they were independent, rather than as coupled constraints. In 21.7% of the analyzed cases, syntactically valid code contradicts its own specifications. One striking example: a model generated assert 0 ;; within otherwise reasonable code, illustrating a recurring pattern of internally contradictory assertions rather than an isolated slip. More subtle misalignments prove equally fatal: loop invariants like forall appear plausible but fail verification because the implementation's array access patterns violate the stated bounds. This pattern is consistent with prior mechanistic analyses suggesting that transformer circuits often implement relatively local algorithmic behaviors rather than globally consistent proof construction (Olsson et al. 2022 Elhage et al. 2021) Current training paradigms further exacerbate this issue: models learn from code-specification pairs without explicit feedback mutual consis- lency, yielding impressive syntax but brittle semantic alignment in composed settings.
:: 0 <= k < ==> ent [k]
We summarize the second key takeaway about implementation-ppeeee proof misalignment as:
Takeaways (ii) Code and specs generated independently. leading plausible but in- consistent invariants. Future training for this task should enforce better alignment.
5.3 REASONING INSTABILITY: INDUCTION AS ACHILLES' HEEL
The third failure pattern, which we refer to as reasoning instability, highlights a major barrier sustaining the inductive arguments required by formal verification. Formal proofs often re- quire establishing a base case, maintaining invariants through iterative updates, and composing these guarantees across function boundaries. In DAFNYCOMP, models frequently fail to main- tain this inductive chain: loop invariants that should accumulate state (c.g., invariant st IrringToIntteeppp recursive functions often miss well-founded decreases measures inductive strengthening needed to generalize beyond base cases. These failures suggest that, while LLMs can match locally plau- sible invariant templates, they often struggle maintain globally consistent inductive reasoning. especially in compositional settings where relationships must be preserved across calls.
(s [. [..)) ))) break as models lose track of how program state evolves, and
We summarize the third insight about reasoning instability as:
Takeaways (ii): LLMS approximate base cases but fail to sustain inductive reasoning, posing a structural gap in formal verification.
LIMITATIONS AND FUTURE WORK
While DAFNY COMP exposes fundamental limitations in compositional reasoning, we want gently mention several constraints of our evaluation, which indicate some interesting future work.
Compositional Patterns. Our primary evaluation focuses on acyclic chain-struetured compo- sitions for tractability, and the benchmark additionally includes 100-program non-chain DAG extension. More complex patterns such as cyclic graphs, mutual recursion, or saaeed-tatte-hevyy dependencies remain out of scope. Extending these settings would require addressing synthesis and verification tractability in the presence of cyclic dependencies.
Specification Types. Our benchmark tests functional correctness (preconditions, postconditions, invariants) but not liveness properties, resource bounds, security policies. These orthogonal

---

Published as a conference paper at ICLR 2026
concerns e.g., proving memory consumption remains constant across compositions- -enruiitooo--ruuiree different verification techniques and evaluation metrics.
Data Scarcity. The core challenge may be training data availability. Repositories contain only a few verified multi-function programs with compositional specifications. Synthetic data generation or bootstrapped program synthesis could address this gap, at scale and with stronger coverage, although ensuring semantic diversity remains a challenge.
7 CONCLUSION
We introduce DAFNYCOMP, the first benchmark specifically designed to evaluate the generation of compositional specifications for formal verification. DAFNYCOMP contains 400 mechanically verified multi-function Dafny programs, including 300-program chain split and 100-program non-chain DAG extension. We evaluate 11 frontier LLMs under a unified prompting and fixed veri- fication protocol. Our results reveal a stark Syntax- Verified gap: while models achieve high syntax correctness, verified success on compositional tasks remains extremely low (average 0.55%, best 2.00% at P on the chain split). Stronger test-time scaling via verifier-in-the-loop multi-turn self-refinement improves success but remains far from saturation, and the difficulty persists on the DAG extension with lower verified rates. Through manual failure analysis, we identify three recur- ring breakdown patterns spasiificcpciiiiioonnnnn fragility, implementation pnppep misalignment, and reason- ing instability- that systematically hinder end-to-end compositional verification. Overall, DAFNY- COMP provides a diagnostic target for tracking progress in verifiable code generation and highlights that bridging local correctness to compositional verification remains a key open challenge. We re- lease the benchmark, evaluation framework, and synthesis pipeline to support future research.
ACKNOWLEDGMENT
This work is supported by the HKUST startup grant R9895 from CSE; RGC-ECS project 26218024; RGC-NSFC project CRS_ HUS6600124 We thank the support from National Supercomputer Cen- ter in Guangzhou Nansha Sub-Center.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. Xiv preprint arXiv:2108.077 077732, 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained code. arXiv preprint ar arXiv:2107.0337 2021.
Leonardo De Moura and Nikolaj Bjørner. 73: An efficient smt solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems, pp. 337-340. Springer, 2008.
Leonardo De Moura, Soonho Kong, Jeremy Avigad, Floris Van Doom,, and Jakob von Raumer. The lean theorem prover (system description). In International Conference on Automated Deduction, pp. 378-388. Springer, 2015.
DeepSeek-AI. Deepseek-v3.1 model introduction. cessed: 2025-09-21.
ccmpeeeeeeecc.../o 2025. Ac-
Quinn Dougherty and Ronak Mehta. Proving the coding interview: benchmark for formally verified code generation. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLMMCode), pp. 72-79. IEEE, 2025.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, al. Faith and fate: Limits of trans- formers on compositionality. Advances in Neural Information Processing Systems, 36:70293- 70332, 2023.
10

---

Published as a conference paper at ICLR 2026
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Cireuits Thread, 1(I):12, 2021.
Google DeepMind. Gemini 2.5 models.
gemi n 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint MiXiv:2501.12944 2025.
Wenhao Hu, Jinhao Duan, Chunchen Wei, Li Zhang, Yue Zhang, and Kaidi Xu. Dynacode: A dynamic complexity-avee aware code benchmark for evaluating large language models in code gener- ation. arxiv preprint arXiv:2503. 2025.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 121 12186 2024.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Daniel Keysers et al. Compositional generalization in natural language processing. Transactions of the Association for Computational Linguisties, 8:11-23, 2020.
K. Rustan M. Leino. Dafny: An automatic program verifier for functional correctness. In Logic for Programming, Artificial Intelligence, and Reasoning, pp. 348-370. Springer, 2010a.
K. Rustan M. Leino. A Tour of the Dafny Program Verifier. Verified Software: Theories, Tools, Experiments, 2010b.
K. Rustan M. Leino et al. Compositional verification of a railway protection system with Dafny. Formal Aspects of Computing, 2017.
Yue Chen Li, Stefan Zetzsche, and Siva Somayyajula. Dafny as verification-aware intermediate language for code generation. arXiv preprint SrXiv:2501.06283, 2025.
Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. Understanding and patching compositional reasoning in llms. arXiv preprint 2xXiv:2402.14320 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.194477, 2024.
Evan Lohn and Sean Welleck. minicodeprops: a minimal benchmark for proving code properties. arXiv preprint 19Xiv:2406.11915, 2024.
James Loughridge et al. DafnyBench: A Benchmark for Formal Software Verification. Transactions Machine Learning Research, 2024. Describes the 'fill annotations task.
Thomas McCabe. A complexity measure. IEEE Transactions on software Engineering, (4):308-
320, 1976.
Md Rakib Hossain Misu, Cristina V Lopes, Iris Ma, and James Noble. Towards ai-assisted synthesis of verified dafny methods. Proceedings of the ACM on Software Engineering, 1(FSF)):12-835, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 11895, 2022.
OpenAl. Gpt-4.1 system card. httppp://peeaai.ccm//eeeeeee
2024.

---

Published as a conference paper at ICLR 2026
OpenAL.
Introducing openai o3 and 04-mini.
https://ppenaa.com//neet
2025. Accessed: 2025-09-21.
Gabriel Poesia, Chloe Loughridge, and Nada Amin. dafny-annotator: Ai-assisted verification of dafny programs. arXiv preprint 11Xiv:2411.151433 2024.
Qwen Team. Qwen3-coder: Agentic coding in the world. htt.p:://qeenmm.iithhbbb io/
blog/ /tlog//wen--coerr///// 2025a. Accessed: 2025-09-21.
Qwen Team. Qwq-32b: Large-scale reinforcement learning for reasoning models. https://
gwenlm. github. ic/ http:/hhbllgg//gg/ hwtpp//////////////// 32b/ 2025b. Accessed: 2025-09-21.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 2024.
Cheng Wen, Jialun Cao, Jie Su, Zhiwu Xu, Shengchao Qin, Mengda He, Haokun Li, Shing Chi Cheung, and Cong Tian. Enchanting program specification synthesis by large language models using static analysis and program verification. In International Conference on Computer Aided Verification, pp. 302-328. Springer, 2024.
Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms. arXiv preprint arXiv:2504. 14655, 2025.
Chuanhao Yan, Fengdi Che, Xuhan Huang, Xu Xu, Xin Li, Yizhi Li, Xingwei Qu, Jingzhe Shi, Zhuangzhuang He, Chenghua Lin, Yaodong Yang, Binhang Yuan, Hang Zhao, Yu Qiao, Bowen Zhou, and Jie Fu. Re:Form Reducing Human Priors Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny. arXiv preprint mrXiv:2507.16333 2025.
Zhehao Zhang, Jiaao Chen, and Diyi Yang, Darg: Dynamic evaluation of large language models via adaptive reasoning graph. Advances in Neural Information Processing Systems, 37:135904- 135942, 2024.
Kaijie Zhu, Jíaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dy- namic evaluation of large language models for reasoning tasks. arXiv preprint 171677, 2023.
Kaijie Zhu, Jindong Wang. Qinlin Zhao, Ruochen Xu, and Xing Xie. Dyval 2: Dynamic evaluation of large language models by meta probing agents. arXiv preprint arXiv:2402. 14865, 3, 2024.
12

---

Published as a conference paper at ICLR 2026
A THE USE OF LLMS IN WRITING
During the preparation of this manuscript, we employed large language model (OpenAl GPT-5) to assist with language refinement and editorial improvements. Specifically, the LLM was used to enhance sentence fluency, improve clarity of expression, and ensure consistency with academic writing conventions. The tool was applied exclusively for linguistic polishing- all research design, experimental work, data analysis, and core intellectual contributions remain entirely original.
B INTRODUCTION TO DAFNY
Dafny Leino 2010a), developed at Microsoft Research, is a verification-orienteed programming language specifically designed to support formal reasoning about software. Unlike conventional languages where correctness is primarily assessed through testing. Dafny integrates an automated program verifier directly into the development workflow, enabling developers to construct code that is mathematically proven to satisfy its specifications. This approach shifts the discovery of defects from the testing phase the design and implementation phases, thereby improving software relia- bility.
distinctive feature of Dafny is that specifications are treated as first-class citizens. Methods can be annotated with preconditions, postconditions, and logical properties that describe intended behavior. For example:
method Example (a: array<int>) returns (b: array<int>) requires Length > 0
// Precondition
ensures b. Length LeIghh
// Postcondition
ensures forall :: <= b. Lenghh b[i] // Property
// Implementation
The Dafny verifier relies on automated theorem proving (via Z3 solver (De Moura & Bjørner 2008 to ensure that implementations conform these specifications, providing mathematical certainty about program behavior. Crucially, the ability to reason about the composition of verified compo- nents determines whether verification can scale from toy examples to real-world systems. With- out compositional reasoning, verification remains confined to small, isolated programs rather than production- software.
C AUTOMATED THEOREM PROVING
A complementary line of work contrasts automated verification frameworks with interactive theorem proving (ITP) systems. Languages such as Dafny and Verus rely on SMT solvers to discharge proof obligations, requiring only lightweight annotations (e.g., invariants, assertions). This design lowers the barrier to entry but is constrained by the solver's limited reasoning scope and opaque failure modes. contrast, ITPs such as Lean expose every proof step explicitly, enabling iterative refinement and eиror diagnosis. Recent studies even show that LLMs can generate competition-level mathematical proofs in Lean. However, existing Lean-based benchmarks (e.g., miniCodeProps, FVAPPS) either focus narrowly on proof synthesis or lack human validation. By comparison, Dafny offers a more balanced environment for benchmarking LLMs: it combines code, specifications, and automated verification in a way that remains close to mainstream programming practice.
How Dafny Works and Its Core Strengths. Dafny's approach stems from its verification-awar design. Developers embed formal specifications. such as preconditions, postconditions, and loop invariants, directly within the code (Leino) 2010a). These specifications are not merely comments; they are integral components checked by the built-in verifier. The verifier translates Dafny code and its specifications into an intermediate verification language, Boogie, which then generates proof obligations. These obligations are processed by an SMT solver (e.g., Z3) to prove their validity. If all obligations are proven, the code is confirmed to be correct according to its specifications. If a proof fails, Dafny provides precise feedback on the inconsistencies. This methodology supports correctness construction, helping to reduce common errors like null pointer dereferences ray -oound-- access (Poesia et al., 2024). Once verified, Dafny code can be translated into mainstream languages such as Python for execution (Li et al. 2025).
13

---

Published as a conference paper at ICLR 2026
Dafny Python: A Fundamental Difference in Approach. To understand Dafny's position, it's useful to compare it with a widely used language like Python. While both are effective, their fundamental design philosophies and primary objectives differ, shown in Table 4
Feature
Dafny
Python
Year Introduced
2010 (Microsoft Research)
1991 (Guido van Rossum)
Type System
Static typing. compile-time checks built-in contracts and proofs
Dynamic typing, run-time checks
Formal Verification
Yes -
No only basic assert
Main Use
Verified algorithms, critical systems
General-purpose programming Interpreted (e.g., CPython)
Execution Model
Compiled with verification
Table 4: Key differences between Dafny and Python.
In summary, Dafny offers a distinct approach to software development by integrating formal verifi- cation into the language itself. While Python excels in agile development and broad applicability, Dafny is particularly suited for domains where software correctness and formal guarantees are criti- cal. For more. please refer the Dafny official website
D DAG TOPOLOGY TEMPLATES
To broaden topology diversity beyond chains, we instantiate acyelic DAG templates and synthe- size 100 DAG-structured programs (10 per template) extension set in DAFNYCOMP.
ID
Topology description
1
4 nodes, splits into two, one branch splits again
2 4 nodes, root splits into three nodes, chain with a branch at the second node
3
4
5 nodes, chain then splits into three
5
nodes, root splits into four
6
5 nodes, root splits into three, leftmost branch splits again
7
5 nodes, chain with a branch at the second node, both branches split again
8
5 nodes, root splits into two, left branch splits again
9
5 nodes, root splits into two, both branches are chains of two
10 5 nodes, root splits into two, one branch is chain of three
Table 5: DAG topology templates used to instantiate the 100-program DAG extension set.
14

---

Published as a conference paper at ICLR 2026
E MODEL SELECTION FOR FORMAL TRANSLATION
Concretely, we randomly sampled 100 test cases from the 1,200 instances obtained after our as- sembly procedure evaluate each model with identical prompts and decoding settings. We define success rate as the fraction of generations whose specifications are valid under the Dafny verifier.
Model
Success rate (%)
Claude-4-oomne-2225555544
58.00
haade--.5-sonnet-22244100
55,00
gpt-3.5-turbo
45.00
gpt-4o
31.00
gpt-4o-mini
41.00
01
36.00
ol-mini
33.00
03-mini
37.00
gemini-2.0-flash
38.00
Table 6: Success rates for the formal translation step.
As shown, we ultimately chose valid specifications on these cases. This aligns with our design goal prioritize the quality of generated specifications used downstream in our experiments.
because it yielded the highest rate of
15

---

Published as a conference paper at ICLR 2026
F PROMPTS FOR SYNTHESIS
The prompt templates used for annotating data with Claude-4 Sonnet are shown the following boxes.
Prompt for Initial Dafny Code Generation
SYSTEM
You are an expert Al assistant that writes Dafny programs. You excel at writing code with formally verified correctness, providing precise preconditions and postconditions, and finding the appropriate loop invariants to ensure all verification conditions are met.
TASK
Below the Python code:
python
<python.code>
Please translate this Python code into Dafny, ensuring:
1. Method Signatures: Each piece of functionality should be expressed as a Dafny method (or of methods) with well-defined signature.
2. Preconditions: Clearly state any 'requires" clauses for each method (e.g., array
length constraints, non-null references, numeric domain restrictions, etc.).
3. Postconditions: State the logical guarantees about the returned values or final state as 'ensures' clauses (e.g., correctness of returned results, absence of side effects, etc.).
4. Verification Details: Include all necessary loop invariants (or other verification hints) Dafny prove the postconditions, along with brief explanation. For example: Explain how you chose your invariants. Describe how they ensure the correctness of the loop.
Return the final Dafny code as a self-contained snippet that can be verified by Dafny as-is, with short explanation of how it connects the original Python functionality.
AT ASSISTANT
<he LLM's generated Dafny code with specifications here.>
Dynamic Debugging Prompt for Code Generation
SYSTEM
You are an expert Al assistant that writes and debugs Dafny programs. You excel at diag- nosing and fixing verification errors based on Dafny solver messages, while maintaining correct preconditions, postconditions, and loop invariants.
TASK
Below is the Python code:
python
<python.code>
And the Dafny code you previously provided (which tried to verify):
dafny
<main_spec>
16

---

Published as a conference paper at ICLR 2026
ran dafny verify dff and received this error message:
anaffyy result>
Can you please fix the main function specification so that it parses successfully? Output the corrected main function specification only, without any other text.
AI ASSISTANT
<The LLM's generated Dafny code with specifications here.>
G DATA CONTAMINATION ANALYSIS
To validate the novelty of DAFNYCOMP, conducted rigorous data contamination analysis against the widely-used MBPP dataset (Austin et al. (021). used to assess contamination in Python source data. We confirm that benehmark source data shows significant overlap, ensuring model performance reflects genuine reasoning capabilities rather than memorization.
Our analysis, focusing solely code, employs standard metrics: Exact Match detect ver- batim copies, and n-gram Jaccard Similarity to identify structurally similar code, We performed this analysis under four distinct configurations, the results of which are summarized in Table 7
Across all scenarios, we found zero exact matches. The n-gram Jaccard similarity remains neg- ligible, peaking at strong evidence that DAFNYCOMP is free from training data contamination.
0.0078 under the most aggressive settings. These findings provide
Table 7: Summary Data Contamination Analysis. The table shows results for four testing config- urations: A (Conservative) with minimal preprocessing: B (Default) with moderate preprocessing; C (Aggressive) with extensive preprocessing; and D (Holistic) for structure-level check. Across all configurations, results show zero matches and negligible n-gram similarity when compar- ing DAFNYCOMP source data against MBPP, confirming the benchmark's integrity.
Analysis Configuration
N-gram (n)
Exact Overlap
Max Jaccard
vs. sanitized-mbpp
A: Conservative
15 11, 13, 15
0.000078
0
B: Default
0.000389
0
C: Aggressive
9, 11, 13
0
0.007757
0
D: Holistic
11, 13,
0.000234
O
H PROMPT FOR EVALUATION
The prompt template used for evaluation shown in the following box. Note that all model outputs are used directly for Dafny verification.
Evaluation Prompt for Dafny Specification Generation
SYSTEM
You are an expert in Dafny. You will be given tasks dealing with Dafny programs includ- ing precise annotations. You should only return code body in all circumstances. No text is allowed.
TASK
Given Dafny program with function signature, preconditions, postconditions, and code, but with annotations missing. Please return a complete Dafny program with the strongest possible annotation (loop invariants, assert statements, etc.) filled back in. Do not explain or output any text. If you have to explain, put all explanations comments form. There
17

---

Published as a conference paper at ICLR 2026
should only be code body in your output. Please exactly the same function signature, preconditions, and postconditions. Do not ever modify the given lines.
Below is the program:
dafny
AI ASSISTANT
''dafny
<The LLM' s generated Daīny code with specifications here.>
I COMPUTE SETTINGS
Details of Compute Settings
Token budgets:
No hard constraint is imposed on the number of generated tokens. For each model, max_ output tokenn (or equivalent) is set to the largest value allowed by the provider, so that long reasoning traces are not truncated.
caasoinng-riiente (c.g., RI) and may follow long, itera- tive reasoning trajectories, so token usage largely emergent and model-dependent.
Many evaluated systems
Inference sampling:
Default decoding for most models: temperature = 0.7, top-p = 0.8.
04-MINI: provider default configuration, with fixed temperature and no top-p parameter.
Verifier retry policy:
Exactly one verification attempt per output.
SMT solver timeout: s0
If the solver does not finish within 60 s, the instance is treated as unsolved (not counted as a success).
18

---

Published as a conference paper at ICLR 2026
J VERIFER-IN-THE-LOOP MULTI-TURN SELF-REHINMMENT (FULL RESULTS)
Table 8: Full multi-turn results on the chain split (300 problems), We report Syntax Correct Rate and Verified Rate (end-to-end verifier success) each refinement turn T { (1,2,3). All values are percentages.
Syntax Correct Rate (%)
Verified Rate (%)
Model
T1
T2
T3
T1
T2
T3
DEEPSEEK-RI
85.67
94.33
95.00
0.33
5.67
9.67
DEEPSEEK-VE V3.1
49.00
91.67
96.00
0.00
1.67
5.33
DEEPSEEK-V3
74.33
95.00
96.00
0.00
2.67
5.00
QWQ-32B
44.67
56.67
61.67
0.00
0.00
1.00
FLEN2.5-CODER-22B--SSTTI
59.67
62.33
63.67
0.33
0.33
0.67
QWEN3-CODER-448 --NS-INTTUCT
78.33
93.33
95.67
0.33
0.67
3.00
GPT-4.1
46.67
86.67
92.67
0.00
1.33
3.33
GPT-40
94.00
98.33
98.67
2.00
4.67
04-MINI
37.33
92.00
96.00
0.00
2.00
9.67
Table 9: Full multi-turn results on the DAG extension (100 problems), We report Syntax Correct Rate and Verified Rate (end-to-end verifier success) each refinement T values are percentages.
(1) All
Syntax Correct Rate (%)
Verified Rate (%)
Model
T1
T2
T3
T1
T2
T3
DEEPSEEK-RI
$2.00
79.00
79.00
1.00
4.00
6.00
DEEPSEEK- V3. I-TERMINUS
59.00
94.00
98.00
1.00
3.00
7.00
DEEPSEEK-V3
67.00
96.00
98.00
0.00
4.00
4.00
QwQ-32B.DAR _DAG
29.00
42.00
49.00
0.00
0.00
2.00
TWEN2.5-CODER-32BB --NTRRUCCT
51.00
62.00
63.00
0.00
1.00
1.00
OWEN3-CODER-48000 -INS--UTTTTCT
69.00
95.00
97.00
0.00
1.00
4.00
GPT-4.1
26.00
77.00
96.00
1.00
2.00
4,00
GPT-40
69.00
99.00
99.00
0.00
0.00
2.00
04-MINI
22.00
76.00
93.00
0.00
1.00
4.00
19

---

Published as a conference paper at ICLR 2026
K
REAL EXAMPLES FROM BENCHMARK DATA
K.1
CHALLENGE 1: FORMAL. SPECIFICATION SYNTHESIS
The following examples demonstrate failures generating correct formal specifications, particularly preconditions and postconditions.
K.1.1 EXAMPLE 1.1: POSTCONDITION FAILURE (ORGLINPUT.ID. 2, SELF ID: 20001)
LLM Response:
method éigitSum (n: int) returns (sum: int)
requires n
ensures sun >= 0
temp n; sum :- 0;
while temp 0
invariant temp >=
invariant sum >= 0
sum :- sum (temp 10)
temp = temp / 10;
method countiargesstrraup_ (n: int) returns (result: int)
requires n >- ensures result >=
ensures result 1000
var counts :- new int [50];
var 1 : 0;
while 50
invariant <= 50
invariant forall j :: <= j i ==> counts 0
counts/il := 0; :
var maxCount :- 0;
var groupawithMaxCoum :- 0;
i :- 1; while
invariant
invariant maxCount >=
invariant CoupsWithMaxCoung >= invariant maxCount > 0 ==> groupsWithMaxxount >=
invariant forall j :: j 50 counts[j]
var ds :- digitSum ( if ds < 50 { countsids] countsids
it maxCount < countsida maxCount := dslnts/ds];
groupswithMaxCoun :- 1;
} else if maxCount ccunts [ds]
groupswithMaxcoont grwupswithhaeeee thMaxCount
i ==
result := groups/ithMaxConnt; if result == 0 result 1;
assert result
20

---

Published as a conference paper at ICLR 2026
if result 1000 result 1000;
method main_3node, (o: int) returns (result: int)
requires o >=
ensures result = -1 II result >= 2
cl countLargestGreup (399
var 02 if 02 sunoffulldiplee_25521cl))
var o3 :- nextireaterileeenn_55566 result o3;
if result != result
result := 2;
clsc
result -1;
Dafny Error:
ex. dfy (137, 0): Error:
postcendition could not be proved on this return
path
dxy (136, 25) Related location: this is the postcondition that could not be proved
21

---

Published as a conference paper at ICLR 2026
K.1.2 LLM Response:
EXAMPLE 1.2: MISSING PRECONDITION (ORG_INPUT_D: 6, SELF.ID: 60005)
method DigitSummn: (n: int) returns (sum: int)
requires >=
ensures sum >=
var num :- nj
sum :- 0;
while
invariant sum
invariant num >= 0
cecreases num
sum sum (num 1
num =- num 10;
method countLargesttGroo__1333 (n: int) returns (result: int)
requires n >= ensures result >=
var cigittumcounts map[l; var maxCount :- 0;
var i :- 1;
while <=
invariant <= i
invariant maxCount >=
invariant forall k :: in digitsumCounts --> digitSumCounts [k]
>=0 decreases - +
var digitSum :- Digit.Sumi)) (i);
var currentCount : if digitSum in digitsumCounts then digousus digitsumm else 0; SigitSumCounts
it currentCount sigitSunCounts l IdigitSum
currentCount 1]
maxCeunt | maxCount :- currentCount. 1;
i
result := 0;
var possiblesum := 0;
while possibleSum <= 9 4
invariant possibleSum >=
invariant result 0
invariant forall :: in digitSumCounts ==> digittSumCounts [k] >- >=0 0
decreases
possiblesum
it possibleSum in cigitSunCounts 4& digitSunCounts !possibleSun
== maxCount
result : result + 1;
possiblesum possiblesum 1;
if result = 0 { result 1;
method 4node 4 (o: int) returns (result: int) ensures result >=
22

---

Published as a conference paper at ICLR 2026
cl nextGreaterfllement_55 556(o)
var 02: int;
if o1 -1 {
o2 : 1;
else if al > 10000
02 10000;
} else
o2 := ol;
o3 countLargestGreup ( (
var clamped03: int;
if 03 1000000000000000 {
clamped03 :- 100000000000000000
else if o3 < -100000000000000 clamped03 else
-10000000000000000
clampod03 := o3;
var 04 smallestMumber 2165 (clamped03);
var Innul: int;
if 04 <
final Input :- 1;
else is 04 > 15
finalInput 15;
else
final Input :- o4;
result
Dafny Error:
ex (177, 3 Error: precondi ti for this cal1 could not be proved
23

---

Published as a conference paper at ICLR 2026
K.2
CHALLENGE 2: ENSURING CODE CORRECTNESS
These examples show implementation issues where the code logic doesn't I satisfy the specifications.
K.2.1 EXAMPLE 2.1: ARRAY BOUNDS ERROR (ORGJINPUTLI: 1, SELFJID 10002)
LLM Response:
method smallestNumber. 2165 (num: int) returns (result: int)
var neg : num < 0;
var if num < then -num else num;
var ent new int [10];
var i :- 0;
while < 10
invariant 0 <= <= 10
invariant ent Length 10 invariant forall ::
ent [k] 0
cnt 0; 0; i i
1/ rest of implementation
method main_5nodde (o: int) returns (result: int)
var o1 : smallestNumber_21 (c);
var o2 :- cl0msy_1006 (if ol <= 10000 then ol else 10000);
var c3 lastRemaiinin__390.co (c2)
var 04 getMoneyAAoout__37734 (if <= 200 then else 200);
var o5 numberOfWays_3193dd 04 100000 then 04 else 100000);
result := ob;
Dafny Error:
ex. ex.dfy(e (44, 30) : Error: this loop invariant could not be proved on entry Related message: loop invariant viclation
24

---

Published as a conference paper at ICLR 2026
K.2.2 LLM Response:
EXAMPLE 2.2: ASSERTION FAILURE (ORG.INPUT_ID: SELF_ID: 60001)
method DigitSum(n: (n: int) returns (sum: int)
requires >=
ensures sum >=
var num :- nj
sum :- 0;
while
invariant sum
invariant num >= 0
cecreases num
sum sum (num 1
num =- num 10;
method countLargesttrrop__1B9 (n: int) returns (result: int)
requires n >= ensures result >=
var cigittumcounts map[l; var maxCount :- 0;
var i :- 1;
while <=
invariant <=
invariant maxCount >=
invariant forall k :: in digitsumCounts --> digitSumCounts [k]
decreases -
var digitSum :- (oggitSummi) var currentCount : if digitSum in digitsumCounts then Simoutus digitsumm else 0; SugitSussounts
it currentCount sigitsunCounts I IdigitSum
currentCount 1l
maxCeunt | maxCount :- currentCount. 1;
ii=i+li
result := 0;
var possiblesum : 0; while pessibleSum <= invariant possibleSum invariant result 0
decreases * possiblesum
if possiblesum in digitsunCounts digitSunCounts Ipossiblesun maxCount
result := result 1;
possiblesum possiblešum + 1;
if result == 0 result 1;
method 4node 4 (o: int) returns (result: int) ensures result >=
25

---

Published as a conference paper at ICLR 2026
cl nextGreaterflementt_55 5
var 02: int;
if 01 -1 {
o2 : 1;
else if al > 10000
02 10000;
} else
02 == ol;
assert c2
var 03 :- countlarrgeettrroupl133999 (
var clamped03: int;
if o3 10000000000000000 {
clamped03 10000000000000000; else if < -1000000000000000 clampod03 := clsc
-1000000000000000;
clampedo3 o3;
var 04 :- smalless Nesbbberrr __ 65 (clamped03)
var finalInput: int;
if 04
FinalInput := 1;
} else if 64 15 {
finalinput : 15;
else final o4;
assert finalInput >= 1;
result countArrangeneet_5266... (EinaaInn))))))
Dafny Error:
ex (),14)) Error: assertion might not hold assort 02 1;
26

---

Published as a conference paper at ICLR 2026
K.3
CHALLENGE 3: PROOF GENERATION AND FORMAL REASONING
Examples demonstrating failures in generating proper loop invariants and formal proofs.
K.3.1 EXAMPLE 3.1: INVARIANT NOT MAINTAINED (ORG_INPUS ID: O, SELF _II: 6)
LLM Response:
method stringToInt (s: seq<char>) returns (result: int)
requires |s| > 0
requires forall :: <= i |sl =-> '0 <- s <- gg
ensures result 0
var res := 0; var i := 0;
while i |s|
invariant
|s|
invariant res >= 0
invariant res = stringtoonnteelprrsss.....
digit (sgitTonnt assert <= digit 9; res := res 10 + digit; i i 1;
return res;
method menotoneeincreessnnniittsssssss (38(n: int) returns (result: int)
requires 0 ensures result ensures result <= 1000000000
if 0 return 1;
var s :- intToString(m))
var i := 1;
while i 1sl & (sl <= digitToInt (slil)
invariant
lsl
invariant forall j :: <- j [s] --> for <- stil <= 19
{
i
if i lsl {
while > & < 1s] ss (s (s
invariant 0 <= i < Is]
invariant forall decreases
0 <= lsl ==> fo' <= s(J] <= 19'
var prevDigit :- digittoInt (s(i-1));
if prevDigit > 0
sli-l intToChar (prevDigit 1]
:= 1==1-1;
i i while [s]
invariant i |s|
invariant forall j :: 0 <= j x [sl ==> 10' <= s[i] <= 19
ali '9' := 1
if string(ooltt
1000000000
27

---

Published as a conference paper at ICLR 2026
return res; else { return 1;
method main_ Seit__nnde___ae_ (o: int) returns (result: int)
requires >= 0
ensures result >= 0
var :- anallesstratttrriaaiiooo.. if ol (
return 0;
:= monotone//oorrasiiiiii.iig..sa..
(o1))
var o3 := min0perations__2711e.2
return c3;
Dafny Error:
ex. dfy (25, 18) Error: this invariant could not be proved to be maintained by the loop
Related message: loop invariant viclation
28

---

Published as a conference paper at ICLR 2026
K.3.2
EXAMPLE 3.2: RECURSION TERMINATION (ORG.IPPUT_D: SELF_ID: 40003)
LLM Response:
Function gud_ (a: int, b: int): int
if then else ged_ (b, $ b)
method god (a: int, b: int) returns (result: int)
requi 0
ensures result fune (a, b)
var
var y :=
while y !-
invariant > a y >= 0
invariant god_ func (a, b) -- fed_ (x, y)
decreases
var temp : = x yi
x := yi
:= temp;
result x;
method
2318 (n: int) returns (result: int)
requires ensures result >=
1f == 1 { return 6;
var med :- 1000000007;
var dp :- new int [n + 1, 6, 6];
var k := 0;
while <=
invariant 0 cnnt
var i 0;
while < 6
invariant <= i <=
var ;= 0;
while <
invariant <= <= 6
dplk, i, j] : 0;
i :- j 1;
}
i
k == k 1;
var i
while
invariant <= i 6
var := 0;
while 6
invariant
var god (1 + 1, 3 1);
if - 1 65 !- j
dp 12, 1, ]] :- 1;
29

---

Published as a conference paper at ICLR 2026
3 j 1 1 + 1;
: while <= invariant <= k c
i ::00 while invariant <- 6
var j 0;
while <
invariant <= j 6
var g1 := god (1 + 1, + 1);
if gl == 1 2& != j
h while <
invariant <= h <= 6
g2 ged 1, 1);
if g2 & != i a2 != 1
dp (k, i, 31 : (dolk, 1, j1
dp [k 1, h,
11 $ mod;
h
3 : j 1;
== 1;
}
k = k 1;
var ans :- 0;
:= 0;
while < 6
invariant <= i <= 6
invariant ans >= 0
var J := 0;
while j< <
invariant <= j <= 6
invariant ans
ans := (ans apln, 1, 111 8 mod;
3 1= + 1;
result := ans;
method main_3node_2 (o: int) returns (result: int)
requires o >=
ensures result >=
var ol clumsy_1006
var o2 : numSquares_2790 (ol))
var o3 := distinct 2318 (o2);
result o3;
30

---

Published as a conference paper at ICLR 2026
Dafny Error:
ex. dfy (3, 34) Error: cannot prove termination; try supplying decreases
clause if == 0 then a else fud_ (b, $ b)
31

---

Published as a conference paper at ICLR 2026
K.4 CHALLENGE 4: SPEC-CODE-PROOS ALIGNMENT
These examples show misalignment between specifications and implementation, particularly method vs. function usage.
K.4.1 EXAMPLE 4.1: METHOD EXPRESSION CONTEXT (ORG 0, SELFJID: 5)
LLM Response:
method stringToInt (s: seqschar>) returns (result: int) requires res forall i <= < [s| =-> f0' <= [ii] <= gg ensures result
var res 0; var i := while lsl
invariant <= <= |s|
invariant res >= 0
invariant res = if i == 0 then 0 else stringToInt (.])
res := res 10 (s
i == 1 1;
result res:
method monctoneIn hww.eeeeinnsiiiiiiinnnnnna (n: int) returns (result: int) requires >= 0 ensures result ensures result <= 1000000000
if n == - ( return 1;
var s :- int
var :- 1; while < |sl & digitToInt (sli-1)) <= digitToInt (s(il) (slii)
invariant <= i <= lsl
invariant forall i :: :: < Isl
s131 1
i 1-mitl
if i
while i && < |s] is (s (s())
invariant 0 <= i 1s]
invariant forall j :: <= j |sl ==> for <= s[3] <= 19
decrcases i
var prevDigit :- digittoInt (s(i))));
if prevDigit 0 :- s[i-1
Cpr 1]];
i i 1;
i :- i 1;
while i [s]
invariant i |s|
invariant forall j :: 0 <= j < [s| ==> 10' s[j] <= 19'
== sli := '9";; i i
var res := stringToontt (s);
if res >- 1 && res <= 1000000000
res;
32

---

Published as a conference paper at ICLR 2026
else { return 1;
method main_3node_22 (o: int) returns (result: int) requires >= ensures result >= 0
if amalesstaactoriaaiiooo.0.....0
return 0;
o2 := m (ol);
:= Siioperaations_27111 (c2);
return c3;
Dafny Error:
ex. df.dff (26, 43) Error: expression is not allowed to invoke a method ( stringToint) invariant res if i 0 then 0 else stringToint (s[..1])
33

---

Published as a conference paper at ICLR 2026
K.4.2 EXAMPLE 4.2: FUNCTION VS METHOD CONFUSION (ORG.NPPUT.D; 16, SELF
160000)
LLM Response:
method isBeautiful (n: int) returns (result: bool)
// Implementation for checking if number is beautiful result true; // Simplified
method next 1tw.uuoorueee0000999908000 (n: int) returns (result: int)
var :-
while true
invariant x >- +
invariant forall :: n <= x ==> !isBeautiful iful (y)
var beautiful := isBeautiful ;x);
if beautiful {
result x;
return;
x x
method 2ain_5node__(o. (o: int) returns (result: int)
ol reverse_ [
result := 1;
return;
var := LastRemaining_
var o3 :- closestFair 22131ir__2111_ (c2);
var 04 : nextGrreaterFlemett_BB5
if 04 == -1 {
result
return;
var o5 :- BextBeaut Number (o4)
result c5;
Dafny Error:
dxy (248, 46); Error: expression is al lowee invoke method
isHeauti ful)
invariant forall y :: n <= y ==> !isBeautifal (y)
34

---

Published as a conference paper at ICLR 2026
K.S CHALLENGE 5: COMPLEXITY AND SCALABILITY ISSUES
Examples where complex programs or large data structures cause verification failures.
K.5.1 EXAMPLE 5.1: LARGE ARRAY ALLOCATION INRUTINPUT.NN SELF 10002)
LLM Response:
method smallestNumber_22 (num: int) returns
(result: int)
var neg num
then else num; var ent := new int [101;
var if
var i := 0;
while < 10
invariant 0 <= <= 10
invariant Length 10
invariant forall k :: <= k < i =-> cnt 0
[i] 07
i
// Count digits
var temp := while temp
invariant temp >
invariant forall k :: <- k < 10 --> ent [k] >-
decreases temp
var digit := temp 10; ent [digit] : ent [digit] 1; temp = temp / 10;
// Construct result result :- 0;
i : if neg then 1 else 0; while 10
invariant
10
invariant result 0
decreases 10
while entli]
invariant [nt 0
invariant result >= 0
decreases ent
result result 10
ent [i] :- ent 1;
i
if neg
result :- -reuult;
method
(o: int) returns (result: int)
var ol smallestNumber_20 226 (c);
var o2 clumsy_1006 (if ol 10000 then o1 else 10000);
var o3 := LastRemaining__90.co (c2);
var 04
1ttXooneyAmoutt_33///18 o3 <= 200 then o3 else 200);
var o5 numberofrwass31183 (i c4 <= 100000 then 04 else 100000)
result o5;
35

---

Published as a conference paper at ICLR 2026
Dafny Error:
ex. ex.dły (44, 30) Error: this Loop invariant could not proved entry Related message: loop invariant viclation invariant ans >
36

---

Published as a conference paper at ICLR 2026
K.6 CHALLENGE 6: AVOIDING TRIVIAL SOLUTIONS
Note: This category had limited representation in the dataset. The theoretical examples that would fit this category include:
Overly permissive postconditions like ensures true
Methods with missing functional specifications
Weak preconditions that don't prevent edge cases
Specifications that allow degenerate solutions
Missing bounds on return values
Incomplete coverage of input space
Trivial loop invariants that don't 't capture semantics
Weak assertions that don't prevent bugs
Generic error handling without specific constraints
K.7 CHALLENGE 7: LIMITED TRAINING IN FORMAL DOMAINS
Examples showing basic syntax errors and misuse of Dafny constructs.
K.7.1 EXAMPLE 7.1: PARSE ERROR (ORGLINPUT_D: 0, SELF-ID: 0)
LLM Response:
function digitToInt (c: char): int requires '0" <= <= 19 ensures digitToInt (e)
(c as int) - ('0' as int)
function Tntharr int): char requires <= i <= 9 ensures '0' <= intToChar (i) <= 19r
(1'0' as int) i) as char
method stringToint (s: string) returns (result: int) requires forall ensures result
<= i Isl '0' s[i] rg
var res :- 0; 1 0;
while |sl
invariant
|s|
invariant res >= 0
10 (
1;
result :- res;
method monotoneeiccreasinnniiiiii (n: int) returns (result: int)
requires n >=
ensures 1 <= result <= 1000000000
if return 1;
var int var digits s;
37

---

Published as a conference paper at ICLR 2026
var i
while Idigits| & digitreint (digitss [i-11) digitToInt (digits [1
1)
invariant 1 <= 1 <= Idigits|
invariant forall :: <= |digits] ==> Orr <= digits <=
'g'
invariant forall j,k :: 0 <= i k < ==> digitfonnt (digitts <= digitToInt (digits(k))
i ==
if < Idigits| {
while i 0 & i < Idigitsl & digitroint (d-gits(i-1))
digitToInt (di)
invariant <= |digitsl
invariant forall j :: <= i Idigitsl ==> '0' digits(3) '9/
decreases
var prevDigit digitTeInt (digits
if prevDigit 0 {
digits :- digits(i-1 :- intToChar (prevDigit. 1)];
i i:=i-
i -- i
while i Idigits!
invariant 0 <= i <= Idigits!
invariant forall :: <= Idigitsl ror digits(j) 'g'
digits digitsll := 1 i 1 + 1;
var res :- (Eringtoont
if res -- 0
1;
if res > 1000000000 1 return 1000000000;
return res;
method 2iin_3node_2(o: int) returns (result: int)
requires C >= ensures result
var el smallestFaatorriatiio... (25
var o2 Innotaone
(o1);
var o3 (astRemaiiingg_330.co (c2);
return o3;
Dafny Error:
ex. dły (22, 12): Error: invalid ForLoopStnt reads
22
method
stringToInt string) returns (result: int)
parse errors detected in ex.dfy
38


Reviews:
[{"review_id": "2dlmYfu2GO", "reviewer": "Reviewer_M3WA", "summary": "The paper presents DAFNYCOMP, a collection of 300 programs used to provide stress on the compositional generation of specifications in Dafny by chaining 2-5 functions with real dependency of data. In 13 frontier LLMs, syntax is well-behaved (96%-99% Pass at t8) but the verification is still low (3.7 percent overall; best model about 7 percent), indicating that there is a big discrepancy between syntax and semantic correctness. The failures the authors consider are three in general, namely specification fragility, implementation-proof misalignment, and reasoning instability.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Timely evaluation of compositional verification\n- Clear, reproducible benchmark spec with Dafny ground truth.\n- Concrete numbers that expose a large syntax verification gap. \n- Insightful error analysis", "weaknesses": "- Data programs and specs are generated using an LLM from a single family employed in evaluation, which might be biased toward one style and negatively impact others.\n- The evaluation is limited to topologies: only chains are considered, whereas real systems include trees, DAGs, shared states, and more complex dependencies.\n- There is a lack of evaluation on more advanced test-time scaling strategies, such as multi-turn self-refinement.\n- The evaluation described in the main text does not consistently standardize or control details about compute and prompting across models (tokens, temperature, verifier retry policy).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SOtoaUA4Dg", "reviewer": "Reviewer_KdjF", "summary": "The paper introduces DafnyComp, a benchmark to evalaute LLMs in . Unlike existing benchmarks, DafnyComp focuses on programs consisting of multiple functions interacting with each other. Experiments on 13 frontier LLMs show that this leads to significantly a more challenging and realistic benchmark.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The paper introduces DafnyComp, a benchmark for evaluating LLMs on the generation of compositional specifications in Dafny. Existing benchmarks for verifiable code generation only test problems with a single function. DafnyComp focuses on programs composed of multiple interacting functions with necessary data dependencies, addressing a significant gap in the real-world applications of formal verification. \n* A comprehensive set of LLMs is evaluated on DafnyComp. Results show that multi-function compositional problems present a major challenge to LLMs compared to simpler problems.\n* The authors performed a detailed and insightful analysis of LLMs' failures on DafnyComp.", "weaknesses": "* The benchmark is relatively small (300 programs). Is it possible to scale the benchmark construction method in Sec. 3 to collect more programs?\n* The paper focuses on Dafny. It would be great to discuss potential extensions to other systems such as Lean and Verus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTwo6tFz4O", "reviewer": "Reviewer_2F9W", "summary": "This paper presents a benchmark set, DafnyComp, to evaluate LLMs ability to perform compositional\nreasoning. The benchmark set contains Dafny programs that are compositions of 2-5 functions. The goal\nis to prove these programs correct. To do this, one has to generate pre and postconditions (requires\nand ensures) for each function in a way that they compose to yield a proof for the main program.\nThe LLMs task is to annotate the functions enough to enable Dafny verifier to complete the proof \n(by discharging the verification conditions using an SMT solver).\n\nThe main result shows that LLMs across the board are really bad at doing this annotation task. \nThey are able to do so for single functions to some extent, but really struggle to do that for\nmultiple functions in a way that the annotations compose.", "questions": "Please refer to the Weaknesses section for the questions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strengths:\n1. DafnyComp is a valuable benchmark since state-of-the-art LLMs currently perform very poorly\n(about 3% for the best models).\n2. DafnyComp tests the ability to think compositionally, which is useful to evalaute.\n3. DafnyComp is also interesting because models can generate correct syntax (for annotations) with\nhigh probability, so it does demonstrate that models really struggle with compositional reasoning\n(and not getting the syntax right for a potentially low-resource language).\n4. The pipeline for constructing DafnyComp guarantees that the benchmarks are all high quality\n(i.e. they all can be formally proved correct.)", "weaknesses": "Weaknesses:\n\n1. The benchmarks in DafnyComp are artificial - the individual functions are natural, but the\ncompositions are contrived (authors, please correct me if I am wrong.)  So, the 92% drop from\nsingle-function to multi-function programs can also be attributed to the fact that the single-function\nprograms are natural programs that people may write and care for, and the multi-function programs\nare more arbitrary. In other words, if I were to write completely unnatural single function programs,\nwouldn't LLMs performance already deteriorate? That puts the main claim into question: maybe it is \nnot the lack of compositional reasoning, but just that the compositional programs are unnatural. \nIf the single-function programs were unnatural, or the compositional programs were natural, then \nwould we get different conclusions?\n\n2. The paper makes very strongly worded claims all through the paper, but there are a lot of \nfactors that could be responsible for the observed behavior. Many such factors are overlooked and\nnot discussed.\n\n 2.1 If the function call graph is linear (which means it can be topologically sorted), then couldn't\nwe turn the multi-function benchmarks to single-function benchmarks? If so, then would LLMs \nperform better on these derived single-function benchmarks? And would this finding impact the main\nclaims of the paper?\n\n 2.2 Is Dafny prover guaranteed to prove things if they are provable -- couldn't the SMT solver fail -- \ne.g. time out -- even when the verification condition holds true? Would this not somehow impact\nthe conclusions? For example, purely hypothetically, maybe the LLM is generating the correct \nannotations and the SMT solver is failing to prove them? I know that this is unlikely, but how \nare we guarding against that possibility?\n\n3. There is not enough explanation for how benchmarks (and their correctness assertions) are\ngenerated.\n\n 3.1 It is unclear how the assertions of the composed program are generated. Line 215 says \"We execute\neach composed program against the reference unit tests from LeetCodeDataset\" -- but my understanding\nis that you are taking individual functions from LeetCodeDataset and then creating new compositions\n(with 2-5 functions). So, LeetCodeDataset would not have reference unit tests for these composed \nprograms. There is clearly some misunderstanding here on my part.\n\n 3.2 Some of these misunderstandings could be avoided if you had a running example either in the main\npaper or at least in the Appendix. It would be good to have an example benchmark in DafnyComp. My best\nSource is Appendix G. But most of the programs there have 1 function -- clearly, they can't be part\nof DafnyComp. So, that adds to more confusion. G.4.2 is the first place where there is a program that\nuses multiple functions. It has one function, reverse_7, that is unused, that is a but confusing, but\nmaybe DafnyComp has benchmarks that contain unused functions. But what is really confusing is that\n\"nextBeautifulNumber_769\" is not a linear composition of two functions, but a loop over the \"isBeautiful\"\nfunction. \n\n 3.3 The examples in Appendix G just give some code snippet and a Dafny error. That is not helpful. \nMore explanation is needed. I could not understand most examples. For example, take G.1.1: sum >= 0\nis the only postcondition there. It is clearly implied by the loop invariant (because it contains\na conjunct sum >= 0) -- but Dafny error says \"a postcondition could not be proved...\" ?\n\nOther questions:\n\n- Contribution (ii) on Line 111 is related to a single function/procedure, right? So, it is not\nrelated to compositions, is that true?\n\n- Does DafnyComp have 300 benchmarks or 900 benchmarks (Table 2)? Or Table 2 is analyzing 3 runs for \neach of the 300 benchmarks?\n\n- Line 422-423: \"not a typo but a systematic failure to...\" -- what do you mean by \"not a typo\"?\n\n- Line 427-428: \"attention mechanisms excel at ...implementations.\" I think there is no experiment\nin the paper that provides evidence of this claim, and I am not even sure what this claim means.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q78fJ1NNBI", "reviewer": "Reviewer_mpGN", "summary": "The paper introduces DAFNYCOMP, a benchmark targeting compositional formal verification for LLMs. It assembles multi-function programs (2–5 functions). Across a series of frontier models, the paper finds a big gap between high syntax success and very low verification results on the benchmark. The authors analyze failures into three primary failure modes: specification fragility, implementation-proof misalignment, and reasoning instability.", "questions": "1. Why do you choose LeetCode problems as the pool programs? Is it possible to switch to other datasets as the initial pools?\n2. Will the compositional strategy reflect the distribution of natural real-world programs?\n3. How will this methodology translate to other frameworks such as Verus? Does such a similar dataset / idea exist in the Verus or other frameworks' prior work? If not, will your work's pipeline generalize to other frameworks?\n4. How to improve the performance of your benchmark? Consider adding a baseline to improve the results", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The evaluation is thorough, covering a wide range of models\n2. The failure taxonomy is insightful, well-articulated, grounded with examples, and provided with the frequencies.\n3. The benchmark is challenging, exposing the limitations of current LLMs.", "weaknesses": "1. The benchmark is only done on Dafny. We don't know whether results and claims will still hold for other frameworks such as Verus / Boogie.\n2. The synthesis pipeline uses the LeetCode programs. LeetCode skews toward short, single-goal algorithms, but may deviate from the real-world software use cases.\n2. There is a limited trial in trying to solve the challenges. E.g., adopting a verifier-in-the-loop repair / agentic setup would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)