From Informal to Formal – Incorporating and Evaluating LLMs on Natural Language Requirements to Verifiable Formal Proofs
Authors: Jialun CAO, Yaojie Lu, Meiziniu Li, Haoyang Ma, Haokun Li, Mengda He, Cheng Wen, Le Sun, Hongyu Zhang, Shengchao Qin, Shing-Chi Cheung, Cong Tian
Accepted at: acl25

From Informal to Formal – Incorporating and Evaluating LLMs on Natural Language Requirements to Verifiable Formal Proofs

Jialun Cao1, Yaojie Lu2, Meiziniu Li1, Haoyang Ma1, Haokun Li3, Mengda He3, Cheng Wen3, Le Sun2, Hongyu Zhang4, Shengchao Qin3,5, Shing-Chi Cheung1, Cong Tian5
1 The Hong Kong University of Science and Technology
2 Institute of Software, Chinese Academy of Sciences
3 Guangzhou Institute of Technology, Xidian University
4 Chongqing University 5 ICTT and ISN Laboratory, Xidian University
jcaoap@cse.ust.hk, luyaojie@iscas.ac.cn

Abstract
The research in AI-based formal mathematical reasoning has shown an unstoppable growth trend. These studies have excelled in mathematical competitions like IMO and have made significant progress. However, these studies intertwined multiple skills simultaneously—problem-solving, reasoning, and writing formal specifications—making it hard to precisely identify the LLMs’ strengths and weaknesses in each task. This paper focuses on formal verification, an immediate application scenario of formal reasoning, and breaks it down into sub-tasks. We constructed 18k high-quality instruction-response pairs across five mainstream formal specification languages (Coq, Lean4, Dafny, ACSL, and TLA+) in six tasks by distilling gpt-4o and evaluated against ten open-sourced LLMs, including recent popular DeepSeek-R1. We found that LLMs are good at writing proof segments when given either the code, or the detailed description of proof steps. Also, the fine-tuning brought about a nearly threefold improvement at most. Interestingly, we observed that fine-tuning with formal data also enhances mathematics, reasoning, and coding capabilities. Fine-tuned models are released to facilitate subsequent studies at https://huggingface.co/fm-universe.

1 Introduction

“The more we formalize, the more of our implicit knowledge becomes explicit.”

— Terence Tao (Tao, 2024)

As AI-based formal mathematical reasoning reached an inflection point (Yang et al., 2024b), significant attention and progress in this field have been observed. AlphaProof (AlphaProof and Teams, 2024) achieved silver medal level in the International Mathematical Olympiad (IMO), Alpha-Geometry (Trinh et al., 2024) specialized in proving Euclidean geometry theorems. As reported, the number of publications in this field nearly doubled in 2023, indicating an unstoppable growth trend (Li et al., 2024). As Fields Medalist Terence Tao imagined, “In the future, instead of typing up our proofs, we would explain them to some GPT” (Tao, 2024).

However, most current benchmarks cannot precisely reflect the capability to convert informal proofs or requirements in natural language into formal proofs. Most of these benchmarks take mathematical problems (AlphaProof and Teams, 2024; Trinh et al., 2024; Welleck et al., 2021) or theorems to be solved (Yang et al., 2023; Welleck et al., 2022; Yang and Deng, 2019) as input, and informal or formal proofs (or parts of proofs) as output. However, these end-to-end benchmarks assess multiple capabilities (e.g., problem-solving, mathematical reasoning, formal specification writing) in an intertwined manner, making it difficult to isolate and observe LLMs’ true capabilities in writing formal proofs or models for verification.

Therefore, we break down the process from informal requirements to formal verifiable proof, as shown in Figure 1. Inspired by the code generation (shown in blue) which translates a description of implementation into executable code (Chen et al., 2021; Austin et al., 2021a), the formal reasoning process (shown in green) can be seen as translating an informal requirement into a verifiable formal proof or checkable formal model 1. Particularly, we decompose this process and formulate six tasks (Figure 2). By doing so, the intertwined capabilities can be separated and individually assessed, providing a clearer understanding of LLMs’ strengths and weaknesses in each task.

Scope and Targets – We focus on formal verification (Appel, 2011; Klein et al., 2009; Leroy et al., 2016; Hawblitzel et al., 2014) because it is an immediate application scenario of formal mathe-

1For ease of expression, we generally refer to verifiable formal proofs and checkable models as “proofs” for the sake of presentation simplification.
From Informal to Formal

<table>
  <tr>
    <th>Requirement</th>
    <th>Informal Proof/Model in natural language</th>
    <th>Verifiable Formal Proof/Model in formal specification</th>
  </tr>
  <tr>
    <td>Implements a binary search.<br>Given a sorted sequence and a target value, returns the index of the target value in the sequence, or 0 if not found</td>
    <td>Start by modeling the algorithm’s behavior in a formal specification language, which involves defining variables like low and high, and detailing the search procedure using control constructs such as while. Next, specify the essential properties (i.e., accurate target index retrieval or correct non-existence confirmation) for validation.</td>
    <td></td>
  </tr>
  <tr>
    <td>Informal Implementation<br><br>Start by setting low and high pointers at the beginning and end of the sequence. Repeatedly adjust them until the target is found or reach 0 if not found.</td>
    <td>Informal Proof/Model Segment (in natural language)<br><br>Define a set that contains all sequences of integers that are sorted in non-decreasing order.</td>
    <td>Formal Specification Writing<br><br>Define the initial state of the system. Initialize variables such as low pointing to the beginning and end indices of the sequence.</td>
  </tr>
  <tr>
    <td>Executable Implementation (Code)<br><br>def binary_search(arr, target):<br>low = 0<br>high = len(arr) - 1<br>while low <= high:<br>mid = (low + high) // 2<br>if arr[mid] == target:<br>return mid<br>elif arr[mid] < target:<br>low = mid + 1<br>else:<br>high = mid - 1<br>return -1</td>
    <td>Define the transition action. It calculates the middle index of low and high, and retrieves the value of middle index ...</td>
    <td>Formal Proof/Model Segment (in formal specification)<br><br>sortedSeqs == {S | S \subseteq \mathbb{Z}^n, S \text{ is sorted}}<br><br>low = 0<br>high = len(arr) - 1<br>while low <= high:<br>mid = (low + high) // 2<br>if arr[mid] == target:<br>return mid<br>elif arr[mid] < target:<br>low = mid + 1<br>else:<br>high = mid - 1<br>return -1</td>
  </tr>
  <tr>
    <td>Code Generation<br>Convert designs into executable code.</td>
    <td>Relation: Formal proof verifies that code meets its specified requirements mathematically.</td>
    <td>Formal Proof/Model Generation<br>Defines precise system behaviors formally.</td>
  </tr>
</table>

Figure 1: The Illustration of Formal Proof Generation and Its Relation with Code Generation.

matical reasoning and the correctness of the output can be verified mechanically. In this paper, we mainly explore four research questions (RQs):

RQ1. How well do LLMs perform in various formal verification tasks? After decomposing the formal verification task into subtasks, we explore LLMs’ initial performance in these tasks with zero-shot and few-shot, investigating the strengths and weaknesses that vary between LLMs and tasks.

RQ2. Do LLMs show variability in their capability across different formal specification languages? When mathematicians and proof engineers consider using LLMs to assist in formal verification, they often face uncertainty about which formal specification language is best supported by LLMs. This RQ is designed to provide hints on it.

RQ3. Can fine-tuning improve LLMs’ performance in formal verification? Although recent efforts have been made to fine-tune models (Wang et al., 2024; Yang et al., 2023), these LLMs are typically fine-tuned with single formal languages instead of multi-lingual (e.g., combining Coq, Lean, etc.) (Yang et al., 2024b). Therefore, we instruction fine-tuned (Wei et al., 2021; Sanh et al., 2021) three base LLMs to see whether our constructed fine-tuning dataset FM-ALPACA could improve their capability in formal verification tasks.

RQ4. Can fine-tuning with formal verification data benefit other related tasks (mathematics, reasoning, code)? As recent works have shown LLMs’ potential transferability of skills (Tihanyi et al., 2023) we thus extend our study to see if models fine-tuned on formal data could show enhanced capabilities in mathematics, reasoning, and coding.

To facilitate the study, we constructed 18k high-quality instruction-response pairs across five formal specification languages (i.e., Coq, Lean4, Dafny, ACSL, and TLA+) in six formal-verification-related tasks by distilling gpt-4o inspired by prior work (Wang et al., 2024; Ding et al., 2023; Wang et al., 2022), then split them into 14k instruction fine-tuning data (FM-ALPACA) and 4k benchmarking data (FM-BENCH). In particular, we provide executable contexts for all these formal specifications and automated validation scripts to validate the correctness of the generated formal proofs inspired from the prior work’s artifact preparation (Jimenez et al., 2023). Finally, we release the fine-tuned LLMs based on three base models at https://huggingface.co/fm-universe.

Interestingly, there has been recent discussion on the topic of domain transfer (Yang et al., 2024b), particularly the transfer of knowledge from other domains such as coding and reasoning to formal domains in order to increase LLMs’ reliability (Spiess et al., 2024), and the anticipated potential of AI in enhancing formal verification processes to support mathematical proofs (Tao, 2024; Tao et al., 2023). Our experimental results could potentially provide empirical support for these hypotheses or offer directions for further experimental inquiries.

The contribution of this paper includes:
✓ Problem Formulation: We decompose the formal verification process into six essential tasks. By doing so, the intertwined capabilities can be separated and individually assessed, providing a clearer understanding of LLMs’ strengths and weaknesses in each task.

✓ Dataset and Benchmark: We constructed 18k high-quality instruction-response pairs across five mainstream formal specification languages (i.e., Coq, Lean4, Dafny, ACSL, and TLA+) in six formal-verification-related tasks by distilling gpt-4o. They are split into a 14k+ fine-tuning dataset FM-ALPACA and a 4k benchmark FM-BENCH.

✓ Executable context and automated validation mechanism: We provide a Docker container equipped with necessary scripts to facilitate the evaluation of FM-BENCH, significantly lowering the entry barrier for this scenario and making subsequent contributions easier.

✓ Insight and Vision: We fine-tuned several models on FM-ALPACA and observed promising benefits to not only the formal verification tasks, but also mathematics, reasoning, and coding. Our experimental results provide empirical support for the potential of LLMs’ capability transfer and hope to shed some light on future research.

2 Task Formulation

Figure 2 illustrates the six sub-tasks. We elaborate on them in detail as follows.

Task 1. Requirement Analysis (abbrev. ReqAna). Requirement analysis (Davis, 1990; Anton, 1996; Grady, 2010; Jin, 2017) is a critical and long-standing research area in software engineering. It facilitates collecting, identifying, categorizing and modeling the users’ needs and expectations using various techniques (Taggart Jr and Tharp, 1977; Deeptimahanti and Babar, 2009; Javed and Lin, 2021; Wang et al., 2021; Jin et al., 2024; Zhou et al., 2022). In this paper, the requirements are the descriptions in natural language (English) (Jin et al., 2024) that details the requirements of the verification/modeling goal and an overall description of the proofs/models. The task is to analyze and break down the final goal into detailed steps described in natural language. The natural language used in this paper is English.

Task 2. Full Proof Generation (abbrev. ProofGen). This task formalizes a requirement in natural language into verifiable proofs or models written in formal specification languages, similar task formulation to existing works (Fatwanto, 2012; Zhou et al., 2022; Davril et al., 2013).

Task 3. Proof Segment Generation (abbrev. SegGen). Unlike ProofGen, which requires generating complete proofs/models, SegGen provides more detailed descriptions in natural language and requires LLMs to write less. Given a text description articulating how to implement the proofs/modeling, the task outputs a segment written in the formal specification that serves as a component in the complete proof/model. This task formulation is similar to prior work (Wang et al., 2024; Wu et al., 2022; Jiang et al., 2022) and similar to the formulation of code generation (Chaudhary, 2023; Sun et al., 2024; Welleck et al., 2022, 2021).

Task 4. Proof Completion (abbrev. ProofComplete). Similar to code completion (Raychev et al., 2014; Husein et al., 2024; Svyatkovskiy et al., 2019; Dakhel et al., 2023), ProofComplete suggests the suffix of the given prefix, similar to prior work (Song et al., 2024). Note that in order to prevent LLMs from deviating from the original verification goal, we also provide the requirement in our evaluation, although it is not compulsory for this task formulation.

Task 5. Proof InFilling (abbrev. ProofInfill). Given a proof/model with a mask in the middle, the task requires LLMs to fill proper formal specifications so that the completed proofs/models can pass the verifier. This formulation is the same as code infilling (Fried et al., 2022). Also, similar to ProofComplete, we provide the requirement in our evaluation during the infilling to prevent LLMs from deviating from the original verification goal.

Task 6. Proof Generation from Code (abbrev. Code2Proof). In addition to generating formal specifications from natural languages, formal specifications can also be generated from code if the verification goal is the property of a given program. In this paper, we focus mainly on specifications in form of code annotations (Baudin et al., 2021; Hatcliff et al., 2012), expressing specifications (e.g., pre-/post-condition, loop invariants) that help one to verify that (part of) a program satisfies certain properties. The task takes the code with properties to be verified as input and outputs the code with generated annotated formal specifications. Similar task formulation can be found in recent works (Wen et al., 2024; Ma et al., 2024).
Requirement with Informal Specification
Implements a binary search.
To verify a binary search, it starts by modeling the algorithm’s behavior in a formal specification language, ... specify the essential properties (i.e., accurate target index retrieval or correct non-existence confirmation) formulation.

Executable Implementation (Code)
def binary_search(arr, target):
    low = 0
    high = len(arr) - 1
    while low <= high:
        mid = (low + high) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            low = mid + 1
        else:
            high = mid - 1
    return -1

1 Requirement Analysis (ReqAna)
2 Full Proof Generation (ProofGen)
3 Proof Segment Generation (SegGen)
4 Proof Completion (ProofComp)
5 Proof Infilling (ProofInfill)
6 Proof/Model Generation from Code (Code2Proof)

Informal Proof/Model Segment
1 Define a set that contains all sequences of integers that are sorted in non-decreasing order.
2 Declare the initial state of the system. Initialize variables such as high pointing to the beginning and indices of the sequence ...

Formal Proof/Model Segment
3 Formalize the requirement, TLA+ code, ACSL code, etc.
4 Incomplete Formal Proof/Model
5

Verifiable Formal Proof/Model
Verify by verifier/checker

Formal Specification Languages
Dafny Lean4 ACSL TLA+

Figure 2: Six tasks towards Informal to Formal Verification

3 Data Construction

3.1 Formal Specification Language Selection

In this study, we consider five formal specification languages that can be used for formal verification, including Coq (Huet, 1986), Dafny (Leino, 2010), Lean4 (Moura and Ullrich, 2021), ACSL (ANSI/ISO C Specification) (Baudin et al., 2021) and TLA+ (Yu et al., 1999; Lamport, 2002). We selected them in order to cover various verification paradigms (i.e., theorem proving and model checking) and verification scenarios (e.g., mathematical reasoning and program verification).

First, for interactive theorem provers which are suitable for developing rigorous mathematical proofs, we consider Coq (Huet, 1986) and Lean4 (Moura and Ullrich, 2021) because Coq has been extensively used in academia and research for proving mathematical theorems and in formal verification of software for a long history, while Lean4 garnered considerable attention from the mathematical community (Wang et al., 2024; Tao et al., 2023; Avigad et al., 2020) recently. Second, for programming languages with built-in specification, we consider Dafny (Leino, 2010) and ACSL (Baudin et al., 2021; Cuod et al., 2012) because they seamlessly integrate specifications (e.g., pre-/post-conditions, loop invariants) within the code, ensuring the correctness through embedded assertions and conditions. Lastly, for model checking (Jhala and Majumdar, 2009; Clarke, 1997), we consider TLA+ (Yu et al., 1999) since it is a representative math-based formal language for modeling algorithms and programs such as concurrent and distributed systems.

3.2 Data Preparation

The workflow of data preparation for FM-ALPACA and FM-BENCH is illustrated in Figure 3. The workflow begins with the data collection, where formal proofs in the desired formal specification languages and related configurations and dependencies are gathered from open-source repositories in Github. Then, formal proofs are extracted from the collected repositories. Next, the proofs go through the data quality assurance check by execution, the proofs that cannot be verified successfully are filtered out. The remaining ones are split into segments (e.g., definition of functions or conditions).

Given the impracticality of manually writing descriptions for all the collected formal proofs, we leveraged distilled GPT4 (gpt, 2023) to gen-

![A diagram showing six tasks towards Informal to Formal Verification, including requirement analysis, full proof generation, proof segment generation, proof completion, proof infilling, and proof/model generation from code.](page_120_120_1208_482.png)

![A flowchart illustrating the data preparation process, including extraction from open-source repositories, configuration & dependencies, task-wise data pairing, and construction of data for fine-tuning and evaluation.](page_1040_120_384_482.png)

Figure 3: The Illustration of Data Preparation.
<table>
  <tr>
    <th rowspan="2">Spec</th>
    <th colspan="3">Num of Proofs</th>
    <th colspan="2">Num of Segments</th>
  </tr>
  <tr>
    <th>Total</th>
    <th>FM-ALPACA</th>
    <th>FM-BENCH</th>
    <th>Total</th>
    <th>FM-ALPACA</th>
    <th>FM-BENCH</th>
  </tr>
  <tr>
    <td>Coq</td>
    <td>2,126</td>
    <td>1,683</td>
    <td>443</td>
    <td>14,939</td>
    <td>11,638</td>
    <td>3,301</td>
  </tr>
  <tr>
    <td>Lean4</td>
    <td>1,165</td>
    <td>919</td>
    <td>241</td>
    <td>1,579</td>
    <td>1,261</td>
    <td>317</td>
  </tr>
  <tr>
    <td>ACSL</td>
    <td>544</td>
    <td>346</td>
    <td>118</td>
    <td>765</td>
    <td>598</td>
    <td>167</td>
  </tr>
  <tr>
    <td>Dafny</td>
    <td>249</td>
    <td>206</td>
    <td>43</td>
    <td>417</td>
    <td>348</td>
    <td>69</td>
  </tr>
  <tr>
    <td>TLA+</td>
    <td>256</td>
    <td>199</td>
    <td>57</td>
    <td>594</td>
    <td>476</td>
    <td>118</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>4,338</th>
    <th>3,433</th>
    <th>905</th>
    <th>18,293</th>
    <th>14,321</th>
    <th>3,972</th>
  </tr>
  <tr>
    <th>Average</th>
    <th>868</th>
    <th>687</th>
    <th>181</th>
    <th>3,659</th>
    <th>2,864</th>
    <th>794</th>
  </tr>
</table>

Table 1: Formal-Specification-Language-wise Statistics of Formal Verification Data

<table>
  <tr>
    <th>Task</th>
    <th>Total</th>
    <th>FM-ALPACA</th>
    <th>FM-BENCH</th>
  </tr>
  <tr>
    <td>1 Requirement Analysis</td>
    <td>627</td>
    <td>496</td>
    <td>131</td>
  </tr>
  <tr>
    <td>2 Full Proof Generation</td>
    <td>700</td>
    <td>557</td>
    <td>143</td>
  </tr>
  <tr>
    <td>3 Segment Proof Generation</td>
    <td>14843</td>
    <td>11597</td>
    <td>3246</td>
  </tr>
  <tr>
    <td>4 Proof Complete</td>
    <td>658</td>
    <td>520</td>
    <td>138</td>
  </tr>
  <tr>
    <td>5 Proof Infill</td>
    <td>1439</td>
    <td>1146</td>
    <td>293</td>
  </tr>
  <tr>
    <td>6 Code2Proof</td>
    <td>70</td>
    <td>56</td>
    <td>14</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>18337</th>
    <th>14372</th>
    <th>3965</th>
  </tr>
  <tr>
    <th>Average</th>
    <th>3056</th>
    <th>2395</th>
    <th>661</th>
  </tr>
</table>

Table 2: Task-wise Statistics of Formal Verification Data

erate high-quality informal proof descriptions via meticulous prompting. This alternative is well-established and frequently employed in prior literature (Wang et al., 2022, 2024; Ding et al., 2023; Wang et al., 2023). Specifically, for each formal specification language, we designated the model as an expert in that particular language, equipping it with comprehensive domain knowledge about the language’s specifications, essential grammatical cues, and three-shot examples featuring proof segments in the formal language as inputs and natural language descriptions as outputs. This approach ensures that the collected descriptions are of high quality and well-organized. It’s important to note that we did not generate descriptions for proof segments shorter than two lines (such as package imports or constant definitions) because their meaning is self-explained, with the exception of ACSL, whose proof segments are typically 1-2 lines.

After the descriptions for both full and segment proofs were prepared, we then prepared the data pairs for each task as shown in Task-wise Data Pairing in Figure 3. Note that for Task 4, i.e., Proof Completion, to prepare the incomplete formal proof, we randomly choose a line number and delete the lines in the proof after the line. For Task 5, i.e., Proof Infill, we randomly choose two line numbers and mask the lines between them. In case the remaining lines of proof cannot provide sufficient information for the proof generation, we also provide the informal proof for these two tasks.

After pairing the instruction-response for different tasks, we manually designed five task instructions for each task and randomly assigned one for each paired data to increase instruction diversity and avoid overfitting to certain instructions (Lu et al., 2022; Feng et al., 2023; Sanh et al., 2022).

3.3 Data Statistics

The specification-language-wise and task-wise statistics are shown in Table 1 and Table 2. In particular, Table 1 presents a detailed breakdown of the number of proofs and segments across five specification languages. Note that we split the prepared data for all the tasks and specification languages into an 8:2 ratio, i.e., 80% for fine-tuning, named FM-ALPACA, 20% for benchmarking, named FM-BENCH, and show the separate statistics. In particular, there are 4k+ verifiable proofs in total, with 249 ~ 2k proofs for each language. These proofs were split into 18k+ proof segments, with an average of 3.6k segments for each language. The reason why the ratio of segments in FM-ALPACA and FM-BENCH is slightly less than 8:2 is that the train-test split was applied to proofs, while the number of split segments in each proof varies.

Table 2 shows the task-wise statistics. There are 18k instructions across six tasks, with an average of 3k instructions for each task. After splitting the train-test set, FM-ALPACA contains 14k, and FM-BENCH has nearly 4k instructions. It is clear that the number of instructions for the task Segment Proof Generation (SegGen) is far more than that for Requirement Analysis (ReqAna) and Full Proof Generation (ProofGen) because one full proof can be split into numerous pieces of proof segments, and one proof can contribute to only one instruction for ReqAna and ProofGen. Note that the number of ReqAna (627) and ProofGen (700) is unequal because we filtered out the instructions with more than 2048 tokens considering the context limits.

3.4 Validation Mechanism

For the tasks whose outputs are written in formal specification languages, we verify the full proofs against the corresponding verifiers, i.e., Coq, Dafny, Lean4 use their own proving environment; formal specification written in TLA+ can be checked by TLC (Yu and Kuppe); C programs with ACSL specifications can be checked by Frama-C (Cuq et al., 2012; Carvalho et al., 2014). Also, for the proof segments that cannot be verified independently, for each extracted segment, we prepared a proof tem-
plate with a placeholder during data preparation. Whenever a generated segment is to be verified, we replace the placeholder in the template with the segment and verify the completed formal proof.

For ACSL specification verification, we report the results under two modes. First, Basic checking. ACSL uses ‘frama-c’ tool for syntax checking (i.e., whether the syntax is correct), semantics checking (i.e., program behaviors are consistent with the annotated ACSL specifications), and other basic ACSL checking. Second, Verification. ‘ACSL-WP’ uses ‘frama-c’ with the WP plugin for formal verification. This mode requires not only the basic checking (i.e., syntactically correct, semantically consistent with the program), but also that the properties under verification can be proved.

Note that the invalid outputs (e.g., empty response or responses without formal specifications) are considered incorrect directly and will not go through the verification.

For the task whose outputs are written in natural language (i.e., ReqAna), we calculate the Bleu score (Papineni et al., 2002) between the descriptions in FM-BENCH with the predicted outputs.

4 Experiments

4.1 Experiment Setup

Studied LLMs. We selected ten LLMs as baselines without fine-tuning, including llama3.1-instruct-8B/70B (Meta, 2024), qwen2.5-instruct-7B/72B (Yang et al., 2024a), qwen2.5-coder-instruct-7B/32B (Hui et al., 2024), starcoder-instruct-15B (Lozhkov et al., 2024), deepseek-coder-instruct-7B-v1.5, deepseek-coder-instruct-33B (Guo et al., 2024), and deepseek-R1 (Guo et al., 2025). Note that we avoid evaluating the GPT-series LLMs by OpenAI because the descriptions in FM-BENCH were generated by GPT-4o, making the evaluation fairer.

Fine-tuning. Instruction fine-tuning (Wei et al., 2021; Sanh et al., 2021; Ding et al., 2023; Ivison et al., 2023) aims to improve a model’s ability to effectively respond to human instructions and has shown strong experimental potential in model enhancement. We select llama3.1-8B-instruct (Meta, 2024), qwen2.5-7B-instruct (Yang et al., 2024a), and deepseek-coder-7B-instruct-v1.5 (Guo et al., 2024) as base models for fine-tuning. These three models are chosen due to their promising performance on tasks such as coding, mathematics, and reasoning, and fine-tuning models in their scale is relatively affordable compared with fine-tuning larger scale models. We fine-tuned the three aforementioned models over three epochs using a learning rate 2e-5, a warm-up ratio of 0.04, a batch size of 512, and a cosine learning rate scheduler.

Baseline Fine-tuning Datasets: To distinguish whether the capability improvement is simply because more instruction tuning is applied, we also include two commonly used fine-tuning datasets for comparison. We select UltraChat (Ding et al., 2023) and Tulu-v3 (Lambert et al., 2024) as baseline fine-tuning datasets, and use llama3.1–8B-base (Meta, 2024) as the base model due to their popularity. In particular, UltraChat is a large-scale dataset of instructional conversations that contains 1.5 million high-quality multi-turn dialogues and covers a wide range of topics and instructions. Tulu-v3 (Lambert et al., 2024) embraces new data that is either carefully manually curated for quality or generated from GPT models. It is an enhancement of its previous versions (Ivison et al., 2023; Wang et al., 2023), focusing more on core skills of knowledge recall, reasoning, mathematics, coding, instruction following, general chat, and safety.

Benchmarks for Related Capabilities (RQ4). To comprehensively evaluate the model’s capabilities, we tested the fine-tuned models on a series of benchmarks: Math (Hendrycks et al., 2021) and GSM-8K (Cobbe et al., 2021) for mathematical reasoning, BBH (Suzgun et al., 2022) for general reasoning, HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021b) for coding.

Inference Strategies: We adopt different settings for different RQs. In particular, We use (1) the greedy sampling strategy to generate one single greedy sample with a temperature of 0.0 and calculate Pass@1, and (2) nucleus sampling (Holtzman et al., 2020), where five solution samples were randomly generated with a temperature of 0.2 for RQ1 and RQ2. We also consider different in-context learning strategies, including zero-shot and few-shot (we used 3-shot in the experiment). For RQ3, we use a zero-shot greedy search with a temperature of 0.0 and a few-shot nuclear search with a temperature of 0.2 for a fair comparison.

Experiment Environment. The fine-tuning experiment was conducted on 32 Nvidia A800-40G GPUs, while inference was on Nvidia A100-80G GPUs with vLLM (Kwon et al., 2023).
<table>
  <tr>
    <th>LLMs</th>
    <th>Size</th>
    <th colspan="4">SegGen</th>
    <th colspan="4">ProofComplex</th>
    <th colspan="4">Code2Proof</th>
    <th>Ave</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>TLA</th>
    <th>Coq</th>
    <th>Lean</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>ACSL-WP</th>
    <th>TLA</th>
    <th>Coq</th>
    <th>Lean</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>ACSL-WP</th>
    <th>TLA</th>
    <th>Coq</th>
    <th>Lean</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>ACSL-WP</th>
    <th>Ave</th>
  </tr>
  <tr>
    <td>llama3-1-instruct</td>
    <td>8B</td>
    <td>0.00</td>
    <td>4.26</td>
    <td>1.09</td>
    <td>0.00</td>
    <td>14.20</td>
    <td>0.00</td>
    <td>1.43</td>
    <td>1.19</td>
    <td>0.00</td>
    <td>0.51</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>7.47</td>
  </tr>
  <tr>
    <td>llama3-1-instruct</td>
    <td>70B</td>
    <td>0.79</td>
    <td>6.18</td>
    <td>10.17</td>
    <td>20.00</td>
    <td>7.14</td>
    <td>0.00</td>
    <td>2.49</td>
    <td>4.86</td>
    <td>13.64</td>
    <td>11.11</td>
    <td>39.68</td>
    <td>3.17</td>
    <td>0.00</td>
    <td>3.57</td>
    <td>13.49</td>
    <td>28.57</td>
    <td>0.00</td>
    <td>7.27</td>
    <td>24.62</td>
    <td>13.33</td>
    <td>21.43</td>
    <td>0.00</td>
    <td>12.90</td>
  </tr>
  <tr>
    <td>qwen2.5-instruct</td>
    <td>7B</td>
    <td>0.21</td>
    <td>3.08</td>
    <td>0.00</td>
    <td>21.43</td>
    <td>0.00</td>
    <td>1.43</td>
    <td>1.05</td>
    <td>7.58</td>
    <td>5.56</td>
    <td>1.59</td>
    <td>1.59</td>
    <td>0.00</td>
    <td>13.00</td>
    <td>19.44</td>
    <td>28.57</td>
    <td>0.00</td>
    <td>5.45</td>
    <td>18.46</td>
    <td>13.33</td>
    <td>21.43</td>
    <td>0.00</td>
    <td>12.90</td>
    <td>7.56</td>
  </tr>
  <tr>
    <td>qwen2.5-instruct</td>
    <td>72B</td>
    <td>0.69</td>
    <td>6.18</td>
    <td>18.00</td>
    <td>20.00</td>
    <td>21.43</td>
    <td>7.14</td>
    <td>1.29</td>
    <td>12.12</td>
    <td>25.00</td>
    <td>12.50</td>
    <td>4.76</td>
    <td>0.00</td>
    <td>11.90</td>
    <td>33.53</td>
    <td>28.57</td>
    <td>0.00</td>
    <td>9.78</td>
    <td>24.62</td>
    <td>26.67</td>
    <td>35.71</td>
    <td>0.00</td>
    <td>15.82</td>
    <td>15.82</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-instruct</td>
    <td>7B</td>
    <td>0.21</td>
    <td>3.08</td>
    <td>0.00</td>
    <td>21.43</td>
    <td>0.00</td>
    <td>1.43</td>
    <td>1.05</td>
    <td>7.58</td>
    <td>5.56</td>
    <td>1.59</td>
    <td>1.59</td>
    <td>0.00</td>
    <td>13.00</td>
    <td>19.44</td>
    <td>28.57</td>
    <td>0.00</td>
    <td>5.45</td>
    <td>18.46</td>
    <td>13.33</td>
    <td>21.43</td>
    <td>0.00</td>
    <td>12.90</td>
    <td>7.56</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-instruct</td>
    <td>52B</td>
    <td>0.10</td>
    <td>6.18</td>
    <td>18.00</td>
    <td>21.43</td>
    <td>0.00</td>
    <td>12.46</td>
    <td>4.07</td>
    <td>9.85</td>
    <td>25.00</td>
    <td>28.98</td>
    <td>4.76</td>
    <td>27.27</td>
    <td>5.95</td>
    <td>36.11</td>
    <td>42.86</td>
    <td>2.94</td>
    <td>7.88</td>
    <td>35.85</td>
    <td>20.00</td>
    <td>7.43</td>
    <td>14.29</td>
    <td>57.14</td>
    <td>14.29</td>
    <td>19.05</td>
  </tr>
  <tr>
    <td>star-coder-instruct</td>
    <td>15B</td>
    <td>0.69</td>
    <td>2.13</td>
    <td>8.47</td>
    <td>0.00</td>
    <td>28.57</td>
    <td>0.00</td>
    <td>27.14</td>
    <td>1.97</td>
    <td>6.06</td>
    <td>27.78</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>27.17</td>
    <td>11.90</td>
    <td>25.00</td>
    <td>42.86</td>
    <td>0.00</td>
    <td>9.70</td>
    <td>24.62</td>
    <td>20.00</td>
    <td>35.71</td>
    <td>0.00</td>
    <td>28.57</td>
    <td>18.25</td>
  </tr>
  <tr>
    <td>deepseek-coder-instruct</td>
    <td>7B</td>
    <td>0.21</td>
    <td>3.08</td>
    <td>0.00</td>
    <td>21.43</td>
    <td>0.00</td>
    <td>4.29</td>
    <td>1.71</td>
    <td>3.79</td>
    <td>5.56</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>9.09</td>
    <td>2.58</td>
    <td>13.89</td>
    <td>26.57</td>
    <td>0.00</td>
    <td>3.03</td>
    <td>13.85</td>
    <td>35.53</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>5.42</td>
  </tr>
  <tr>
    <td>deepseek-coder-instruct</td>
    <td>72B</td>
    <td>0.10</td>
    <td>6.18</td>
    <td>18.00</td>
    <td>21.43</td>
    <td>0.00</td>
    <td>12.46</td>
    <td>4.07</td>
    <td>9.85</td>
    <td>25.00</td>
    <td>28.98</td>
    <td>4.76</td>
    <td>27.27</td>
    <td>5.95</td>
    <td>36.11</td>
    <td>42.86</td>
    <td>2.94</td>
    <td>7.88</td>
    <td>35.85</td>
    <td>20.00</td>
    <td>7.43</td>
    <td>14.29</td>
    <td>57.14</td>
    <td>14.29</td>
    <td>19.05</td>
  </tr>
  <tr>
    <td>deepseek-coder-instruct</td>
    <td>671B</td>
    <td>0.77</td>
    <td>8.51</td>
    <td>30.51</td>
    <td>30.00</td>
    <td>55.77</td>
    <td>7.14</td>
    <td>22.86</td>
    <td>10.70</td>
    <td>21.21</td>
    <td>22.22</td>
    <td>40.21</td>
    <td>3.17</td>
    <td>47.45</td>
    <td>15.48</td>
    <td>33.53</td>
    <td>28.57</td>
    <td>5.88</td>
    <td>9.70</td>
    <td>24.62</td>
    <td>6.07</td>
    <td>42.86</td>
    <td>21.43</td>
    <td>64.29</td>
    <td>14.29</td>
    <td>24.36</td>
  </tr>
  <tr>
    <th colspan="2">Task-wise average</th>
    <th>7.34</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>9.16</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>18.31</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>12.62</th>
    <th></th>
    <th>16.70</th>
    <th></th>
  </tr>
</table>

<table>
  <tr>
    <th>LLMs</th>
    <th>Size</th>
    <th colspan="4">ReqSplitter</th>
    <th colspan="4">w/ fine-tune</th>
    <th colspan="4">w/o fine-tune</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>TLA</th>
    <th>Coq</th>
    <th>Lean</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>ACSL-WP</th>
    <th>TLA</th>
    <th>Coq</th>
    <th>Lean</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>ACSL-WP</th>
    <th>TLA</th>
    <th>Coq</th>
    <th>Lean</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>ACSL-WP</th>
  </tr>
  <tr>
    <td>llama3-1-fma</td>
    <td>8B</td>
    <td>0.33</td>
    <td>0.26</td>
    <td>0.35</td>
    <td>0.36</td>
    <td>0.28</td>
    <td></td>
    <td>0.33</td>
    <td>0.18</td>
    <td>0.49</td>
    <td>0.49</td>
    <td>0.42</td>
    <td></td>
    <td>0.33</td>
    <td>0.18</td>
    <td>0.49</td>
    <td>0.49</td>
    <td>0.42</td>
    <td></td>
  </tr>
  <tr>
    <td>llama3-1-ultrachat</td>
    <td>8B</td>
    <td>0.33</td>
    <td>0.28</td>
    <td>0.35</td>
    <td>0.41</td>
    <td>0.30</td>
    <td></td>
    <td>0.35</td>
    <td>0.30</td>
    <td>0.44</td>
    <td>0.40</td>
    <td>0.46</td>
    <td></td>
    <td>0.35</td>
    <td>0.30</td>
    <td>0.44</td>
    <td>0.40</td>
    <td>0.46</td>
    <td></td>
  </tr>
  <tr>
    <td>llama3-1-ultrachat-fma</td>
    <td>8B</td>
    <td>0.37</td>
    <td>0.30</td>
    <td>0.56</td>
    <td>0.57</td>
    <td>0.62</td>
    <td></td>
    <td>0.37</td>
    <td>0.30</td>
    <td>0.56</td>
    <td>0.57</td>
    <td>0.62</td>
    <td></td>
    <td>0.37</td>
    <td>0.30</td>
    <td>0.56</td>
    <td>0.57</td>
    <td>0.62</td>
    <td></td>
  </tr>
  <tr>
    <td>llama3-1-tulu</td>
    <td>8B</td>
    <td>0.41</td>
    <td>0.30</td>
    <td>0.43</td>
    <td>0.42</td>
    <td>0.39</td>
    <td></td>
    <td>0.57</td>
    <td>0.33</td>
    <td>0.57</td>
    <td>0.59</td>
    <td>0.71</td>
    <td></td>
    <td>0.57</td>
    <td>0.33</td>
    <td>0.57</td>
    <td>0.59</td>
    <td>0.71</td>
    <td></td>
  </tr>
  <tr>
    <td>qwen2.5-fma</td>
    <td>7B</td>
    <td>0.33</td>
    <td>0.13</td>
    <td>0.53</td>
    <td>0.58</td>
    <td>0.38</td>
    <td></td>
    <td>0.51</td>
    <td>0.21</td>
    <td>0.57</td>
    <td>0.64</td>
    <td>0.73</td>
    <td></td>
    <td>0.43</td>
    <td>0.24</td>
    <td>0.54</td>
    <td>0.43</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>qwen2.5-coder-fma</td>
    <td>7B</td>
    <td>0.51</td>
    <td>0.21</td>
    <td>0.57</td>
    <td>0.64</td>
    <td>0.73</td>
    <td></td>
    <td>0.43</td>
    <td>0.24</td>
    <td>0.54</td>
    <td>0.43</td>
    <td></td>
    <td>0.43</td>
    <td>0.24</td>
    <td>0.54</td>
    <td>0.43</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th colspan="2">Average</th>
    <th>0.38</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>0.44</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>0.44</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>0.44</th>
    <th></th>
    <th>0.44</th>
    <th></th>
  </tr>
</table>

* fma: fine-tuned with FM-ALPACA.

Table 3: RQ1-3: Pass@1 Accuracy of LLMs’ Performance Across Formal Verification Task and Formal Specification Languages with (w/) and without (w/o) fine-tuning. The greener , the better.

<table>
  <tr>
    <th>LLMs</th>
    <th>Size</th>
    <th colspan="4">ReqSplitter</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>TLA</th>
    <th>Coq</th>
    <th>Lean</th>
    <th>Dafny</th>
  </tr>
  <tr>
    <th colspan="6">w/o fine-tune</th>
  </tr>
  <tr>
    <td>llama3-1-instruct</td>
    <td>8B</td>
    <td>0.33</td>
    <td>0.26</td>
    <td>0.35</td>
    <td>0.36</td>
    <td>0.28</td>
  </tr>
  <tr>
    <td>llama3-1-instruct</td>
    <td>70B</td>
    <td>0.33</td>
    <td>0.28</td>
    <td>0.35</td>
    <td>0.41</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>qwen2.5-instruct</td>
    <td>7B</td>
    <td>0.29</td>
    <td>0.25</td>
    <td>0.32</td>
    <td>0.33</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>qwen2.5-instruct</td>
    <td>72B</td>
    <td>0.33</td>
    <td>0.28</td>
    <td>0.33</td>
    <td>0.34</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-instruct</td>
    <td>7B</td>
    <td>0.29</td>
    <td>0.26</td>
    <td>0.35</td>
    <td>0.42</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-instruct</td>
    <td>52B</td>
    <td>0.31</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.39</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>star-coder-instruct</td>
    <td>15B</td>
    <td>0.21</td>
    <td>0.24</td>
    <td>0.29</td>
    <td>0.30</td>
    <td>0.29</td>
  </tr>
  <tr>
    <td>deepseek-coder-instruct</td>
    <td>7B</td>
    <td>0.34</td>
    <td>0.30</td>
    <td>0.39</td>
    <td>0.49</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td>deepseek-coder-instruct</td>
    <td>33B</td>
    <td>0.37</td>
    <td>0.29</td>
    <td>0.47</td>
    <td>0.47</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>deepseek-coder-instruct</td>
    <td>671B</td>
    <td>0.32</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.27</td>
    <td>0.28</td>
  </tr>
  <tr>
    <th colspan="6">Average</th>
    <th>31.85</th>
  </tr>
</table>

<table>
  <tr>
    <th>LLMs</th>
    <th>Size</th>
    <th colspan="4">ReqSplitter</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>TLA</th>
    <th>Coq</th>
    <th>Lean</th>
    <th>Dafny</th>
  </tr>
  <tr>
    <th colspan="6">w/ fine-tune</th>
  </tr>
  <tr>
    <td>llama3-1-fma</td>
    <td>8B</td>
    <td>0.33</td>
    <td>0.18</td>
    <td>0.49</td>
    <td>0.49</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>llama3-1-ultrachat</td>
    <td>8B</td>
    <td>0.35</td>
    <td>0.30</td>
    <td>0.44</td>
    <td>0.40</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>llama3-1-ultrachat-fma</td>
    <td>8B</td>
    <td>0.37</td>
    <td>0.30</td>
    <td>0.56</td>
    <td>0.57</td>
    <td>0.62</td>
  </tr>
  <tr>
    <td>llama3-1-tulu</td>
    <td>8B</td>
    <td>0.41</td>
    <td>0.30</td>
    <td>0.43</td>
    <td>0.42</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>llama3-1-tulu-fma</td>
    <td>8B</td>
    <td>0.57</td>
    <td>0.33</td>
    <td>0.57</td>
    <td>0.59</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>qwen2.5-fma</td>
    <td>7B</td>
    <td>0.33</td>
    <td>0.13</td>
    <td>0.53</td>
    <td>0.58</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-fma</td>
    <td>7B</td>
    <td>0.51</td>
    <td>0.21</td>
    <td>0.57</td>
    <td>0.64</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-fma</td>
    <td>7B</td>
    <td>0.43</td>
    <td>0.24</td>
    <td>0.54</td>
    <td>0.43</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="6">Average</th>
    <th>44.36 (39% ↑)</th>
  </tr>
</table>

* fma: fine-tuned with FM-ALPACA.

Table 4: Evaluation on Requirement Analysis

4.2 RQ1. Basic Performance across Formal Specification Tasks

To understand the current LLMs’ performance in six tasks, we evaluate 10 LLMs against FM-BENCH with model size ranges from 7B to 671B. The upper part of Table 3 and the upper part of Table 4 show LLMs’ basic performance without fine-tuning.

Task-wise: LLMs perform the best in ProofComplete (18.31%) and Code2Proof (16.79%). In contrast, LLMs fall short in generating both the entire formal proof (7.70%) and the proof segments (9.16%). We analyzed the failures and found that syntax errors account for a large proportion, with 12.43% failures caused by syntax errors (See Appendix B). The observation echoes the motivation of prior work (Wang et al., 2024) and is reasonable due to the grammar difference between most formal specification languages and other programming languages like Python. Regarding requirement analysis, as shown in the upper part of Table 4, the Bleu scores between the ground-truth description and LLM-generated ones range from 0.19 to 0.49.

LLM-wise: Without fine-tuning, DeepSeek-R1 achieved the best average (24.36%), followed by qwen2.5-coder-instruct-32B (19.05%).

Model Size: Larger LLMs generally perform better than smaller LLMs. For example, llama3.1-8B only achieved 1.43% in generating TLA+ segments, while llama3.1-70B boosts to 24.29% in the same task. However, there are several exceptions worth noticing, especially for ProofInfill and Code2Proof. For example, llama-3.1-8B achieved 57.14% in Code2Proof (ACSL), yet the performance drops to 21.43% using the llama-3.1-70B model. The decrease in performance is inherently due to the fine-tuning strategy of these instruction models: they are trained to excel in generating rather than filling in the blanks (Fried et al., 2022). Also, we conducted a more detailed examination of generated segments and observed that larger LLMs tend to fill in the proof segments that not only look
more plausibly correct and well-organized but also include extra content. The additional content, yet, is either redundant, as it repeats information that appears in the subsequent proof, or is incomplete. Promisingly, recent model developers have noticed such conundrums and refined their fine-tuning strategy for fill-in-the-middle tasks (Guo et al., 2024).

4.3 RQ2. Formal Specification Languages-wise Capability

Table 5 shows the LLMs’ performance across formal specification languages in the task of generating proof segments (SegGen). This task accounts for the most instructions and serves as the basic capability for other proof generation tasks. We can see that LLMs perform the best in ACSL (average: 34.92%), followed by Dafny (15.97%) while performing unsatisfactorily in other formal specification languages. The observation is reasonable because the syntax of ACSL is basically an annotation of C language, while Dafny shares similar grammar as C# and Java. Thus, generating proof segments in ACSL and Dafny is generally easier than generating other specification languages.

Note that though LLMs are generally good at generating syntactically correct and semantically consistent (i.e., program behaviors are consistent with the annotated ACSL specifications) according to the ‘ACSL’ column in Table 3, the specifications are usually insufficient for the verification according to the ‘ACSL-WP’ column in Table 3. The results in the two ACSL columns indicate that there is a large room to improve for generating not only syntactically/semantically correct specifications, but also that the generated specifications need to be sufficient to verify the properties under verification.

In addition, we explore whether increasing the attempts (1 → 5) with a higher temperature (0.0 → 0.2) and in-context learning could bring about improvement. The improvement ratios are shown in red in Table 5. The results of Pass@5 are better than those of Pass@1, with an average score increase from 10.82% (Dafny) to 63.64% (ACSL) in different languages. Moreover, when using 3-shot, the performance increases dramatically, with 51.33% (Dafny) to over five times (ACSL) improvement compared with zero-shot Pass@5. The results indicate the potential of in-context learning in generating correct specification languages.

4.4 RQ3. Improvement by Fine-tuning

We further investigate whether FM-ALPACA could bring about improvement. The lower part of Table 3 and Table 4 shows the results. From Table 3, dramatic improvements can be observed in generating full and segmental proofs after fine-tuning. Note that the model size of fine-tuned models is 7B ~ 8B, while the performance largely outperforms the 70B+ models without fine-tuning. Furthermore, after fine-tuning with formal data, the 7 ~ 8B fine-tuned models can achieve comparable or slightly better performance than Deepseek-R1-671B, with 27.31% achieved by qwen2.5-coder-7B fine-tuned with FM-ALPACA (R1-671B: 27.11%). It may suggest the possibility of distilling domain-specific small models for handier usage.

Task-wise: Improvements in generation tasks (i.e., ProofGen, SegGen, and ProofComplete) are substantial. ProofGen doubles the performance, and SegGen more than triples. The dramatic increases happen in all models fine-tuned with FM-ALPACA in SegGen Task, from nearly all zeros to 29.98% ~ 90.48%. An increase of 41% can also be observed in Table 4. The experimental improvements make evident the effectiveness of fine-tuning in formal verification tasks.

Yet, drops can be observed in fill-in-the-middle tasks (i.e., ProofInfill and Code2Proof). The results echo the observation made in RQ1 (Section 4.2), where the large LLMs perform worse than small LLMs in fill-in-the-middle tasks. The results also indicate the necessity of adopting different fine-tuning strategies other than instruction tuning only.

Fine-tuning Datasets: Take a closer look at the LLMs fine-tuned with general-purpose datasets (i.e., llama3.1-ultrachat and llama3.1-tulu) in Table 3, with them only, no or opposite effects can be observed. The results indicate the complementarity of FM-ALPACA and existing general-purpose fine-tuning datasets. Additionally, by combining with general-purpose datasets, the performance can be further improved (e.g., llama3.1-tulu-fma).

Comparison with Few-shot: Compared with the best results in Table 5 achieved by 3-shot, the results after fine-tuning (Table 3) still generally outperform the 3-shot results. The results indicate that although in-context learning can improve LLMs’ performance, the enhancement is limited. Further significant improvements still require fine-tuning with formal data. This may also suggest that in-context learning alone cannot adequately address
<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Size</th>
    <th colspan="6">TLA</th>
    <th colspan="6">Coq</th>
    <th colspan="6">Daily</th>
    <th colspan="6">ACSL</th>
    <th colspan="6">ACSL-WP</th>
  </tr>
  <tr>
    <th>Zero-Shot<br>P@1</th>
    <th>Pass@5</th>
    <th>Few-Shot<br>P@1</th>
    <th>P@5</th>
    <th>Zero-Shot<br>P@1</th>
    <th>Pass@5</th>
    <th>Few-Shot<br>P@1</th>
    <th>P@5</th>
    <th>Zero-Shot<br>P@1</th>
    <th>Pass@5</th>
    <th>Few-Shot<br>P@1</th>
    <th>P@5</th>
    <th>Zero-Shot<br>P@1</th>
    <th>Pass@5</th>
    <th>Few-Shot<br>P@1</th>
    <th>P@5</th>
    <th>Zero-Shot<br>P@1</th>
    <th>Pass@5</th>
    <th>Few-Shot<br>P@1</th>
    <th>P@5</th>
    <th>Zero-Shot<br>P@1</th>
    <th>Pass@5</th>
    <th>Few-Shot<br>P@1</th>
    <th>P@5</th>
  </tr>
  <tr>
    <td>llama3-1-instruct</td>
    <td>7B</td>
    <td>1.43</td>
    <td>4.29</td>
    <td>11.43</td>
    <td>1.19</td>
    <td>10.12</td>
    <td>6.06</td>
    <td>8.33</td>
    <td>9.85</td>
    <td>8.33</td>
    <td>11.11</td>
    <td>19.44</td>
    <td>0.00</td>
    <td>1.17</td>
    <td>38.10</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>12.70</td>
  </tr>
  <tr>
    <td>llama3-1-instruct</td>
    <td>7B</td>
    <td>21.29</td>
    <td>27.14</td>
    <td>38.57</td>
    <td>8.86</td>
    <td>6.01</td>
    <td>1.46</td>
    <td>13.42</td>
    <td>17.42</td>
    <td>15.91</td>
    <td>11.11</td>
    <td>13.89</td>
    <td>39.68</td>
    <td>52.38</td>
    <td>76.19</td>
    <td>3.17</td>
    <td>4.76</td>
    <td>14.29</td>
  </tr>
  <tr>
    <td>qwen2.5-instruct</td>
    <td>7B</td>
    <td>1.43</td>
    <td>2.86</td>
    <td>5.71</td>
    <td>1.05</td>
    <td>1.39</td>
    <td>12.80</td>
    <td>7.58</td>
    <td>9.09</td>
    <td>9.85</td>
    <td>5.56</td>
    <td>13.89</td>
    <td>1.59</td>
    <td>3.17</td>
    <td>66.67</td>
    <td>1.59</td>
    <td>3.17</td>
    <td>15.87</td>
  </tr>
  <tr>
    <td>qwen2.5-instruct</td>
    <td>7B</td>
    <td>12.86</td>
    <td>17.14</td>
    <td>24.29</td>
    <td>3.29</td>
    <td>3.80</td>
    <td>21.05</td>
    <td>12.12</td>
    <td>14.39</td>
    <td>19.70</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>12.70</td>
    <td>15.87</td>
    <td>90.48</td>
    <td>4.76</td>
    <td>15.87</td>
    <td>17.46</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-instruct</td>
    <td>7B</td>
    <td>2.86</td>
    <td>5.71</td>
    <td>12.86</td>
    <td>2.14</td>
    <td>2.99</td>
    <td>14.30</td>
    <td>12.12</td>
    <td>18.64</td>
    <td>18.94</td>
    <td>11.11</td>
    <td>11.11</td>
    <td>19.44</td>
    <td>3.17</td>
    <td>4.76</td>
    <td>87.30</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>17.46</td>
  </tr>
  <tr>
    <td>deepset-coder-instruct</td>
    <td>33B</td>
    <td>4.29</td>
    <td>8.57</td>
    <td>4.29</td>
    <td>1.73</td>
    <td>2.92</td>
    <td>8.79</td>
    <td>3.79</td>
    <td>5.30</td>
    <td>17.42</td>
    <td>5.56</td>
    <td>5.56</td>
    <td>19.44</td>
    <td>0.00</td>
    <td>4.76</td>
    <td>60.32</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>9.52</td>
  </tr>
  <tr>
    <td>deepset-coder-instruct</td>
    <td>33B</td>
    <td>2.86</td>
    <td>2.86</td>
    <td>18.57</td>
    <td>1.94</td>
    <td>3.33</td>
    <td>11.27</td>
    <td>4.55</td>
    <td>6.82</td>
    <td>22.73</td>
    <td>11.11</td>
    <td>13.89</td>
    <td>22.22</td>
    <td>3.17</td>
    <td>25.40</td>
    <td>92.06</td>
    <td>0.00</td>
    <td>1.59</td>
    <td>15.87</td>
  </tr>
  <tr>
    <td colspan="2">Language-wise Average</td>
    <td>12.20</td>
    <td>16.15</td>
    <td>6.15</td>
    <td></td>
    <td></td>
    <td>11.99</td>
    <td></td>
    <td></td>
    <td>15.97</td>
    <td></td>
    <td></td>
    <td>34.92</td>
    <td></td>
    <td></td>
    <td>6.35</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="2">Average</td>
    <td>7.86</td>
    <td>10.54</td>
    <td>18.21</td>
    <td>2.53</td>
    <td>3.19</td>
    <td>12.53</td>
    <td>8.71</td>
    <td>11.27</td>
    <td>16.00</td>
    <td>12.85</td>
    <td>14.24</td>
    <td>20.83</td>
    <td>10.91</td>
    <td>17.86</td>
    <td>75.99</td>
    <td>1.79</td>
    <td>2.58</td>
    <td>14.88</td>
  </tr>
  <tr>
    <td colspan="2">Improvement Ratio</td>
    <td>34.09%↑</td>
    <td>72.88%↑</td>
    <td>33.67%↑</td>
    <td>269.80%↑</td>
    <td>29.35%↑</td>
    <td>42.02%↑</td>
    <td>10.81%↑</td>
    <td>46.34%↑</td>
    <td>63.64%↑</td>
    <td>325.56%↑</td>
    <td>33.33%↑</td>
    <td>525.00%↑</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

* The improvement ratios shown in “Pass@5” columns are calculated by comparing with the scores in Pass@1, and the ratios shown in “Few-shot” column are calculated by comparing with the scores in Pass@5.

* ACSL and ACSL-WP: “ACSL” uses “frame-c” tool for syntax, semantics checking, and basic ACSL checking. “ACSL-WP” uses “frame-c” with the WP plugin for specification verification.

Table 5: RQ2: Language-wise LLMs’ Performance. Pass@1 and Pass@5 Accuracy in Generating Proof Segments Across Formal Specification Languages Under Zero/Few-shot without fine-tuning.

<table>
  <tr>
    <th rowspan="2">Fine-tuning Dataset</th>
    <th colspan="3">MATH</th>
    <th colspan="2">Reasoning</th>
    <th colspan="2">Coding</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>MATH</th>
    <th>GSM-8K</th>
    <th>Average</th>
    <th>bbh</th>
    <th>HumanEval</th>
    <th>MBPP</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>UltraChat</td>
    <td>17.54</td>
    <td>61.33</td>
    <td>39.44</td>
    <td>62.64</td>
    <td>19.51</td>
    <td>36.4</td>
    <td>27.96</td>
    <td>39.48</td>
  </tr>
  <tr>
    <td>UltraChat + fma</td>
    <td>16.16</td>
    <td>7.87%↓</td>
    <td>62.32</td>
    <td><b>1.61%</b>↑</td>
    <td>39.24</td>
    <td><b>0.49%</b>↑</td>
    <td>62.14</td>
    <td><b>0.80%</b>↑</td>
    <td>31.71</td>
    <td><b>62.53%</b>↑</td>
    <td>35.2</td>
    <td><b>3.30%</b>↑</td>
    <td>33.46</td>
    <td><b>19.67%</b>↑</td>
    <td>41.51</td>
    <td><b>5.14%</b>↑</td>
  </tr>
  <tr>
    <td>tula3</td>
    <td>27.36</td>
    <td>75.82</td>
    <td>51.59</td>
    <td>62.47</td>
    <td>64.02</td>
    <td>48.8</td>
    <td>56.41</td>
    <td>55.01</td>
  </tr>
  <tr>
    <td>tula3 + fma</td>
    <td>29.48</td>
    <td>7.75%↑</td>
    <td>75.44</td>
    <td><b>0.50%</b>↑</td>
    <td>52.46</td>
    <td><b>1.69%</b>↑</td>
    <td>63.16</td>
    <td><b>1.10%</b>↑</td>
    <td>64.63</td>
    <td><b>0.95%</b>↑</td>
    <td>49.4</td>
    <td><b>1.23%</b>↑</td>
    <td>57.02</td>
    <td><b>1.07%</b>↑</td>
    <td>55.76</td>
    <td><b>1.37%</b>↑</td>
  </tr>
</table>

Table 6: RQ4: Capability Migration from FM-ALPACA to Math, Reasoning, and Coding.

capability deficits in formal verification tasks but rather stem from a lack of knowledge.

4.5 RQ4. Capability Migration from Formal Verification to Related Tasks

Finally, we explore whether fine-tuning with FM-ALPACA could benefit related capabilities. Table 6 shows the results. The base model is llama3.1-8B, fine-tuned under two base fine-tuned datasets with and without FM-ALPACA. On average, with FM-ALPACA, an increase of 1.37% to 5.15% can be observed. Interestingly, a dramatic increase (62.53%) can be observed in HumanEval compared with the performance of the model that is only fine-tuned with UltraChat. The experiment may indicate that feeding more formal data may improve LLMs’ coding, reasoning, and math capabilities.

5 Conclusion

This paper contributes a comprehensive assessment and formulation to understand LLMs’ capability in formal verification. We constructed 18k high-quality instruction-response pairs across five formal specification languages in six tasks. The fine-tuned models, fine-tuning data, and the benchmark are released to facilitate subsequent studies.

Limitations

This paper has two primary limitations that offer avenues for future research. First, the primary limitation of our work is that our benchmark relies on model-generated data. While this approach effectively reduces manual efforts; it may introduce biases and data leakage issues in the dataset towards the models that generated the data. To address this limitation, we use gpt-4o to generate the natural language descriptions, while during the evaluation, we use other LLMs for evaluation. Second, another limitation of our work lies in the validation design. When creating ProofInfill and ProofComplete data, it is possible that the properties to be verified or theorems to be proven are masked. If LLMs happened not to generate these properties/theorems, the generated “proofs/models” could escape the verifier/checker, mistakenly labeling the output as correct. To avoid this scenario, we include the requirement descriptions as part of the input, guiding LLMs to generate the necessary properties or theorems without omission.

6 Acknowledgments

We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported in part by the Hong Kong Research Grants Council/General Research Fund (HKSAR RGC/GRF, Grant No. 16206524), the National Natural Science Foundation of China (Grant Nos. 62372193, 62192734, 62302375, 62472339), CAS Project for Young Scientists in Basic Research (Grant No.YSBR-040) and the Basic Research Program of ISCAS (Grant No. ISCAS-ZD-202402).
References

2023. Gpt-4 and gpt-4-turbo preview. https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4-and-gpt-4-turbo-preview.

DeepMind AlphaProof and AlphaGeometry Teams. 2024. Ai achieves silver-medal standard solving international mathematical olympiad problems.’25 july 2024.

Annie I Anton. 1996. Goal-based requirements analysis. In Proceedings of the second international conference on requirements engineering, pages 136–144. IEEE.

Andrew W. Appel. 2011. Verified software toolchain. In Programming Languages and Systems, pages 1–17, Berlin, Heidelberg. Springer Berlin Heidelberg.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021a. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021b. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Jeremy Avigad, Kevin Buzzard, Robert Y Lewis, and Patrick Massot. 2020. Mathematics in lean.

Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. 2023. Proofer: Autoformalizing and formally proving undergraduate-level mathematics. Preprint, arXiv:2302.12433.

Patrick Baudin, Jean-Christophe Filliâtre, Claude Marché, Benjamin Monate, Yannick Moy, and Virgile Prevosto. 2021. Asci: Ansi/iso c specification.

Nuno Carvalho, Cristiano da Silva Sousa, Jorge Sousa Pinto, and Aaron Tomb. 2014. Formal verification of klibc with the wp frama-c plug-in. In NASA Formal Methods: 6th International Symposium, NFM 2014, Houston, TX, USA, April 29–May 1, 2014. Proceedings 6, pages 343–358. Springer.

Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebbgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.

Edmund M Clarke. 1997. Model checking. In Foundations of Software Technology and Theoretical Computer Science: 17th Conference Kharagpur, India, December 18–20, 1997 Proceedings 17, pages 54–56. Springer.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Stephen A Cook. 2023. The complexity of theorem-proving procedures. In Logic, automata, and computational complexity: The works of Stephen A. Cook, pages 143–152.

Pascal Cuqo, Florent Kirchner, Nikolai Kosmatov, Virgile Prevosto, Julien Signoles, and Boris Yakobowski. 2012. Frama-c: A software analysis perspective. In International conference on software engineering and formal methods, pages 233–247. Springer.

Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software, 203:111734.

Alan M Davis. 1990. Software requirements: analysis and specification. Prentice Hall Press.

Jean-Marc Davril, Edouard Delfosse, Negar Hariri, Mathieu Acher, Jane Cleland-Huang, and Patrick Heymans. 2013. Feature model extraction from large collections of informal product descriptions. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, page 290–300, New York, NY, USA. Association for Computing Machinery.

Deva Kumar Deependamanti and Muhammad Ali Babar. 2009. An automated tool for generating uml models from natural language requirements. In 2009 IEEE/ACM International Conference on Automated Software Engineering, pages 680–682. IEEE.

Giuseppe Della Penna, Benedetto Intrigila, Daniele Magazzeni, Igor Melatti, and Enrico Tronci. 2013. Cgmurphi: Automatic synthesis of numerical controllers for nonlinear hybrid systems. European Journal of Control, 19(1):14–36.
Ning Ding, Yulin Chen, Bokai Xu, Yuja Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051, Singapore. Association for Computational Linguistics.

Agung Fatwanto. 2012. Translating software requirements from natural language to formal specification. In 2012 IEEE International Conference on Computational Intelligence and Cybernetics (Cybernetic sCom), pages 148–152. IEEE.

Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. 2023. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2704–2714.

Deborah Ferreira and André Freitas. 2020. Natural language premise selection: Finding supporting statements for mathematical text. arXiv preprint arXiv:2004.14959.

Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999.

Jeffrey O Grady. 2010. System requirements analysis. Elsevier.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming—the rise of code intelligence. arXiv preprint arXiv:2401.14196.

John Hatcliff, Gary T. Leavens, K. Rustan M. Leino, Peter Müller, and Matthew Parkinson. 2012. Behavioral interface specification languages. ACM Comput. Surv., 44(3).

Chris Hawblitzel, Jon Howell, Jacob R Lorch, Arjun Narayan, Bryan Parno, Danfeng Zhang, and Brian Zill. 2014. Ironclad apps:[End-to-End] security via automated {Full-System} verification. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 165–181.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Gérard Huet. 1986. The calculus of constructions. Ph.D. thesis, INRIA.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186.

Rasha Ahmad Husein, Hala Abrajouh, and Catalay Catal. 2024. Large language models for code completion: A systematic literature review. Computer Standards & Interfaces, page 103917.

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023. Camels in a changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10702.

Muhammad Javed and Yuqing Lin. 2021. imer: Iterative process of entity relationship and business process model extraction from the requirements. Information and Software Technology, 135:106558.

Ranjit Jhala and Rupak Majumdar. 2009. Software model checking. ACM Computing Surveys (CSUR), 41(4):1–54.

Albert Q Jiang, Sean Welleck, Jin Peng Zhou, Wenda Li, Jiacheng Liu, Mateja Jamnik, Timothée Lacroix, Yuhuai Wu, and Guillaume Lamplle. 2022. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. arXiv preprint arXiv:2210.12283.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.

Dongming Jin, Shengxin Zhao, Zhi Jin, Xiaohong Chen, Chunhui Wang, Zheng Fang, and Hongbin Xiao. 2024. An evaluation of requirements modeling for cyber-physical systems via llms. arXiv preprint arXiv:2408.02450.

Zhi Jin. 2017. Environment modeling-based requirements engineering for software intensive systems. Morgan Kaufmann.

Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, Thomas Sewell, Harvey Tuch, and Simon Winwood. 2009. sel4: formal verification of an os kernel. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, page 207–220, New York, NY, USA. Association for Computing Machinery.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. 2024. T\"ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124.

Leslie Lamport. 2002. Specifying systems: the tla+ language and tools for hardware and software engineers.

K Rustan M Leino. 2010. Dafny: An automatic program verifier for functional correctness. In International conference on logic for programming artificial intelligence and reasoning, pages 348–370. Springer.

Xavier Leroy, Sandrine Blazy, Daniel Kästner, Bernhard Schommer, Markus Pister, and Christian Ferdinand. 2016. CompCert-a formally verified optimizing compiler. In ERTS 2016: Embedded Real Time Software and Systems, 8th European Congress.

Weixian Waylon Li, Yftah Ziser, Maximin Coavoux, and Shay B Cohen. 2023. Bert is not the count: Learning to match mathematical statements with proofs. arXiv preprint arXiv:2302.09350.

Zhaoyu Li, Jialiang Sun, Logan Murphy, Qidong Su, Zenan Li, Xian Zhang, Kaiyu Yang, and Xujie Si. 2024. A survey on deep learning for theorem proving. arXiv preprint arXiv:2404.09939.

Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, Ming Zhang, and Qun Liu. 2023. Fimo: A challenge formal dataset for automated theorem proving. Preprint, arXiv:2309.04295.

Anton Lozhkov, Raymond Li, Loumba Ben Allad, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtiar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173.

Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. 2022. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5206–5215.

Lezhi Ma, Shangqing Liu, Yi Li, Xiaofei Xie, and Lei Bu. 2024. Speegen: Automated generation of formal program specifications via large language models. Preprint, arXiv:2401.08807.

The mathlib Community. 2020. The lean mathematical library. In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs, POPL’20. ACM.

AI Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. Blog. Online; accessed 15-January-2024.

Leonardo de Moura and Sebastian Ullrich. 2021. The lean 4 theorem prover and programming language. In Automated Deduction – CADE 28: 28th International Conference on Automated Deduction, Virtual Event, July 12–15, 2021, Proceedings, page 625–635, Berlin, Heidelberg. Springer-Verlag.

Eric Mugnier, Emmanuel Anaya Gonzalez, Ranjit Jhala, Nadia Polikarpova, and Yuanyuan Zhou. 2024. Laurel: Generating dafny assertions using large language models. Preprint, arXiv:2405.16792.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN conference on programming language design and implementation, pages 419–428.

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urnish Thakker, Shanya Sharma Sharma, Eliza Szczecina, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesh Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. Preprint, arXiv:2110.08207.

Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.

Peiyang Song, Kaiyu Yang, and Anima Anandkumar. 2024. Towards large language models as copilots for theorem proving in lean. arXiv preprint arXiv:2404.12534.

Claudio Spies, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. 2024. Calibration and correctness of language models for code. arXiv preprint arXiv:2402.02047.

Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett. 2024. Clover: Closed-loop verifiable code generation. Preprint, arXiv:2310.17807.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. 2019. Pythia: Ai-assisted code completion system. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2727–2735.

William Taggart Jr and Marvin O Tharp. 1977. A survey of information requirements analysis techniques. ACM Computing Surveys (CSUR), 9(4):273–290.

Terence Tao. 2024. Ai will become mathematicians’ “co-pilot”. https://www.scientificamerican.com/article/ai-will-become-mathematicians-co-pilot/.

Terence Tao, Yael Dillies, and Bhavik Mehta. 2023. Formalizing the proof of pfr in lean4 using blueprint: a short tour. Blog post, November.

Norbert Tihanyi, Ridhi Jain, Yiannis Charalambous, Mohamed Amine Ferrag, Youcheng Sun, and Lucas C Cordeiro. 2023. A new era in software security: Towards self-healing software via large language models and formal verification. arXiv preprint arXiv:2305.14752.

Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. 2024. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482.

Ruida Wang, Jipeng Zhang, Yizhen Jia, Rui Pan, Shizhe Diao, Renjie Pi, and Tong Zhang. 2024. Theoremlama: Transforming general-purpose llms into lean4 experts. arXiv preprint arXiv:2407.03203.

Ye Wang, JW Chen, Xin Xia, and B Jiang. 2021. Intelligent requirements elicitation and modeling: A literature review. Journal of Computer Research and Development, 58(4):683–705.

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khayathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764–74786.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alissa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.

Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. 2021. Naturalproofs: Mathematical theorem proving in natural language. Preprint, arXiv:2104.01112.

Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. 2022. Naturalprover: Grounded mathematical proof generation with language models. Advances in Neural Information Processing Systems, 35:4913–4927.

Cheng Wen, Jialun Cao, Jie Su, Zhiwu Xu, Shengchao Qin, Mengda He, Haokun Li, Shing-Chi Cheung, and Cong Tian. 2024. Enchanting program specification synthesis by large language models using static analysis and program verification. In International Conference on Computer Aided Verification, pages 302–328. Springer.

Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. 2022. Autoformalization with large language models. Advances in Neural Information Processing Systems, 35:32353–32368.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024a. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

Kaiyu Yang and Jia Deng. 2019. Learning to prove theorems via interacting with proof assistants. In International Conference on Machine Learning, pages 6984–6994. PMLR.

Kaiyu Yang, Gabriel Poesia, Jingxuan He, Wenda Li, Kristin Lauter, Swarat Chaudhuri, and Dawn Song. 2024b. Formal mathematical reasoning: A new frontier in ai. arXiv preprint arXiv:2412.16075.

Kaiyu Yang, Aidan M Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. 2023. Leandojo: Theorem proving with retrieval-augmented language models. arXiv preprint arXiv:2306.15626.

Yuan Yu and Markus Kuppe. Tlc model checker. https://tla.msr-inria.inria.fr/tlatoolbox/doc/model/executing-tlc.html.

Yuan Yu, Panagiotis Manolios, and Leslie Lamport. 1999. Model checking tla+ specifications. In Advanced research working conference on correct hardware design and verification methods, pages 54–66. Springer.

Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. 2021. Minif2f: a cross-system benchmark for formal olympiad-level mathematics. arXiv preprint arXiv:2109.00110.

Qixiang Zhou, Tong Li, and Yunduo Wang. 2022. Assisting in requirements goal modeling: a hybrid approach based on machine learning and logical reasoning. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems, pages 199–209.
A  Related Work

The formal specification datasets or benchmarks offer a standard, well-defined set of problems, providing a shared challenge that helps build a community of practice among researchers. According to different verification techniques, the existing benchmarks mainly fall into two categories; we discuss them separately.

A.1  Theorem Proving Datasets

Formal theorem proving represents theorems and proofs in a machine-verifiable format (Cook, 2023), ensuring their correctness using rigorous logical rules. A recent survey (Li et al., 2024) summarized the existing datasets for theorem proving. In particular, the informal benchmarks craft the proofs from various sources such as ProofWiki, textbooks, and public corpus. NL-PS (Ferreira and Freitas, 2020) first builds a natural language premise selection dataset source from ProofWiki. Similarly, NaturalProofs (Welleck et al., 2021) further incorporates data from Stacks and textbooks, resulting in a dataset with roughly 25k examples. Adapted from it, NaturalProofs-Gen (Welleck et al., 2022) contains around 14.5k theorems for informal proof generation. Moreover, MATcH (Li et al., 2023) constructs over 180k statement-proof pairs for matching using the MREC corpus 2.

For formal datasets, a line of efforts focuses on extracting and cleaning theorems and proofs written in various specification languages (e.g., Coq, Isabelle, Lean) from established formal libraries and verification projects. For example, LeanDojo (Yang et al., 2023) extracts over 98k theorems and proofs with 130k premises from Lean mathlib (mathlib Community, 2020). Besides extracting data from existing projects, several works manually annotate or formalize the problems in natural language. For example, MiniF2F (Zheng et al., 2021) manually formalizes 488 Olympiad-level problems across 4 proof systems and equally splits them into a validation set and a test set. FIMO (Liu et al., 2023) and ProofNet (Azerbayev et al., 2023) formalize the theorem statements of the International Mathematical Olympiad and undergraduate-level problems in Lean. In addition, datasets for Dafny also attract research contributions because industries like Amazon adopted Dafny to verify cryptographic libraries, authorization protocols, a random number generator, and the Ethereum virtual machine. Dafny datasets such as CloverBench (Sun et al., 2024) and DafnyGym (Mugnier et al., 2024).

A.2  Model checking datasets

Model checking is an automated technique used in computer science and formal methods to verify the correctness of systems, particularly those with finite state spaces. It systematically checks whether a system’s model satisfies a given specification, usually expressed in formal specification languages. The basic idea is to explore all possible system states to ensure the desired properties hold in every conceivable scenario.

Model checking benchmarks are less than that for theorem proving. Currently, there are few model-checking benchmarks for proving, while several model-checking subjects are going with specific model-checking languages such as CMurphi (Della Penna et al., 2013) and TLA+ (Yu et al., 1999). In particular, CMurphi is a software tool used to verify concurrent and distributed systems through explicit state enumeration. It implements the Murphi verification language, which allows users to describe finite-state systems in a procedural style. The core principle behind CMurphi is to explore the state space of a system exhaustively to check for violations of specified invariants or properties. Another example is TLA+ (Temporal Logic of Actions), a high-level language for modeling programs and systems suitable for concurrent and distributed systems.

B  Proportion of Failures Caused by Syntax Error

We listed the proportions of failures caused by syntax errors for each LLM and each task in Table 8. We used a set of pre-defined keywords (summarized in Table 7 to identify if a verification failure is caused by syntax errors. Specifically, we consider a failure caused by syntax error if its error message contains at least one keyword in Table 7.

<table>
  <tr>
    <th>Language</th>
    <th>Keywords</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Coq</td>
    <td>"Syntax Error:"</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Lean4</td>
    <td>"unexpected token ","unknown identifier","type mismatch"</td>
  </tr>
  <tr>
    <td>3</td>
    <td>ACSL</td>
    <td>"unexpected token"</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Dafny</td>
    <td>"type errors detected"</td>
  </tr>
  <tr>
    <td>5</td>
    <td>TLA+</td>
    <td>"***Parse Error***: "Unknown operator"</td>
  </tr>
</table>

Table 7: Keywords for identifying syntax error raised by each language’s verifier.
<table>
  <tr>
    <th>LLMs</th>
    <th>Size</th>
    <th>TLA+</th>
    <th>Coq</th>
    <th>ProofGen</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>TLA+</th>
    <th>Coq</th>
    <th>ProofGen</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>TLA+</th>
    <th>Coq</th>
    <th>ProofGen</th>
    <th>Dafny</th>
    <th>ACSL</th>
    <th>CatPrf</th>
  </tr>
  <tr>
    <td>llama3-1-instruct</td>
    <td>9B</td>
    <td>0/13</td>
    <td>18/65</td>
    <td>34/58</td>
    <td>0/10</td>
    <td>1/14</td>
    <td>4/39</td>
    <td>72/2910</td>
    <td>78/124</td>
    <td>4/73</td>
    <td>4/63</td>
    <td>9/11</td>
    <td>45/81</td>
    <td>16/34</td>
    <td>44/173</td>
    <td>27/60</td>
    <td>0/13</td>
    <td>1/14</td>
  </tr>
  <tr>
    <td>llama3-1-instruct</td>
    <td>70B</td>
    <td>4/12</td>
    <td>10/43</td>
    <td>40/53</td>
    <td>1/8</td>
    <td>1/14</td>
    <td>4/35</td>
    <td>71/280</td>
    <td>75/114</td>
    <td>16/32</td>
    <td>5/61</td>
    <td>6/8</td>
    <td>44/81</td>
    <td>10/31</td>
    <td>1/5</td>
    <td>15/34</td>
    <td>39/153</td>
    <td>22/49</td>
    <td>0/13</td>
    <td>2/14</td>
  </tr>
  <tr>
    <td>qwen2.5-instruct</td>
    <td>7B</td>
    <td>12/11</td>
    <td>13/68</td>
    <td>38/56</td>
    <td>1/10</td>
    <td>0/13</td>
    <td>26/69</td>
    <td>30/224</td>
    <td>14/15</td>
    <td>5/7</td>
    <td>6/12</td>
    <td>9/11</td>
    <td>35/73</td>
    <td>8/29</td>
    <td>1/5</td>
    <td>14/33</td>
    <td>66/156</td>
    <td>31/53</td>
    <td>0/13</td>
    <td>4/14</td>
  </tr>
  <tr>
    <td>qwen2.5-instruct</td>
    <td>72B</td>
    <td>12/11</td>
    <td>14/64</td>
    <td>38/56</td>
    <td>1/8</td>
    <td>0/13</td>
    <td>26/69</td>
    <td>30/224</td>
    <td>14/15</td>
    <td>5/7</td>
    <td>6/12</td>
    <td>9/11</td>
    <td>35/73</td>
    <td>8/29</td>
    <td>1/5</td>
    <td>14/33</td>
    <td>66/156</td>
    <td>31/53</td>
    <td>0/13</td>
    <td>4/14</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-instruct</td>
    <td>7B</td>
    <td>11/13</td>
    <td>18/46</td>
    <td>38/55</td>
    <td>0/10</td>
    <td>0/14</td>
    <td>6/68</td>
    <td>22/282</td>
    <td>72/116</td>
    <td>6/32</td>
    <td>5/63</td>
    <td>10/11</td>
    <td>12/80</td>
    <td>8/32</td>
    <td>2/6</td>
    <td>17/34</td>
    <td>34/159</td>
    <td>26/50</td>
    <td>0/12</td>
    <td>0/14</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-instruct</td>
    <td>32B</td>
    <td>11/13</td>
    <td>12/44</td>
    <td>29/52</td>
    <td>1/10</td>
    <td>2/14</td>
    <td>5/68</td>
    <td>31/282</td>
    <td>68/119</td>
    <td>9/27</td>
    <td>25/60</td>
    <td>5/8</td>
    <td>24/79</td>
    <td>6/23</td>
    <td>3/4</td>
    <td>17/33</td>
    <td>51/152</td>
    <td>16/43</td>
    <td>0/12</td>
    <td>0/12</td>
  </tr>
  <tr>
    <td>qwen2.5-coder-instruct</td>
    <td>128B</td>
    <td>12/13</td>
    <td>14/66</td>
    <td>38/56</td>
    <td>1/10</td>
    <td>0/14</td>
    <td>6/68</td>
    <td>22/282</td>
    <td>72/116</td>
    <td>6/32</td>
    <td>5/63</td>
    <td>10/11</td>
    <td>12/80</td>
    <td>8/32</td>
    <td>2/6</td>
    <td>17/34</td>
    <td>34/159</td>
    <td>26/50</td>
    <td>0/12</td>
    <td>0/14</td>
  </tr>
  <tr>
    <td>deepeek-coder-instruct</td>
    <td>33B</td>
    <td>12/13</td>
    <td>0/47</td>
    <td>46/58</td>
    <td>4/10</td>
    <td>0/14</td>
    <td>6/68</td>
    <td>52/288</td>
    <td>8/126</td>
    <td>8/32</td>
    <td>45/63</td>
    <td>10/11</td>
    <td>3/81</td>
    <td>6/31</td>
    <td>2/4</td>
    <td>19/34</td>
    <td>6/161</td>
    <td>23/50</td>
    <td>0/14</td>
    <td>0/14</td>
  </tr>
  <tr>
    <td>star-coder-instruct</td>
    <td>15B</td>
    <td>8/12</td>
    <td>0/46</td>
    <td>27/54</td>
    <td>2/7</td>
    <td>3/14</td>
    <td>36/51</td>
    <td>10/287</td>
    <td>6/7124</td>
    <td>12/26</td>
    <td>1/63</td>
    <td>6/8</td>
    <td>4/74</td>
    <td>7/27</td>
    <td>1/4</td>
    <td>20/34</td>
    <td>6/149</td>
    <td>23/49</td>
    <td>0/12</td>
    <td>0/14</td>
  </tr>
</table>

1 Denominator represents the total number of failures.

Table 8: Proportion of Verification Failures Caused by Syntax Errors

C Example Specifications in FM-ALPACA and FM-BENCH

The examples of the five formal specification languages are shown in Figure 4.

D Collected Repositories

We listed the repositories that were collected for data construction in the following. Note that one can easily add more repositories into FM-ALPACA and FM-BENCH.
For ACSL:

• https://github.com/manavpatnaik/frama-c-problems

• https://github.com/fraunhoferfokus/acsl-by-example

For TLA+:

• https://github.com/tlaplus/Examples

For Lean4:

• https://github.com/leanprover/lean4

For Coq:

• https://github.com/coq/coq

For Dafny:

• https://github.com/vladstefjeroiu/Dafny-programs

E Complete Evaluation Result

The Pass@1 and Pass@5 are shown in Table 9. It is a completed version of Table 3.

F Prompt Design

We listed the prompts that are used for data preparation and inference in the following. For data preparation, as shown in Figure 5, to generate descriptions for the given proof segments, the prompt template consists of five parts: (1) Role description, (2) Domain knowledge of TLA+, (3) Task description, (4) Few-shot examples (we show one example in the figure, while three-shots were used in RQ2), (5) The proof or proof segment to be summarized.

For the inference, for each task, we designed five different instructions to avoid overfitting. The prompts for each task are shown in Figure 6 ~ Figure 10. For each task, we first randomly choose one instruction and concat the inputs.
Coq
Proof.
intros R Rb H.
decide (pall R true false).
- exists (fun f → true).
  intros f. auto.
  assert (forall a b, f a = b). auto.
  + reflexivity.
    abstract_done: intros HR; exact (f i) insym in HR.
Qed.

Dafny
method factorialFail(nat res) returns (res: nat)
{
  var i := 1;
  res := 0;
  while i ≤ n do
    invariant 0 < i ∧ i ≤ n ∧ 1
    invariant res == fact(i - 1)
    {
      res := i * res;
      i := i + 1;
    }
}

Lean4
theorem mod_add_both_left :
  (n : int) (nat_ab_int : [prof ab int]) i ∧ by
  have |- Nat.mod_arg_left : nat that: b accumulate fun in licl {0}
  unfolding cast_add [nat_ab_int; dll abst; ] at this
lemma Nat.module_right : (n : int) (nat_ab_int : [prof ab int]) : by
  have |- Nat.mod_add_right : nat that: a in MathNat_mod2{oldtgt}={oldref}()
  unfolding N within addict on this = exact Nat.proof this Nat_dvd_self
ACSL
#include <stdio.h>
/**
 * variable: a = 8 & d=8 & b := 9;
 * ensures result == total(a, add(); d);
 * assigns nothing;
 */
int mainFail(int , int ,glob b, prod+1);
/* * */
loop injuring pointer == 0x8a; pend GXSheet ;
  loop assigning prod, x;
     while x <> 0;
        prod = prod + y;
        x;
    end
return prod;
*/
int main2() {
  int extern gpt = muli7_5();
  //g assert ggt == 1 << k;
TLA+
PickADifferentColor
varيئة as
* can't_compare == >
can.white == 1 *
can^* <can EXCEPT 1.black == {1 == 1}
TerminationTheorem:
if (given c & !Er)
THEN (can.black == 1 ∧ can.white == 0)
ELSE !=can.black == 0 ∧ \neg can.white == 1)

(📖orialldc.orm files)*

Figure 4: Formal Specification Languages in FM-bench

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Sites</th>
    <th colspan="3">PrefGen</th>
    <th colspan="3">NatGen</th>
    <th colspan="3">PrefComplex</th>
    <th colspan="3">ProfsAssets</th>
    <th rowspan="2">C2Nat</th>
  </tr>
  <tr>
    <th>TLA</th><th>Cog</th><th>Learnt</th><th>Define</th>
    <th>ACSL</th><th>ACSL-WP</th><th>TLA</th><th>Cog</th><th>Learnt</th><th>Define</th>
    <th>ACSL</th><th>ACSL-WP</th><th>TLA</th><th>Cog</th><th>Learnt</th><th>Define</th>
    <th></th>
  </tr>
  <tr>
    <td rowspan="2">llama-3-instruct</td>
    <td>coldicorm</td>
    <td>80</td><td>0.05</td><td>0.80</td><td>0.41</td><td>28.62</td><td>19.78</td><td>0.00</td><td>14.37</td><td>8.63</td><td>0.83</td><td>0.00</td><td>1.37</td><td>5.56</td><td>28.57</td><td>0.00</td><td>0.85</td><td>7.69</td><td>13.35</td><td>21.43</td><td>0.00</td><td>5748</td><td>0.00</td>
  </tr>
  <tr>
    <td>nshllama</td>
    <td>82</td><td>0.04</td><td>0.62</td><td>1.12</td><td>0.14</td><td>15.79</td><td>11.37</td><td>0.13</td><td>8.91</td><td>24.21</td><td>0.68</td><td>0.00</td><td>1.74</td><td>9.61</td><td>12.19</td><td>0.00</td><td>0.85</td><td>6.93</td><td>21.83</td><td>10.56</td><td>0.00</td><td>5891</td><td>0.00</td>
  </tr>
  <tr>
    <td rowspan="2">qewas-5-instruct</td>
    <td>coldicorm</td>
    <td>76</td><td>0.02</td><td>0.58</td><td>0.19</td><td>4.61</td><td>1.82</td><td>0.00</td><td>5.14</td><td>24.14</td><td>0.50</td><td>0.00</td><td>2.94</td><td>9.02</td><td>35.08</td><td>0.00</td><td>0.58</td><td>15.12</td><td>11.18</td><td>21.53</td><td>0.00</td><td>14202</td><td>0.00</td>
  </tr>
  <tr>
    <td>nshllama</td>
    <td>zuhal</td>
    <td>78</td><td>0.02</td><td>0.58</td><td>0.19</td><td>4.61</td><td>1.82</td><td>0.00</td><td>5.14</td><td>24.14</td><td>0.50</td><td>0.00</td><td>2.94</td><td>9.02</td><td>35.08</td><td>0.00</td><td>0.58</td><td>15.12</td><td>10.38</td><td>19.41</td><td>0.00</td><td>14202</td><td>0.00</td>
  </tr>
  <tr>
    <td rowspan="2">qewas-2-codes incontro-with</td>
    <td>coldicorm</td>
    <td>78</td><td>0.02</td><td>0.56</td><td>1.57</td><td>0.00</td><td>4.61</td><td>44.22</td><td>0.00</td><td>9.37</td><td>45.74</td><td>0.50</td><td>0.00</td><td>2.94</td><td>19.22</td><td>43.05</td><td>0.00</td><td>0.47</td><td>15.77</td><td>8.91</td><td>25.68</td><td>0.00</td><td>14220</td><td>0.00</td>
  </tr>
  <tr>
    <td>nshllama</td>
    <td>78</td><td>0.02</td><td>0.58</td><td>1.18</td><td>0.00</td><td>2.94</td><td>9.02</td><td>0.00</td><td>2.94</td><td>9.02</td><td>35.08</td><td>0.00</td><td>2.94</td><td>9.02</td><td>35.08</td><td>0.00</td><td>0.58</td><td>15.12</td><td>10.38</td><td>19.41</td><td>0.00</td><td>14202</td><td>0.00</td>
  </tr>
  <tr>
    <td rowspan="2">dpcook6-corolen-fiction-with</td>
    <td>coldicorm</td>
    <td>78</td><td>0.04</td><td>0.58</td><td>0.19</td><td>8.04</td><td>4.28</td><td>0.00</td><td>8.04</td><td>18.69</td><td>22.61</td><td>0.50</td><td>0.00</td><td>5.82</td><td>13.85</td><td>23.86</td><td>0.00</td><td>0.57</td><td>13.04</td><td>12.43</td><td>19.20</td><td>0.00</td><td>9637</td><td>0.00</td>
  </tr>
  <tr>
    <td>nshllama</td>
    <td>78</td><td>0.04</td><td>0.58</td><td>0.19</td><td>8.04</td><td>4.28</td><td>0.00</td><td>8.04</td><td>18.69</td><td>22.61</td><td>0.50</td><td>0.00</td><td>5.82</td><td>13.85</td><td>23.86</td><td>0.00</td><td>0.57</td><td>13.04</td><td>12.43</td><td>19.20</td><td>0.00</td><td>9637</td><td>0.00</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Sites</th>
    <th colspan="3">PrefGen</th>
    <th colspan="3">NatGen</th>
    <th colspan="3">PrefComplex</th>
    <th colspan="3">ProfsAssets</th>
    <th rowspan="2">C2Nat</th>
  </tr>
  <tr>
    <th>TLA</th><th>Cog</th><th>Learnt</th><th>Define</th>
    <th>ACSL</th><th>ACSL-WP</th><th>TLA</th><th>Cog</th><th>Learnt</th><th>Define</th>
    <th>ACSL</th><th>ACSL-WP</th><th>TLA</th><th>Cog</th><th>Learnt</th><th>Define</th>
    <th></th>
  </tr>
  <tr>
    <td rowspan="2">llama-3-instruct</td>
    <td>enzur</td>
    <td>85</td><td>0.20</td><td>0.20</td><td>13.96</td><td>0.00</td><td>4.94</td><td>0.92</td><td>25.10</td><td>0.00</td><td>8.08</td><td>9.87</td><td>28.67</td><td>0.00</td><td>1.46</td><td>7.43</td><td>6.49</td><td>20.27</td><td>0.00</td><td>3247</td><td>0.00</td>
  </tr>
  <tr>
    <td>maki</td>
    <td>68</td><td>0.08</td><td>0.30</td><td>0.00</td><td>2.00</td><td>3.87</td><td>0.00</td><td>10.25</td><td>32.50</td><td>7.06</td><td>0.00</td><td>3.99</td><td>13.02</td><td>18.20</td><td>0.00</td><td>0.99</td><td>6.95</td><td>14.29</td><td>0.00</td><td>5427</td><td>0.00</td>
  </tr>
  <tr>
    <td rowspan="2">qewas-5-instruct</td>
    <td>enzur</td>
    <td>85</td><td>0.21</td><td>0.21</td><td>-17.57</td><td>-0.00</td><td>-18.02</td><td>0.00</td><td>-15.79</td><td>0.00</td><td>-18.16</td><td>-0.08</td><td>-15.27</td><td>0.96</td><td>-15.00</td><td>0.00</td><td>-14.05</td><td>-0.04</td><td>-14.80</td><td>0.00</td><td>1467</td><td>0.00</td>
  </tr>
  <tr>
    <td>maki</td>
    <td>100</td><td>0.21</td><td>0.21</td><td>-23.53</td><td>0.00</td><td>-24.77</td><td>0.00</td><td>-24.82</td><td>0.00</td><td>-24.69</td><td>-0.07</td><td>-24.74</td><td>0.01</td><td>-24.67</td><td>0.00</td><td>-24.54</td><td>-0.03</td><td>-24.61</td><td>0.00</td><td>2049</td><td>0.00</td>
  </tr>
  <tr>
    <td rowspan="2">qewas-2-codes incontro tai-n</td>
    <td>enzur</td>
    <td>78</td><td>0.06</td><td>0.79</td><td>30.00</td><td>0.00</td><td>4.37</td><td>0.68</td><td>13.38</td><td>0.50</td><td>2.57</td><td>12.42</td><td>0.00</td><td>1.28</td><td>7.46</td><td>6.48</td><td>20.27</td><td>0.00</td><td>3247</td><td>0.00</td>
  </tr>
  <tr>
    <td>maki</td>
    <td>78</td><td>0.06</td><td>0.79</td><td>31.00</td><td>0.00</td><td>4.37</td><td>0.68</td><td>13.38</td><td>0.50</td><td>2.57</td><td>12.42</td><td>0.00</td><td>1.28</td><td>7.46</td><td>6.48</td><td>20.27</td><td>0.00</td><td>3247</td><td>0.00</td>
  </tr>
  <tr>
    <td rowspan="2">dpcook6-corolen-fiction one-trait</td>
    <td>enzur</td>
    <td>78</td><td>0.04</td><td>0.95</td><td>20.06</td><td>0.00</td><td>8.04</td><td>2.29</td><td>26.06</td><td>27.17</td><td>16.41</td><td>1.T4</td><td>17.70</td><td>31.35</td><td>20.27</td><td>0.00</td><td>1.42</td><td>10.09</td><td>12.42</td><td>0.00</td><td>963</td><td>0.00</td>
  </tr>
  <tr>
    <td>maki</td>
    <td>78</td><td>0.04</td><td>0.76</td><td>20.06</td><td>0.00</td><td>8.04</td><td>2.29</td><td>26.06</td><td>27.17</td><td>16.41</td><td>1.97</td><td>17.78</td><td>31.35</td><td>20.27</td><td>0.00</td><td>1.42</td><td>10.09</td><td>12.42</td><td>0.00</td><td>963</td><td>0.00</td>
  </tr>
</table>

Table 9: Full Experiment Results on Pass@1 and Pass@5
Prompt for Description Generation. Take TLA+ as an example.

# Role description
As an expert in TLA+, you are good at understanding and writing TLA+. TLA+ is a formal specification language used for modeling and verifying concurrent and distributed systems.

# Domain knowledge
1. The logical operators supported by TLA+ include: \land (and), \lor (or), \neg (not), \implies (Implication), \iff (Bidirectional implication), TRUE, FALSE, \forall (Universal Quantification), \exists (Existential Quantification)
2. The sets operators supported by TLA+ include: = (Equality), # (not equal), \cup (Union), \cap (Intersection), \in (Membership), \notin (Not in), \subseteq (Subset Equal), \setminus (Difference).
3. The temporal Operators supported by TLA+ include [] x > 0, which is an example of [] (Always). It means that at all times, the value of variable x is greater than 0. <> x = 0 is an example of <> (Eventually). It means that at some point in time, the value of variable x becomes 0.
4. Built-in keywords and operators in TLA+ include: ‘MODULE’, ‘EXTENDS’, ‘CONSTANT’, ‘INSTANCE’, ‘VARIABLE’, ‘ASSUME’, ‘PROVE’, ‘INIT’, ‘NEXT’, ‘ACTION’, ‘SPECIFICATION’, ‘IF’, ‘ELSE’, ‘WITH’, ‘CASE’, ‘THEN’, ‘LET’, ‘IN’, ‘CHOOSE’, ‘ENABLED’, ‘UNCHANGED’, ‘DOMAIN’.

# Task description
Given a TLA+ code snippet, you need to summarize the given TLA+ in several sentences in detail.

# Example Input and Output
## Code
Return(c,S) == 
/\ S # {} /\ S \subseteq alloc[c]
/\ alloc' = [alloc EXCEPT !c] = \ S
/\ UNCHANGED unsat
## Description:
An operation ‘Return(c,S)’ that represents the return of a set of resources by a client. It satisfies the following conditions:
- The set ‘S’ is not empty and ‘S’ must be a subset of the set of allocated resources to the client ‘c’.
- The ‘alloc’ is updated by assigning the difference between the current set of allocated resources and the set ‘S’ to the ‘c’ index of ‘alloc’ except ‘c’.
- The ‘unsat’ remains unchanged.

# Code to be described:
<A proof segment>
Prompt for SegGen Task (lang: a placeholder to be replaced by each formal specification language name.)

# Task Description (SegGen)
1. Translate the given natural language into {lang} syntax.
2. Model the intention written in natural language using {lang}.
3. Express the requirement using {lang}.
4. Model the given natural language into {lang}.
5. Translate the given requirement using {lang}'s syntax and semantics.

(Randomly choose one of the above.)

You only need to return the {lang} formal specification without explanation.

# Input
<Input goes here>

Figure 6: Prompt for SegGen Task.

Prompt for ProofGen Task (lang: a placeholder to be replaced by each formal specification language name.)

# Task Description (ProofGen)
1. Translate the given requirements into lang syntax.
2. Model the given requirements written in natural language using lang.
3. Express the requirements using lang.
4. Model the given requirements written in natural language into lang.
5. Translate the given requirements into lang’s syntax and semantics.

(Randomly choose one of the above.)

(For ACSL): You only need to return the lang formal specification with the code without explanation.
(For others): You only need to return the lang formal specification without explanation.

# Input
<Input goes here>

Figure 7: Prompt for ProofGen Task.
Prompt for ProofComplete Task (lang: a placeholder to be replaced by each formal specification language name.)

# Task Description (ProofComplete)
1. Please complete the following formal proof in formal specification language lang according to the given requirement.
2. Please complete the following formal proof in lang according to the given requirement.
3. Please complete the given formal proof in lang following the requirement below.
4. Please complete the following formal proof in lang according to the requirement below.
5. Please complete the following formal proof in lang according to the given requirement.

(Randomly choose one of the above.)

You only need to return the completed lang formal specification (together with the provided formal specification) without explanation.

# Input
<Input goes here>

Figure 8: Prompt for ProofComplete Task.

Prompt for ProofInfill Task (lang: a placeholder to be replaced by each formal specification language name.)

# Task Description (ProofInfill)
1. Please fill in the [MASK] in the following formal proof in formal specification language lang according to the given requirement.
2. Please fill in the [MASK] in the following formal proof in lang according to the given requirement.
3. Please complete the given formal proof in lang following the requirement below by filling in the [MASK].
4. Please fill in the [MASK] in the following formal proof in lang according to the requirements below.
5. Please fill in the [MASK] in the following formal proof in lang according to the given requirement.

(Randomly choose one of the above.)

You only need to return the completed lang formal specification (together with the provided formal specification) without explanation.

# Input
<Input goes here>

Figure 9: Prompt for ProofInfill Task.
Prompt for Code2Proof Task

# Task Description (Code2Proof)
1. Please fill in the [MASK] in ACSL according to the given requirement and ACSL specification.
2. Please fill in the [MASK] in ACSL according to the given requirement.
3. Please fill in the [MASK] in ACSL according to the given ACSL specification.
4. Please fill in the [MASK] in ACSL according to the given requirement and ACSL specification.
5. Please infill the [MASK] in ACSL according to the given requirement.

(Randomly choose one of the above.)

You only need to return the completed ACSL formal specification (together with the provided formal specifications and C programs) without explanation.

# Input
<Input goes here>

Figure 10: Prompt for Code2Proof Task.

Reviewer paper summaries: This paper divides formal verification into 6 tasks, and constructed 18K instruction-response data across 5 formal languages. The paper then benchmarks several papers and provides insights on several research questions including domain transfer. I think the field of LLMs for formal verification would benefit from having a comprehensive evaluation.
Summary of strengths: 1. The paper is well written, clearly structured and motivated. E.g. in the introduction, the list of contributions, alongside motivations for each one, are well presented, natural and convincing.

2. The 6 tasks in formal verification are clearly separated out from each other, and they are all convincingly important. This effort entangles the complexity and effectively makes the evaluation more to-the-point.

3. With this work mainly focusing on evaluation, it is good practice that the authors make substantial efforts to clearly introduce the data, from creation to statistics. It is believed that the data should have a beneficial impact on the community.
Summary of weaknesses: 1. For RQ3, the authors mainly compare finetuning with 3-shot. Here an interesting question would be how does the number of shots affect the results, and whether/how scaling up the number of shots slightly helps. This should be a quick experiment to do and will add more insights to this part.

2. The authors discuss how LLM finetuning on the formal verification tasks can benefit coding, reasoning, and math capabilities. A question that is at least as important is whether finetuning on some of the 6 tasks benefit each other. That is, would finetuning on one/some of the 6 tasks help with LLMs' capability on the others ones. Assuming the answer is yes, how significant? This may be an experiment requiring more time and compute, so I would like to suggest this addition that would further improve this paper, but by no means view it as required/necessary.

3. Are the authors planning to release full codebase and detailed instructions on how to reproduce all results? My current rating of this paper assumes yes but please let me know if that is not the plan. I would also suggest adding such instructions in the appendix to help others know how to reproduce/extend the evaluation.
Comments, suggestions and typos: 1. Line 479 -- the authors may want to have a new paragraph correctly indented.

Otherwise, I would like to comment again that I appreciate how this paper is very well written and the related works are thoroughly discussed.
Overall assessment scores: 4
Metareview score: 4
Metareview: This paper discusses a new method to understand the roles of LLMs in Verifiable Formal Proof, which is very interesting and will push our understandings of VFP forward. Therefore, this paper should be discussed in the conference.
Summary of reasons to publish: This paper is well written and the results are solid. All the reviewers support the acceptance and the authors have addressed most of the issues raised by the reveiwers.