Rethinking and Improving Autoformalization: Towards a Faithful Metric and a Dependency Retrieval-based Approach
Authors: Qi Liu
Xinhao Zheng
Xudong Lu
Qinxiang Cao
Junchi Yan
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Model
Formal Verification
Autoformalization
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: For statement autoformalization, we propose a faithful neural-symbolic evaluation method, a new task "dependency retrieval", a new data synthesis method, a research-level benchmark and a new paradigm that is augmented by dependency retrieval.

Abstract: As a central component in formal verification, statement autoformalization has been widely studied including the recent efforts from machine learning community, but still remains a widely-recognized difficult and open problem. In this paper, we delve into two critical yet under-explored gaps: 1) absence of faithful and universal automated evaluation for autoformalization results; 2) agnosia of contextual information, inducing severe hallucination of formal definitions and theorems. To address the first issue, we propose **BEq** (_**B**idirectional **E**xtended Definitional E**q**uivalence_), an automated neuro-symbolic method to determine the equivalence between two formal statements, which is formal-grounded and well-aligned with human intuition. For the second, we propose **RAutoformalizer** (_**R**etrieval-augmented **Autoformalizer**_), augmenting statement autoformalization by _Dependency Retrieval_, retrieving potentially dependent objects from formal libraries. We parse the dependencies of libraries and propose to _structurally informalise_ formal objects by the topological order of dependencies. To evaluate OOD generalization and research-level capabilities, we build a novel benchmark, _Con-NF_, consisting of 961 informal-formal statement pairs from frontier mathematical researches. Experiments validate the effectiveness of our approaches: BEq is evaluated on 200 diverse formal statement pairs with expert-annotated equivalence label, exhibiting significantly improved accuracy ($82.50\\% \mapsto 90.50\\%$) and precision ($70.59\\% \mapsto 100.0\\%$). For dependency retrieval, a strong baseline is devised. Our RAutoformalizer substantially outperforms SOTA baselines in both in-distribution ProofNet benchmark ($12.83\\% \mapsto 18.18\\%$, BEq@8) and OOD Con-NF scenario ($4.58\\%\mapsto 16.86\\%$, BEq@8).

Published as a conference paper at ICLR 2025
RETHINKING AND IMPROVING AUTOFORMALIZATION. TOWARDS A FAITHFUL METRIC AND A DEPENDENCY RETRIEVAL-BASED APPROACH
Qi Liu, Xinhao Zheng, Xudong Lu, Qinxiang Junchi Yan*1
Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University (purewhite, void 2xh, 1ww.dong20uoo https://github. wtwphhww....ooeeeeeeiieeccccecee te20 http://///eoet.iiioooooooooooooo ization
yan jts on
ABSTRACT
As central component in formal verification, statement autoformalization has been widely studied including the recent efforts from machine learning commu- nity, but still remains widely-recognizzed difficult and open problem. In this paper, we delve into two critical yet under-explored gaps: 1) absence of faith- ful and universal automated evaluation for autoformalization results; agnosia of contextual information, inducing severe hallucination of formal definitions and theorems. To address the first issue, we propose BEq (Bidirectional Extended Definitional Equivalence), an automated neuro-symbolic method to determine the equivalence between two formal statements, which is formal grounded and well- aligned with human intuition. For the second, we propose RAutoformalizer (Retrieval-augmeentdd Auteformalizer). augmenting statement autoformalization by Dependency Retrieval, retrieving potentially dependent objects from formal librarics. We parse the dependencies of librarics and propose to structurally infor- malise formal objects by the topological order of dependencies. To evaluate OOD generalization and research-level capabilities, we build a novel benchmark, Con- NF, consisting of 961 informal-formal statement pairs from frontier mathemat- ical researches. Experiments validate the effectiveness of our approaches: BEq is evaluated on 200 diverse formal statement pairs with expert-annotated equiv- alence label, exhibiting significantly improved accuracy (82.50% + 90.50%) and precision (70.59% + 100.0%). For dependency retrieval, a strong base- line is devised. Our RAutoformalizer substantially outperforms SOTA baselines in both in-distribution ProofNet benchmark (12.83% + 18.18%, BEq@8) and OOD Con-NF scenario (4.58% 16.86%. @).
Philosophy is written in this grand book, the universe. It is written in the language of mathematics.
Galileo Galilei, The Assayer
INTRODUCTION
Theorem provers, such as Lean (Mora & Ullrich 2021), Coq (Bertot & Castéran, 2013) and Is- abelle (Nipkow et al.) 2002). can check the validity and correctness of mathematical statements and proofs by strict algorithms, whose own soundness and completeness are proven in theory. However, instead of directly working on natural language mathematics, these tools define their own formal languages, which hinders the democratization of formal mathematics.
Statement autoformalization aims translating mathematical statements from natural language to formal verifiable statement. Readers unfamiliar with formal theorem proving are advised to read Yang et al. (2024). Due to its rigorously logical nature, this task wedely-recognizeed to chal- lenging, requiring profound understanding of both informal semantics and formal syntax (Li al.) 2024a). Beyond a fundamental component in formal mathematics and software verification, strong autoformalization methods have far broader impacts and could result in the creation of general
"Equal correspondence. *Also affiliated with Shanghai Artificial Intelligence Laboratory. This work was in part supported by NSFC (92370201, 62222607) and Shanghai Municipal Science and Technology Major Project under Grant 2021 X0222X00002

---

Published as a conference paper at ICLR 2025
purpose reasoning module (Szegedy
2020 Outside-the-box applications of autoformalization in-
clude synthesizing training dataset for formal theorem provers (Wu et al. 2022 Xin et al. (0024)
especially AlphaProof (Castelvecchi] 2024). enhancing informal math reasoning by rejection sam-
pling (Zhou et al. 2024). and automating code verification (Lin et al. 2024).
Current mainstream methods work in the following process. A large language model (LLM) is ei- ther prompted et 2022) or fine-tuned (Azerbayev et al.] 2023 Jiang et al., (023a) to directly generate a formal statement given its informal counterpart. The predicted statements are then evalu- ated by laborious human annotation (Azerbayev el al.,
2023) or unreliable proxy automated metrics
including machine translation metrics such as BLEU
(W et al. 2022) and perplexity (Wang al.)
2018), symbolic type check pass rate (Lu et al. 2024) or LLM grader (Ying et al. (024a)).
Rethinking this paradigm, we find out two key limitations, Firstly, an effective, human-aligned and universal automated evaluation metric is absent. Machine translation metrics are fragile to equiv- alent transformations in human perspective, for example 3-reduction (function application). Type check is too weak to filter out syntactically correct but semantically absurd autoformalization. It is a necessary but not sufficient condition for the ideal equivalence. LLM graders are non-determinant and highly dependent on prompts, and are easily misled by imperceptible but fundamental differ- ences or huge but nonessential transformations. Murphy et al. (2024) are pioneers to utilize SMT solver for faithful automated evaluation, but is restricted to Euclidean geometry only. Secondly, the current paradigm directly generates formal statements, ignoring the context of previously formal- ized statements and definitions. This might result in severe hallucination of identifiers and syntax, especially in -ii-r---ittrittiinn (OOD) cases. A similar issue is reported in Wu et al. (2022), where definition misalignment between informal mathematics and formal libraries is the major cause of failure cases. Our experiments on both in-domain and OOD scenarios, shown in Table 3 show the severity of this problem and exhibit a promising path to address it.
For the first issue, we propose BEq (Bidirectional Extended Definitional Equivalence), neural- symbolic equivalence relation between formal statements. This metric aligns well with collective human opinions. In formal systems built upon dependent type theory (Univalent Foundations Pro- gram| 2013), such as Lean 4 (Moura & Ullrich 2021), definitional equality is symbolic equivalence relation under a variety of intuitive transformations, such bound variable renaming, function ap- plication, and definition unfolding. However, heavily relies on the definitions of objects and con- version rules, hence it is too strict and inflexible from human perspective. For example, + 0 and n are definitional equal for a natural number n, but n and 0 + n are not. Worse still, definitional equality struggle with handling metavariable differences. We extend definitional equivalence by 1) equipping it with a restricted set of symbolic transformation primitives and a neural transformation function aiming to convert one formal statement to be definitionally equivalent to the other, and 2) loosing the equivalence criteria to bidirectionally "convertible" under the transformation func- tion. To evaluate its performance, we build a benchmark consisting of 200 formal statement pairs with expert-annotaterd equivalence labels. BEq significantly outperforms previous SOTA methods, improving the precision from 70.59% to 100% and the accuracy from 82.50% to 90.50%.
For the second, we propose new task, Dependency Retrieval, and a new method, RAutoformalizer Sketrieval-augmeen
Autoformalizer). Dependency retrieval seeks to select potentially dependent formal objects given an informal statement. R RAutoformalizer uses the retrievals to enhance autofor- malization. To enable this new paradigm, we propose to parse the dependencies in formal libraries and construct training data by topological informalization, informalizing formal objects by topolog- ical order. An immense dataset of 243,797 formal objects (including 139,933 theorems) is synthe- sized upon Mathlib 4. We also build the Con-NF benchmark" to evaluate -fsttrribtttttionn (ODD) generalization and research- level capabilities of current methods. A baseline is built for dependency retrieval, with 35.52% Recall@5 on ProofNet and 24.32% Recall@5 on Con-NE. RAutoformal- izer exhibits substantial improvement over previous methods, improving BFq@8 8 from 12.83% to 18.18% on ProofNet and from 4.58% to 16.86% on Con-NF.
To sum up, we identify two key limitations in statement autoformalization: 1) absence of faithful and universal automated evaluation; 2) agnosia of contextual information. The contributions are:
1) We give a neural-symbolic equivalence metric, BEq (Bidirectional Extended Definitional Equiv- alence), extending Definition Equality dependent type theory more aligned with human intuition.
Based on Lean 4 Con(NF) library (A formal consistency proof of Quine's theory New Foundations)
2

---

Published as a conference paper at ICLR 2025
2) We propose new dependency retrieval task and introduce novel paradigm, RAutoformalizer (Rerrievl-Augmmented Autoformalizer). We further propose topological informalization to synthe- size high-quality training data for these initiatives. To evaluate research-level autoformalization and out-of-distribution (OOD) performance, we create a new benchmark, Con-NF, which consists of 961 informal-formal statement pairs from New Foundations (Holmes & Wilshaw 2024).
3) We validate BEq by expert evaluation on 200 formal statement pairs and set a baseline for de- pendency retrieval. Extensive experiments of RAutoformalizer show its superior performance on statement autoformalization. Ablation studies further validate the effectiveness of our technical modifications, and also exhibit the great potential of the retrieval-augment paradigm.
2 RELATED WORKS
Autoformalization. It aims automatically translate the natural language (informal) mathemat- ics into formal verified code. Current autoformalization methods can be roughly divided into three levels. Statement autoformalization focuses on autoformalizing statements (Wang et al. 2020: Wu
et al. [022]] Azerbayev et al.] [023] Jiang et al., [2023a] Gulati et al.) 2024, Poiroux et al. 2024); proof autoformalization focuses on translating informal proofs (and sometimes including corre-
sponding statements) into formal code (Cunningham et al., 2023 Jiang et al.) [023b) Zhao et al.)
2023 Murphy et al., 2024 Lu et al., [024]; theory autoformalization, translating whole theory including definitions, axioms, theorems, and proofs, remains under-explored. Patel et al. (2024)
proposes a three-stage plan to break the difficulty into casier subtasks.
Methods of Autoformalization. Autoformalization is notoriously challenging for prevalent data- driven approaches (Li et al.] 2024b). Existing informal-formal parallel corpora are fairly scarce, which impedes machine learning training. To alleviate this, researchers synthesize informal-formal pairs by rule-based informalization (Wang et al. 2018 Cunningham et al. 2023). LLM-based back- translation (Azerbayev al.] 2023) Jiang al. 2023a), training with multilingual corpus (Tiang
et al. 2023a). or utilizing in-context learning (ICL) capabilities (Wu et al. 2022). Ying et al. (2024a)
proposes an expert iteration pipeline by iteratively synthesizing and filtering training data.
A major difference from machine translation is the existence of verifiers. Another line of work focuses on utilizing verifier feedbacks. Poiroux et al. (2024) uses rejection sampling to enhance aut- oformalization by typecheck results; [u et al. (2024) introduces a neural step-level verifier and per- form expert iteration: Jiang et al. (2023b): Murphy et al. (2024) combines LLM and formal verifier for proof autoformalization, and Zhao et al. (2023) enhances it with subgoal- -baed demonstration.
Evaluation of Autoformalization. There are many benchmarks for statement autoformalization, covering undergraduate-level math problems (Azerbayev et al. (023). more complex areas from Mathlib Gulati et al. 2024), and Euclidean geometry (Murphy et al. 2024).
Due to the high flexibility of natural language and the rigor of formal language, faithfully evaluating autoformalization is videly-recognized to be challenging and under-explored (Szegedy) 2020 Azer-
bayev et al.) 2023 Diang et al.) 2023a; Murphy et al.) 2024). Wu et al. (2022); Jiang al. (2023a);
Ying et al. (2024a) evaluate autoformalization results by human experts. Wang et al. (2018) reports
identical matching accuracy. Proxy metrics, including perplexity (Wang et al. 20188) BLEU Wang
eet al., 2018] Poiroux et al. 2024 Azerbayev et al. [2033] Wu et al. (002) and compiler typecheck
pass rate (Lu et al. 2024 Azerbayev al. [023]] Jiang al. 2023a) utilized to automate evalu-
ation. (Ying et al. (2024a); Gulati et al. (2024) prompts LLMs to determine the equivalence between
predicted formal statement and ground-truth. Murphy al. (2024) propose to use SMT solver to evaluate the equivalence between formal statements in Euclidean geometry.
For proof autoformalization, current evaluation focuses on theorem proving, only verifying formal proofs' correctness while potentially overlooking semantic inconsistencies between informal and formal proofs. The evaluation of theory autoformalization is also insufficiently researched.
Retrieval-augmmented Generation. It has been extensively studied in NLP. For code generation, code documentations (Zhou al.) (023). APIs (Zan al.) [2022), repository files (Zhang et al., 2023) and dynamic knowledge soup (Su et al. (024) are retrieved to augment generation. In formal verification, (Azerbayev et al. (2023) proposes to augment statement autoformalization by retrieving relevant prompts. ReProver (Yang al. 2024) enhances theorem proving with premise selection.
BBLEE (Papineni et al. 2002) is metric for evaluating machine translation based on n-gram matching.
3

---

Published as a conference paper at ICLR 2025
Statement Sp
theemen sq (6: : Tape") [Group 6] [Mantppe 6] (te2 Fuen (card 61)
(tactics) exact exact? have
theonen oP (6 Type*) [Group 6]
[Fintype 6] (h. Fennype.card 6
inddractionll Detrittional Implictioo tn TT(s 3R)
(a: 9), a
and te by
6, a
Edirectional Ditanded Defiiiional Eguivalance Sp mi Sq
Ithenne oP (6 Typery [Geeup 6] [Mattype 6] (h 1 Finbyse.car 6 2 by
G,
Underctiona Dedrrittatoo T(p R)
Statement Typu")
[Eintype (662 Fves (card 3.3. (a:: 6), a A
have 10 Evan (card asact?
exect
T(A Sz. R)
Figure I: Illustration of BEq (Bidirectional Extended Definitional Equivalence) and Unidirectional Desinitional Implication. 8p ~B &Q if and only if both &p <- &Q and &Q k-u 8p hold. To deter- mine the first, we assume se holds. Then the transformation function (implemented with a LLM) T is called to generate transformation (proof of sp using sq) conditioned sq and transformation primitive (tactic) set R. If the transformation holds, we conclude that Sp + U SQ. Otherwise, we believe Sp f- sQ- Vice versa for the second direction.
BIDIRECTIONAL EXTENDED DEFINITIONAL EQUIVALENCE
3.1 BACKGROUND
A fundamental problem for all generative tasks is to faithfully, effectively, and interpretably evaluate the results. statement autoformalization, we follow prevalent benchmarks such as ProofNet (Azer-) bayev et al. )023) and LeanEuclid (Murphy et al. (024) to evaluate by comparing model predictions with ground-trute truths: let denote the set of all formal statements, given a predicted formal statement Spred E S and the corresponding ground-truth truth Sgt S, equivalence relation ~e: S x S used to determine the correctness of autoformalization should be:
( equívalence relation: a binary relation with reflexivity, symmetry and transitivity. ..) is well aligned with human intuition.
(.>.) is universally applicable in all domains.
Intuitively, equivalence from a human perspective is generally one that can be quickly determined and reasoned. Hence, the key lies in defining an equivalence relation that can be demonstrated through brief proofs. We choose to build an equivalence relation that aligns with humans by 1) extending definitional equality, and 2) restricting the degree of proof automation.
Definitional Equality. In Lean 4 (Moura & Ullrich 2021), two expressions are definitionally equal if they are equivalent w.r.t. a series of conversion rules, such as a-conversion (renaming bound vari- able), n-expansion (modifying unused arguments in functions)), proof irrelevance (proofs of the same Prop), f-reduction (function application), Ç-reduction (eliminating let-in definitions), 5-reduction (unfolding variable and constant definitions), 6- t-reduction (application of recursive functions defindd on inductive types to an explicit constructor) (Bailey et al. 2024p. This equality is a binary relation with reflexivity, symmetry, and transitivity, and it is applicable in all math areas formalized in Lean And it has many intriguing characteristics that fit more closely with human instinct. For exam- ple, fun (b: Nat) => b is equivalent to fun (u Nat) => u because definitional equality allows cocovversoon, in which bound variable b is renamed to u.
However, several critical weaknesses hinder definitional equality from becoming good and intu- itive metric for autoformalization, Firstly, some expressions that are naturally "equivalent" from a not definitionally equal. For example, for natural number n ::Nat, and r. are definitionally equal, but 0 + n and n are not definitionally equal. Definitional equality heavily relies on the definitions of objects and conversion rules, while many intuitive equivalences, are neglected. Worse still, typecheck often get stuck in typeclass instance problems due to metavari- ables, which hinders evaluating definitional equality between statements.
human perspective
3.2 EXTENDING DEFINITIONAL EQUALITY
Formulation. Suppose there are two formal statements, sp and sQ. Without loss of generality, sp and SQ are assumed syntactically valid, since is nonsense to talk about equivalence between invalid formal statements. Definitional equality is denoted <pp
The main reason behind the aforementioned limitations of definitional equality is its strictness on reductions and conversions. We hence loose the limitation and extend definitional equality to align
4

---

Published as a conference paper at ICLR 2025
with human intuition. Let R be the set of all transformation primitives, (ss, R) : 2 + 2 to be the set of all valid formal statements that can be constructed by applying transformations in R c R on 8, and T (S (S x 2) + S to be restricted transformation function such that
8pp=
s'p E (((P,, R AQ ~D s/p
T(splsq, R)
(1)
Vslp E ((), sQ *D 5/p- L
Intuitively, given transformation primitives R. C R. T transforms sp definitionally equal to sQ if possible and returns the transformed statement. Otherwise, returns a dummy statement I, which is not definitionally equal to any other valid statement (e.g., an invalid statement).
In Lean 4, a formal statement can be converted to a proof goal by entering tactic mode. A proof goal ((8p;)) 8Q) consists of some assumptions (8p,)== and a conclusion sq. where all sp, and sQ are statements, and n can be 0. Then tactics, which are metaprograms, reduce one goal to another, which is often easier to solve by assumptions. For example, transforming ([S), R - S) to (), S) by tactic intro and trivially prove it by exact. formal statement 8p can be transformed proof goal by simply setting assumptions
be empty set and conclusion to be sp, resulting in the proof goal (6, sp). And a proof goal (/sp,))=1..800 can be transformed back to formal statement 8,,1 1.,1/ 8p2 ... A 8p, level, leaving semantics unchanged. Therefore, we can determine semantic equivalence in the space of proof goals and concretize IR be the set of all tactics in Lean. The restricted transformation function T can be approximated by sampling tactic sequences from a large language model and symbolically executing on Lean kernel multiple times, until a valid s'p is found, or the time limit ex- ceeds. With a slight abuse of notation, we denote both the formal statement sp and its corresponding proof goal as Sp.
sQ. These transformations occur at the syntax
Then, Unidirectional Definitional Implication (. 5 .) is defined as
8p <u 8Q
> 8p <' T(sQ (s, R)
Intuitively, this implication from SQ to 8p indicates whether the proof goal of the statement 8p can be definitionally equal to a restrictively transformed sq by T. Correspondingly, BEq (Bidirectional Extended Definitional Equivalence) (- ~B -) is defined as
8p
SQ
8p Fu SQ A SQ tu 8p
(3)
which is
a superset of definitional equality: Let R = 0, then, T becomes identity mapping (-)) and
8p ~g 8Q
sp ~m A(sQ) A sQ ~D A(sr)
sp ~p sQ
equivalence relation, which is binary relation with
1. Reflexivity: 8p <BB 8p holds because
8p.
2. Symmetry: 8p ~B sQ + sQ ~B sp holds by unfolding the definition of BEq.
Transitivity: If sp "B SQ and sQ ~' SR holds, we have sp ~p T(sglsp; R) and 8Q ~D T(sR|5Q, R). Suppose T(sq 8p, R) applies tactic sequence [[ pm to transform proof goal sQ to be definitionally equal to sp, and T sQ, R) applies (1()) [n Therefore, by applying Concat((u (J) [ 1 (m) on 8n. we can trans- form proof goal sæ be definitionally equal sp. Therefore, sp ~p T(sR|sp, R).
(
Implementation. An overview of BEq is depicted in Figure ! To implement the transformation function T, we perform 5-shot prompting MMtrrl.M-MMath-Puue
(Ying et al. 2024b) served on vLLM (Kwon et al.) 2023). If not mentioned otherwise, model prediction is sampled by beam search where temperature T 0.0, attempt number n = 8, and beam size 6 = s. The choice of trans- formation primitives is sophisticated and critical for aligning with humans. We set R {apply, cases', constructeo or, exact, exact? ?, ext, have, intro int rw, use] to extend vanilla definitional equality (for higher recall) while preventing U(- R) and the equivalence class being too large (for higher precision). More experiments on the choices of attempt numbers, trans- formation primitives, and sampling strategies can be found in Appendix A.1
5

---

Published as a conference paper at ICLR 2025
Comparison of automated evaluation metries for statement autoformalization. R, S. T denote reflexivity, symmetry, and transitivity, respectively. Universal indicates whether metric is applicable in all domains; 0/0 denotes division by zero; I and D denote InternLM2- MatMM-alsPPP2O0 and Deepseek-V2.5, respectively; ~ represents the metric is unsuitable for the method. "We report the best results among all thresholds; Reflexiiity and symmetry depends on the implementation.
Table 1:
Binary Relation
Alignment with Human
Metric
Universal
S
Precisiont
Recallf
Accuracyt
Identity Match
0/0
0.00%
65.00%
Typecheck
35.00%
100.00%
35.00%
BLEU Threshold
x²
62.96%
24.29%%
68.50%
Majority Voting
40.00%
94.29%
48.50%
Majority Voting (D)
x
70.59%
85.71%
82.50%
Definitional Equality
100.00%
11.43%
69.00%
E3 Murphy al. 2024
7
BFq
/
100.00%
72.86%
90.50%
Given two formal statements 8p and sq, we first check 8p tu 8Q- 8Q is assumed to be true by closing its proof with sorry. Then, symbolic heuristic exact? is called to generate a proof for 8p. If it fails, n candidates are sampled from the LLM given tactic restriction R and sQ. If there exists at least one successful proof that uses 8q. 8p < sQ holds. Otherwise, 8p t- sQ does not hold. Then 8Q u 8p is similarly checked. If and only if both directions hold, 8p ~g 8Q holds.
3.3 EVALUATION OF BEQ
Human Equivalence Benchmark. To fairly and reliably evaluate BEq and baseline metrics, we uniformly sampled 200 formal statements from the typechecked predictions generated by RAuto- formalizer and OpenAI ol-preview (100 predictions from each). Then the statements are paired with the ground-truths in ProofNet (Azerbayev et al., 2023 Experts in math and formal verifica- tion are invited to discuss and label the equivalence in their opinion for the 200 statement pairs. The discipline distribution of these samples is visualized in Appendix[A. A..4
Experiment Setting. In our evaluation, identical matching is optimized to neglect spaces in formal statements. BLEU computation is identical to Azerbayev et al. (2023). To determine pairwise equiv- alence, we binarize BLBU by a threshold. The best results over all possible thresholds are reported. Precision, recall, and accuracy curves of different thresholds can be found in Appendix A.5 For LLM grader, we use the prompts in Ying et al. (2024a) but a stronger setting: InterLLM2-Math Plus-20B (Ying et al. 2024b) and DeepSeek- V2.5 (DeepSeek-A] 2024) with 16-shot majority vot- ing and temperature
2024) is not evaluated, as it is only available in Euclidean Geometry. BEq also samples 16 tactic sequences candidates for each sample.
0.7. E3 (Murphy al.
Experiment Results 5 As summarized Table | BEq reaches 100.0% precision and 90.50% ac- curacy, showing landslide advantages over baselines. However, BEq falls short recall (-12.85% compared with "Majority Voting (D)") because of 1) rigor of formal verification systems: and 2) failure of approximated transformation function (the LLM), as analyzed in Appendix A.4 For baselines, AAzerbyyvv et al.] (2023) concludes that BLEU has a low correlation with ground-truth accuracy, with which our experiment result agrees. The distribution of BLEU scores of equivalent and inequivalent pairs is visualized in Appendix [A.5] L.LM Majority Voting sets strong baseline, reaching 82.50% accuracy, but the expense of precision. As subset of BEq, definitional equal- ity performs well in precision but has too many false negatives. With BEq, we can better evaluate statement autoformalization. the following, we will address the second issue, agnosia of context.
4 RETRIEVAL-AUGMENT
AUTOFORMALIZATION
The current autoformalization paradigm suffers from the agnosia of context. Autoformalizers, with- out priori knowledge of previously formalized definitions and theorems, frequently hallucinate
'etailled prompt template can be found in Appendix A.7
4 'll relevant open-source libraries are summarized in Appendix A.9
"More comprehensive results can be found in Appendix A.2.1

---

Published as a conference paper at ICLR 2025
Embedding
Informal Statement
Prove S/0 29078 not simple.
Mathlib
Formal Data
Informalizations
Encode
X
class Group (6: Type us extends divinottonlds
liches Group reprosens
Type
Renatd, Zen..
clace BivTaolonetd Type ectense
class DLV (at lypc v)) 1spa
field
(Slas Menooed repecants the concept of [Cllass Da expresents the concapt division, wlich
Punction Monold notural number
Encader
ArgMax Cusine Siilarity
Group
Clas time additann
Group Retrieve Fintype
Dxpendendes:
Retekkent Gan Beteediee
Generate
Autoformalizer
Cade:
Formal Statement
class Group 26 : Type
thworws the (G Type u,2) [Group
Topolooical Informalize
Informalizer
[Fintype 6] (h.0 Fintype. card 6 2907) IsSimpletroup 6 :- by sorry
A'trazt' au 'Noaddd
Figure 2: RAuteformalizer. Train: @Dependencies in a library (e.g. Mathlib 4) are parsed. Formal objects informalized by topological order, each given its and dependencies' information. The resulting parallel data is used to train the retriever (encoder) and autoformalizer. Inference: @Fach informal statement is encoded to embedding, whose cosine similarities computed with pre-computed library embeddings. (3Ojeect corresponding to top-k similarities are retrieved. 4 @Conditioned on the informal statement and retrieved dependencies, it predicts formal statements.
formal objects that are nonexistent in the library. This drawback is also observed as definition misalignment by Wu et al. (2022); Azerbayev et al. (2023); Jiang al. (2023a). Although these hal- lucinated identifiers and function applications are semantically correct from the human perspective, formal verification fails because of the soundness of symbolic verifiers. Our preliminary experiments support this observation, with hallucination worsening in OOD scenarios like frontier research.
4.1 RAUTOFORMALIZER
We propose RAutoformalizer Reetrieval-Annmmente Autoformalizer), which addresses the issue by incorporating dependency retrieval, selecting relevant Formal objects for given informal statement.
Dependency Retrieval. Suppose we are autoformalizing an informal statement Lp with ground- truth formal statement 8p with dependencies Dr from formal library D (e-g., Mathlib 4). Depen- dency retrieval aims to retrieve a subset of formal objects D, maximizing the number of dependent formal objects of 8p in while minimizing the inclusion of irrelevant ones, i.c.,
arg max D n Dpl ID Df. DE
(4)
Our retriever, tho : + Sh which embeds a string onto the surface In-dimensional unit sphere, uses Dense Retrieval (Karpukhin al.] (2020) for its popularity, simplicity, and efficiency. Before inference, the embeddings of the whole library are precomputed as {*))(82) 8a E W.. Then, when an informal statement l is provided, we only need single forward pass embed it as (b)()p) and retrieve formal objects with top-k maximal cosine similarities, see Figure 2 (Upper Right).
D arg max DE2P,
Σ #LED
(bb (1r),, (hosa)
(5)
Dataset. We build the dependency graphs for Mathlib 4, illustrated Figure (Bottom Left), by pars- ing the declarations of all formal objects and linking identifiers with accessible formal objects in the corresponding context. total, 243,797 formal objects (including 139,933 theorems) are collected along with their full names, positions, types, declarations, code, comments, and dependencies.
(
We propose to topologically informalize Mathlib 4 to synthesize a training dataset. Concretely, all formal objects topologically sorted and split into 24 topological generations based on their dependency graph. Informalization performed from the bottom (e.g.. basic definitions) to the top (more sophisticated concepts), as Figure 2 (Bottom Middle) shows. We use 10-shot prompted

---

Published as a conference paper at ICLR 2025
IntemLL2-MMen
Ying et al. 2024b) the informalizer. For formal object, the infor-
malizer is provided with the object's s declaration, codd comment, and its dependencies' informal- izations. The high quality of informalizations is shown in subsequent experiments.
RAutoformalizer. Building upon dependency retriever the an LLM P can predict formal state- ments given informal statements and retrieval results, in Figure 2 (Bottom Right):
8p ~ pa (.(.)1p,)) D)
(6)
The retriever tho is fine-tuned from BGE-M3 (Chen et al. 2023 using informalized theorems and dependencies in Mathlib 4 and hyperparameters in Appendix |A.8| We retrieve the top- candi- dates using pretrained BGE-M3, remove true dependencies, and take the remainings hard nega- tives (Xiao et al. 2023). By default, formal declarations of objects are used to generate embeddings.
For each theorem object, top-5 retrievals of zhe are collected to fine-tune the autoformalizer Ps from IntemL.M2-Math Maai-MM2-MMt--Bass---- (Yig et al. 2024b) using the training recipe in Appendix A.8]
During inference, given an informal statement 1p and formal library D, the retriever the selects top-5 candidates from the library, then the autoformalizer P generates formal statements based on the informal statement and retrievals.
Existing benchmarks (Azerbayev et al,, 2023 Zheng et al., 2022 Tsoukalas et al. 2024 Liu al.] 2023) Murphy et al.] (024) rely on Mathlib 4 and concentrate on high-school or undergraduate level mathematics. To evaluate the out-of-distribution generaliza- tion capabilities and research-level mathematics, we build novel benchmark, Con-NF, based on Lean 4 Con(NF) (Holmes & Wilshaw 2024) library. Con(NF) is recently published digitization of Randall Holmes' proof (Holmes) 2015) that Quine's New Foundations (Quine) 1951) consistent.
Con-NF: OOD Benchmark.
We parse dependencies in this library, topologically informalize all 85,763 762 formal objects, dedupli- cate theorems from Mathlib 4, and eliminate unused formal objects of the remaining theorems. The cleaned benchmark consists of 961 theorems based on a different theoretical basis to merely Mathlib 4, along with a total of 1,348 formal objects and their informalizations.
4.2 EVALUATION OF RETRIEVAL AND AUTOFORMALIZATION
Dependency Retrieval. We choose pretrained BGE-M3 and BM25 RRobertson al. 2009) base- lines. BGE-M3 is state-of- the-----thhhtaate art embedding model that can perform accurate semantic retrieval for more than 100 languages. BM25 a classical information retrieval method based on frequency and document length and is the main baseline in ReProver (Yang et al. 2024). For BGE-M3 baseline, we evaluate the pretrained model; For BM25, BPE tokenizer with 30,000 vocabularies trained on the topologically informalized Mathlib 4 dataset. For each ablative setting in experiments, separately fine-tuned one retriever with the same recipe in Appendix A.8 Evaluation is conducted on the ProofNet (Azerbayev et al. 2023) and the Con-NF benchmark.
Results in Table 2 suggest the superiority of our method. Models fine-tuned on dependency re- trieval dataset show landslide victory over baselines, exhibiting more than 10x improvement of recall on ProofNet and 2x on Con-NF. The huge performance gap between baselines focused on semantic similarity and our model indicates that dependency retrieval is novel retrieval task, which relies more on logical dependency. For a more intuitive analysis, a case study can be found in Ap- pendix A.6 Ablative results on topological informalization also demonstrate consistent advantage over vanilla informalization, especially in OOD generalization (Con-NF), where relative improve- ments can reach 50% on Recall and Precision @5 Comparisons between formattings of formal objects indicate that incorporating informalizations in dependency embedding might introduce noise and degrade retrieval performance in-distribution settings but improve OOD performance. We leave the exploration of this intriguing phenomenon for future work.
Statement Autoformalization. We evaluate a wide range of baselines, including in-context learn- ing (Wu et al. (2022) using GPT-4o (OpenAl et al. (024) and DeepSeek- V2.5 (DeepSeck-Al) (024)) and fine-tuning on MMA (Jiang et al.| 2023a). PDA (Lu et al., 2024), and Lean-Workbood (Ying et al. 2024a). Since LLM API calling does not support beam search with T 0.0. DeepSeek is
evaluated using temperature decoding = 0.7, and GPT-4o using version 9pt-40-2222408-
FFor theorems, only use their declarations since their code (except proofs) is identical with their decla- rations in semantics.
8

---

Published as a conference paper at ICLR 2025
Table Comparisons between our dependency retriever and baselines, and ablations of topological informal- ization. Cyan numbers in brackets show ablative improvements over vanilla informalization (U); Bold numbers emphasize the highest values in each benchmark; Fmt indicates the method to format a formal object into string to emhed, where F denotes using only formal declarations and F+IF means using both formal dec- larations and informalizations; DR represents dense retrieval; Dataset indicates the training dataset, where means directly using pretrained model, U represents unstructurally informalized dataset, and represents topo- logically informalized dataset; R@ k and P@ & denote the recall and precision of top-k retrievals, respectively.
Bench
Fmnt
Method
Dataset
R@st
R@
R@1001
P@51
P@10
1@100
BM25
0.16%
0.16%
1,00%
0.11%
0.05%
0.03%
F
1.93%
2.13%
7.14%
1.02%
0.61%
0.24%
DR
U
33.74%
40.31%
65.30%
21.55%
13.61%
2.22%
ProofNet
T
35.52%
43.63 %
67.71% (2429)
22.89%
14.57%
2.25%
BM25
0.00%
0.11%
0.29%
0.00%
0.05%
0.01%
F+1F
P
0.41%
0.98%
5.46%
0.32%
0.40%
0.20%
DR
U
28.66%
34.57%
63.55%
18.18%
11.28%
2.16%
32.47%
40.35% on9u
67.33% usme
20.32% (2309)
12.81 n.90%)
2.26%
4.41%
7.31%
31.13%
2.37%
2.23%
1.06%
F
5.66%
9.10%
34.50%
3.73%
3.02%
1.15%
DR
15.28%
20.31%
72.70%
7,95%
5.47%
2.39%
Con-NF
34% aiai:
37.44% 1991
88.86% (19109)
14.05%
11.29% (vers
199% (199))
BM25
T
9.86%
14.95%
34.50%
6.95%
5.28%
1.26%
F+IF
13.84%
19.19%
44.16%
9.51%
6.72%
1.59/%
17.34%
23.10%
84.25%
10.39%
7.29%
3,05%
T
27.91 (1176)
37.00% (3.000)
86.43%
(210
17.57 % (2199))
11.99% % (1.a))
3.21
(2.000
and default hyperparameters. For both, we set repeat count i = 8 (retry if fail to extract formal state- ments from model outputs) and use 3-shot demonstrations. Notably, ProofNet participates in the data synthesis process of Lean- Workbook. But we still include it as strong baseline. For faimess, all fine-tuning methods use M22ernL-M2-Mtth--Bass
(Ying et al. 2024bb as base model and train- ing recipe in Appendix A.8 We also report the performance of R RAutoformalizer without retrieval module (RA -R) and given ground-truth dependencies (RA +R). Both are fine-tuned respectively on correspondingly constructed datasets. For ProofNet. additional objects defined beyond Mathlib are retrieved in priority. For reproducibility, all fine-tunin fine-tuning methods are evaluated using beam search with temperature T = 0.0, generation number
8, and beam size b 8.
We use BEq (introduced in Section 3.2 to evaluate the equivalence between model predictions
and ground-truth formal statements. We define BEq @ k as the portion of samples where predicted statements are BEq to ground-truths at least once in k attempts:
BEq@k
1
Σ
max 18, JE(I...,kk
where N is the number of samples; is the number of attempts; is the indicator function, and & the j-th generation attempt for the i-th sample. Similarly, Typecheck @ k is defined as the portion of samples where model predictions pass Lean typecheck at least once in & attempts.
Typecheck@te
max [LemtTypcchec((s...) jefl....kk
We report BEq @1 1, BEq@8, Typecheck@1. @1 and Typecheck@8 for a more thorough evaluation.
Table shows the great superiority of RAutoformalizer over baselines. On in in-distribution ProofNet benchmark, the non-retrieval ablative model already surpasses all baseline methods, including Lean- Workbook (Ying et al. 2024a) (by 6.69% in BFq@88) showing the high quality of our topological in- formalizations. RAutoformalizer further improves 1.60%. The ideal model reaches 23.26% BFq @ and 31.28% BEq @8, exhibiting the potential of dependency retrieval.
On OOD Con-NF, without retrieval, all methods, including trng-sccle-ppeeraine GPT-4o and DeepSeck-V22 result poor performance. The non-retrieval ablative model still shows the highest
Another group of experiments fine-tuned DeepScck- Baaeec---ahBBBassteae 7B can be found in Appendix A.3

---

Published as a conference paper at ICLR 2025
Table 3: Comparisons and ablations of retrieval-augiee -uggmett.. Cyan numbers in brackets show ablative improve- ments over bare autoformalizer ("RA RR"); Bold numbers emphasize the highest values excluding oracle ("RA +R") results; BEq @ k indicates the portion of samples where predictions are equivalent to ground -ruths under BEq at least once in l attempts, defined in Eq. / Typecheck @ k indicates the portion of samples where predic- tions pass typecheck least once in & attempts, defined in Eq. 9 ICL (D) and (4o) represents in-context learning using Deepseek- V2.5 and GPT-4o, respectively; MMA, MMA (Lean), PDA, and LW represent fine- tuning on MMA, MMA's Lean subset, PDA, and Lean-workbook, respectively; RA is the main method: RA
R is the ablation removing dependency retrieval; RA +R is the ablation using oracle dependencies.
ProofNet
Con-NF
Metlod
Typechock
BEq@1:
Typechecká8))
BEq08
Typecheck(11
BEq@11
Typecheck08;
BEg@8
ICL. (D)
40.37%
9.89%
$1,07%
10.96%
9.37%
281%
16.23%
4.27%
ICL
43.58%
7.22%
66.31%
12.83%
9.78%
1.46%
20.71%
4.16%
12.57%
1.87%
22.99%
2.94%
3.64%
1.8%
8.74%
4.37%
MMA (L)
10.96%
2.14%
23,53%
2.67%
3.33%
1.77%
8.01%
4,58%
PDA
14.71%
0,27%
24.33%
2,14%
4.37%
1,04%
10.61%
3,64%
44.92%
8.56%
49.20%
9.89%
28.10%
0.94%
37.67%
1.04%
RA -R
52.14%
11.50%
71.39%
16.50%
8.12%
3.02%
11.97%
4.58%
RA
57.22%
12.30%
77.27%
18.18%
20.50%
11.45%
28. 62.
16,86% 9
+R
72.99% (uraiN
23.26%
80.48%
31.28% (17791
60.46%
OLANN
44.85%
12.115 (ALLON)
55.36% (40291
@Eq@1 1 and @ among them. With aeurent RAutoformalizer has 3x improvement on BEq @1 I and @ and the oracle model exhibits over 10x potential for improvement. This significant gap shows the necessity of dependency retrieval and draws community attention to OOD settings. A more detailed ablative study can be found in Appendix A.2
CONCLUSION
We have presented a thorough rethink on existing statement autoformalization paradigms, identi- fying and addressing two critical problems: absence of universal human- human-aligned evaluation metric and agnosia of contextual information. For the first, we propose BEq (Bidirectional Extended Def- initional Equivalence), faithful, effective and universal neural-symbolic approach to determine the equivalence between formal statements. For the second, we propose a new task, Dependency Retrieval, finding dependent formal objects from math libraries, and new paradigm, RAutofor- malizer deerieter--uummented Autoformalizer), enhancing statement autoformalization with depen- dency retrieval. We also propose to parse dependencies and topologically informalize formal objects to synthesize high-quality data. For more comprehensive evaluation, we extend ProofNet benchmark for dependency retrieval and construct novel research-level OOD benchmark, Con-NE.
6 LIMITATION AND BROADER IMPACTS
Limitations of BEq. As an equivalence metric between formal statements, the accuracy of BEq depends on the quality of the ground-truth formal statements of the benchmarks. Therefore, BEq is not suitable for benchmarks with low-quality ground-truths or those lacking formal ground-truths.
Moreover, human opinions equivalence are diverse. Therefore, carefully designing the limitation of transformation primitives R (available tactics) and the approximation of transformation function I' (the LLM) is crucial, for which extensive experiments are conducted in Appendix A..] For the more detailed case study of BEq, please refer to Appendix A.4 We sincerely invite community efforts to delve into refining BEq and set domain standard to facilitate subsequent research.
Limitations of RAutoformalizer For retrieval-augnee -aggmet generation, high-ranking retrievals mainly impact its performance (Cuconasu et al. (024). Although RAutoformalizer surpasses all baselines by significant margin, the experiment of oracle retrieval (RA +R) exhibits large room to improve the retriever. This project focuses on setting basic working baseline for dependency retrieval and leaves sophisticated upgrades such as multi-vector embeddings (Khattab & Zaharia 2020), re- ranking (Zhuang et al., (002) and query augmentation (Gao et al. 2024g for future work.
Broader Impacts. We hope the idea of bidirectionally "convertible" under restricted transforma- tions can inspire more areas, such as symooli, formal verification, and general reasoning. For example, faithful automated evaluation in other symbolic generative tasks. Furthermore, searchers can also extend RAutoformalizer to broader neural-symbolic tasks such as the autofor- malization of specifications, proof, and even theories.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY STATEMENT
It aims to contribute to the field of statement autoformalization by proposing a faithful equivalence metric, research-level benchmark, and new paradigm for mitigating agnosia of context and en- hancing OOD generalization. We fully understand the importance of reproducibility in scientific research and therefore, details of datasets, models, and experiments are summarized follows:
Implementation details of BEq in Section 3.2
Experiment settings and baselines for BEq in Section 3.3
Training dataset for dependency retriever and RAutoformalizer in Section 4.1 and string formatting details in Appendix [A.8]
Construction and composition of the Con-NF benchmark Section 4.1
For dependency retriever, implementation details and experiment setting in Section 4.2 and detailed training recipe in Appendix (A.8
For RAutoformalizer, implementation details, experiment setting, and evaluation metric in Section 4.2 and detailed training recipe in Appendix A.8
All dependent open-source libraries, along with their repository URLs and versions in Ap- pendix A.9
Evaluation results uploaded as supplementary materials. Code, data, and model checkpoints released at httpps://gt thub. wwwp///ueee.tttteeee/reeeeeeeeee htt.://iiiiinaaaattoooooaaaaaaaa
REFERENCES
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizigg and formally proving undergraduate-level mathematics, 2023. URL.
htttsss/arraiv..rr//b////333
12433
Chris Bailcy, Pietro Monticone, Martin Dvořák, Kevin C, and Kitamado. Type checking in Ican 4. https: 2024.
//github. ccm http:///nnnntttyyy__ocom http://aiaiiiittppoooooooooooooo
Yves Bertot and Pierre Castéran. Interactive thcorem proving and program development: Coq' Art: the calculus of inductive constructions. Springer Science & Business Media, 2013.
Davide Castelvecchi. Deepmind hits milestone solving maths problems-ai's next grand challenge. Nature, 632(8024):236 237, 2024.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi- lingual, multi-functionality. multi-granularity text embeddings through self- saff-knowledge distillation, 2023.
XTuner Contributors. Xtuner: toolkit for efficiently fine-tuning
11m.
com/
InternIM/ xtuner 2023.
Florin Cuconasu, Giovanni Trappolini. Federico Siciliano. Simone Filice. Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for rag systems. In
Proceedings of the 47th International ACM SIGIR Conference Research and Development Information URL
Retrieval, volume of SIGIR 2024, pp. 719-729. ACM, July 2024, doi: 10. 00.114//66267723667777 http:/ / dx. dci, org/ 10. 1141 32236626772222 3657834
Garett Cunningham, Razvan C. Bunescu, and David Juedes. Towards autoformalization of mathematics and code conrectness: Experiments with elementary proofs, 2023.
02195
DeepSeck-AL. AL. Deepseck-v2 v2: A strong, economical, and efficient mixture-of- ex-eeettss language model, 2024,
Guoxiong Gao, Hancheng Ju, Jicdong Jiang, Zihan Qin, and Bin Dong. A semantic search engine for mathlib4, httpa/bb///224030 13310
2024. URLtttts:///axiii http:///axvvvv..o
Aryan Gulati, Devanshu Ladsaria, Shubhra Mishra, Jasdeep Sidhu, and Brando Miranda. An evaluation bench- mark for autoformalization in Iean4, 2024,
06555
M Randall Holmes. New foundations is consistent. Change, 2015.

---

Published as a conference paper at ICLR 2025
M. Randall Holmes and Sky Wilshaw. NI is consistent, 2024. [ttppss////// wtt.ggggg/b/ac..3aa
01406
Albert Q. Jiang, Wenda Li, and Mateja Jamnik. Multilingual mathematical autoformalization, 2023a. URL
https://arxin: 2 03755
Albert Qiaochu Jiang, Sean Welleek, Jin Peng Zhou, Timothee Lacroix, Jiacheng Liu, Wenda Li, Mateja Jam- nik, Guillaume Lample, and Yuhuai Wu. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. The Eleventh International Conference on Learning Representations, 2023b. URL https://cpenrest
net/ ? SWa 9EAOvKMC
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. Linguistics. doi: 10. 1...8663/112222..mmll m50n.550. URL
Online, November 2020. Association for Computational orgg/20000..
mam.lppmmii..500
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, pp. 39-48, New York, NY, USA, 2020. Associa- doi: 00.1145/399277133400
tion for Computing Machinery. ISBN 97814500 9781450380164.
URL https:
7/doi www./rrggg. 10. 1145/ 3397271 3401075
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL /Bttp::///aaxx http:/aaawiw...aaaeaaae http//bbs///9//// 220./002890000
Zhaoyu Li, Jialiang Sun. Logan Murphy, Qidong Su, Zenan Li, Xian Zhang, Kaiyu Yang, and Xujie Si. A survey on deep learning for theorem proving. First Conference on Language Modeling, 2024a. URL
bittps:/cppenreeee wttpeeeweeeeeeeeeeeeeeeeeeeeeeee www.mooiddaaa. 2==
Zhaoyu Li, Jialiang Sun, Logan Murphy, Qidong Su, Zenan Li, Xian Zhang, Kaiyu Yang, and Xujie Si. survey on deep learning for theorem proving, 2024b. URL http::tpp::// http:://axvvvvv..o abs/ 09939
Xiaohan Lin, Qingxing Cao, Yinya Huang, Haiming Wang, Jianqiao Lu, Zhengying Liu, Linqi Song, and Xiaodan Liang. Fvel: Interactive formal verification environment with large language models via theorem proving, 2024. htttht:s:///xxxt wwwp:xxwwii..rraooooooo otgg/bb//244555 14408
Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, Ming Zhang, and Qun Liu. Fimo: A challenge formal dataset for automated theorem proving, 2023. URL B https:t//axxiiii wrg. abs 2309. 04295
Jianqiao Lu, Zhengying Liu, Yingjia Wan, Yinya Huang, Haiming Wang. Zhicheng Yang, Jing Tang, and Zhijiang Guo. Process-driven autoformalization in lean 4, 2024. URL
http//xiivv.rr//bb... /
2406.01940 01940
Leonardo de Moura and Sebastian Ulirich. The lean 4 theorem prover and programming language. In Automated Deduction-CADE 28: 28th International Conference Automated Deduction, Virtual Event, July 12-15, 2021, Proceedings 28, pp. 625 635. Springer, 2021.
Logan Murphy, Kaiyu Yang, Jialiang Sun. Zhaoyu Li, Anima Anandkumar, and Xujie Si. Autoformalizing cuclidean geometry, 2024. URL http::/t/////t
17216
Tobias Nipkow. Lawrence C Paulson, and Markus Wenzel. Isabellc/HOL: proof assistant for higher-order logic, volume 2283. Springer Science Business Media, 2002.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadeut-haapin
Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Made- laine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fish- man, Juston Forte, Isabella Fulford, Ieo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni,
12

---

Published as a conference paper at ICLR 2025
Gabriel Goh, Rapha Gontijo-Lpess, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufci Guo, Chris Hallaey, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidocke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Eukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger. Vishal Kuo, Michael Lampe, Ikai Lan. Teddy Lee. Jan Leike, Jade Leung. Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin. Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katic Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mchta, Jacob Mcnick, Luke Mctz, Andrey Mishchenko, Pamela Mishkin, Vinnic Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reücchro Nakano, Rajeev Nayak, Arvind Neelakanta, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto. Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Pow- ell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rac, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ry- der, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens. Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever. Jie Tang, Nikolas Tezak. Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian. Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong. Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang. Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https: (tlps://arx.x arvvvv.vvv... org/ abs/ 2303. 2303.08774
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Fugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting the Association for Computational Linguistics. pp. 311-318. Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics. doi: 00.3115/1073083.107333 URL bttps: //aclanthe owwp:y/-0rr0/00200000000000
Nilay Patel, Rahul Saha, Jeffrey Flanigan. A new approach towards autoformalization, 2024. URL https:
www.wi.ccg/.as///./eaeea. 07957
Auguste Poiroux, Gail Weiss, Viktor Kunčak, and Antoine Bosselut. Improving autoformalization using type checking, 2024, URL
wttp://g//b///.....///// 07222
WV Quine. On the consistency of "new foundations". Proceedings of the National Academy of Sciences, 37 59:00, 1951.
Stephen Robertson. Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, ((4):333-389) 2009.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning open language models, 2024. https:/ //a/aaxxvvvirr..///a abs/ 2402.03300
Hongjin Su, Shuyang Jiang, Yuhang Lai, Haoyuan Wu, Boao Shi, Che Liu, Qian Liu, and Tao Yu. Arks: Active retrieval in knowledge soup for code generation, 2024. URL.
12317
Christian Szegedy. A promising path towards autoformalization and general artificial intelligence. In Christoph Benzmüller and Bruce Miller (eds.). Intelligent Computer Mathematics, pp. 3-20, Cham, 2020. Springer International Publishing. ISBN 978-3-030-55518..
George Tsoukalas, Jasper Lee, John Jennings, Jimmy Xin, Michelle Ding, Michael Jennings, Amitayush Thakur, and Swarat Chaudhuri. Putnambench: Evaluating neural proverss the putnam mathe- matical competition, 2024. URL httl.hhtts:////// wttp/aawwwwwvr.raa//aaaaaa/////e 11214
13

---

Published as a conference paper at ICLR 2025
The Univalent Foundations Program. Homotopy Type Theory: Univalent Foundations of Mathematics.
httpoa///onnccccyyypcciooooooo./ ortp//ooor..gg/oooooooooooo Institute for Advanced Study, 2013.
Qingxiang Wang, Cezary Kaliszyk, and Josef Urban. First experiments with neural translation of informal to formal mathematics, 2018.
06502
Qingxiang Wang, Chad Brown, Cezary Kaliszyk, and Josef Urban. Exploration of neural machine translation in autoformalization of mathematics in mizar. In Proceedings of the 9th ACM SIGPLAN International
Conference Centified Programs and Proofs, volume 5 of POPI. '20, pp. 85-98. ACM, January 2020. doi: 10. 10.11445/377288533333777 URL http:ttp: //dx:::/dd..e www.ddxddoooooooo..ooooo org. 100 1et.e111133388888 33777788888833888 3373827
Yuhuai Wu, Alhert Q. Jiang. Wenda Li, Markus N. Rahe, Charles Staats, Mateja Jamnik, and Christian Szegedy. Autoformalization with large language models, 2022. URL httthh:s:///// http:/awwivv......oaaaaa abs////////// 22tp//205... 12615
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. peepseek-prooerr Advancing theorem proving in Ilms through large-scale synthetic data, 2024. URL
https:
14333
Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song. Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with serieval-augmennee language models. Advances in Neural Information Processing Systems, 36, 2024.
Huaiyuan Ying, Zijian Wu, Yihan Geng, Jiayu Wang. Dahua Lin, and Kai Chen, Lean workbook: A large-scale lean problem set formalized from natural language math problems, 2024a. URL httts:://axxi.. org/ abs / 03847
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang. Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang. Kai Chen, and Dahua Lin. Internlm-math: Open math large language models toward verifiable reasoning, 2024b. URL iuttps://arxe org/ abs 222224222003333
Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, and Mian-Gunng Lou. When language model meets private library, 2022. URL http:pp:////xxw...r...///
17236
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-leve code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods Natural Language Processing, pp. 2071244722222 14444, Singapore, December 2023, Association for Com- putational Linguistics. doi: 10.18663//1/2023.coom htt.::3..mmmlpmmmimmmm 2023. emn p-mmann..... 15
Xueliang Zhao, Wenda Li, and Lingpeng Kong. Decomposing the enigma: Subgoal-based demonstration learning for formal theorem proving. 2023. JtLLhttps::///// www.exivv.oog.aao. /t//a/ab///33// 221//335..11 16366
Kunhao Zheng, Jesse Michael Han, Stanislas Polu. Minif2f: cross- benchmark for formal olympiad-leve level mathematics, 2022. URL/attps://arxl http:////aaaaw....ga////////0// 00110
Jin Peng Zhou, Charles Staats, Wenda Li, Christian Szegedy, Kilian Q. Weinberger, and Yuhuai Wu. Don't trust: Verify grounding II quantitative reasoning with autoformalization, 2024. URL https: sttps://arxiv. /arxiv. org www.////4433 18120
Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. Decprompting: Generating code by retricving the docs, 2023. URL htttpp://arxxi.orgg//eeee
05987
Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. Rank15: Fine-tuning 15 for text ranking with ranking losses, 2022. URL ateps: // /arxiv..
org abs /222222111.. 10634
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
COMPARATIVE EXPERIMENTS ON HYPERPARAMETERS OF BEQ
Extensive experiments are conducted to evaluate the influence of different engineering choices, as shown in Table 4 Experiment dimensions include the restrictions of transformation primitives, choices between BEq and only Unidirectional Definitional Implication, number of attempts generate transformations, and sampling strategy.
As for the restrictions, Basic denotes only fexact, exact?, have} are allowed, Normal additionally in- cludes fapply, yy, cases', constructon ext, intro, intros, rw, use Advanced additionally allows powerful tactics (assumption, by_cases, by.ccntra, change, choose, convert, exfalso. left, nth_rw, obtain, rcases, refine, Ifi, right, rintro, special notes all tactics are allowed. Experiment results show that Basic setting enough for most cases and Normal setting shows superior performance, while Advanced and All may lead to false positives.
triv), and All de-
Comparison between "Bidirectional" and "Unidirectional" shows landslide advantage of "Bidirectional", Ex- periments of K show that symbolic heuristic exact? is able handle most cases, but the incorporation of large language models can solve more cases. It also reveals that current implementation, few-shot prompt- ing LLM,, not capable handling more difficult cases. Failure case analysis is done Appendix [A.4] The sampling strategy does not have much influence, so we use beam-search with temperature = 0 in the main experiments.
A.2
MORE RESULTS ON RAUTOFORMALIZER
A.2.1
COMPREHENSIVE HUMAN EVALUATION OF BEQ
Table 3 compares the autoformalization performance of RA with other baselines ProofNet and OOD Con-NF using two automated metries: Typecheck BEq. Because the robustness BEq itself limited discussed ahove, the significance of table results compromised unless human evaluations are provided.
To more reliably evaluate BFq and R Autoformalizer, for each experiment on each benchmark, about 100 model predictions that pass the typecheck are sampled for human evaluation. To reduce the variance, we perform stratified sampling in 3 groups: 1) both directions of UDI (Unidirectional Definitional Implication) fail; 2) single directional UDI succeeds; 3) both directions UDI succeed (BEq). The results are shown in Table
Results on ProofNet benchmark are consistent with Table Moreover, BFq demonstrates nearly perfect accu- racy on Con-NF. Therefore, BEq is robust as an automated evaluation metric for autoformalization tasks.
A.2.2 HUMAN-RECTIFIED RESULTS
According the human evaluation results in Appendix A.2.
estimate the gold accuracy of our methods
Human Typecheck@1 x HymennAc/Tyyotees
(9)
where Typecheck@1 is the portion of samples where predictions pass typecheck in one attempt; HumanAce typecheck is the human evaluated model accuracy among sampled typechecked predictions Ap- pendix 1 Results are shown in Table (6] which demonstrates clear ablative improvement among RA -R. RA and RA +R on the estimated goal accuracies.
A.2.3 TYPECHECK ERROR DISTRIBUTION
To quantitatively delve into the underlying mechanies of the ablative improvement brought by RAutoformalizer, for each experiment, we count Lean emoor samples
typecheck and classify them into two sources: "Hallucination" (error caused by hallucination of identifiers) and "Others" (all other errors). The results are in Table ? which show retrieval-augment can reduce both types of errors, especially Hallucination errors.
The detailed taxonomy is follows:
function expected: Others
invalid field notation: Hallucination
type expected, got: Others
unknown constant: Hallueination
15

---

Published as a conference paper at ICLR 2025
Table 4: Comparative experiments proposed equivalence metric on the human annotated equivalence benchmark. Green- backgrounedd numbers are those reported in Table | bad-backgodddeed numbers high- light false positives, which we' re trying our best to avoid. Restriction represents the allowed transformation primitives.; Bidirectional indicates to determine equivalence by BEqq Unidirectional indicates to determine equivalence Unidirectional Definitional Implication; K denotes the number of attempts to generate trans- formations; T=0.0 means beam-search with temperature T 0.7; FP denotes the number of false positives.
0; T=0.7 means temperature sampling with
T=0.7
Restriction
Direction
K
FP Precision
Recall
Accuracy
FP
Precision
Recall
Accuracy
100.00%
67.14%
88.50%
100.00%
67.14%
88.50%
100.00%
70.00%
89.50%
100.00%
70.00%
89.50%
Ridirectional
100.00%
70.00%
89,50%
100.00%
70.00%
89.509
100.009 00%
70.00%
89.50%
100.00%
70.00%
89.50%
100.000 100.00%
70.00%
89.50%
100.00%
70.00%
89.50%
16
100.00% 16
71.43%
90.00%
100.00%
71.43%
90.00%
Basic
0
75.00%
68.57%
81.00%
16
75.00%
68.57%
81.00%
74.29% 18
73.91%
72.86%
81.50%
18
74.29%
74.29%
82.00%
Unidirectional
74.29%
82.00%
73.61%
75.71%
82.00%
73.61%
75.71%
82,00%
74.32%
78.57%
83.00%
8
19
74.32%
78.57%
83.00%
21
72.37%
78.57%
82.00%
16
71,25% 23
81.43%
82.00%
70.89% 23
80.00%
81.50%
100.00%
67.14%
88.50%
100.00%
67.14%
88.50%
100.00%
71.43%
90.00%
100.00%
70.00%
50%
100.00% .00%
71.43%
90.00%
100.00%
70.00%
89.50%
Bidirectional
100.00%
71.43%
90.00%
100.00%
71.43%
90.00%
100.00%
71.43%
90.00%
100.00%
71.43%
90.00%
16
100.00%
72.86%
90.50%
100.00%
72.86%
90.50%
Normal
75.00% 17
68.57%
81.00%
75.00% 18
68.57%
81.00%
74.63%
71.43%
81.50%
74.29%
74.29%
82.00%
Unidirectional
2
18
74.29%
74.29%
82.00%
18
74,29%
74.29%
82.00%
19
73.97%
77.14%
82.50%
18
74.29%
74.29%
82.00%
19
74.32%
78.57%
83.00%
74.32%
78.57%
85.00%
16
71.79%
80.00%
82.00%
74.03%
81.43%
83.50%
100. 100%00%
67.14%
88.50%
100.00%
67.14%
88.50%
100.00%
70.00%
89.50%
100.00%
71.43%
90.00%
100.00%
71.43%
90.00%
100.00%
71.43%
90.00%
Bidirectional
100.00%
71.43%
90.00%
100.00%
71.43%
90.00%
71.43%
90.00%
100.00%
71.43%
90.00%
16
100 100%00%
72.86%
90.50%
98 08%
72.86%
90.00%
Advanced
U
16
75.00%
68.57%
81.00%
16
75.00%
68.57%
81.00%
18
73.53%
71.4%%
81.00%
17
75.36%
74.29%
82.50%
74.29%
74.29%
82.00%
75.71%
75.71%
83.00%
Unidirectional
75.00%
77.14%
83.00%
74.32%
78.57%
83.00%
22
71.43%
78.57%
81.50%
71.79%
80.00%
82.00%
16
24
70.37%
81.43%
81.50%
26
68. 29%
80.00%
80.00%
0
O
100.00%
67.14%
88.50%
100.00%
67. 64.14%
88.50%
100.00%
68.57%
89.00%
100.00%
68.57%
00%%
Bidirectional
100%00%
71.43%
90.00%
100.00%
70.00%
89.5 50%
100. 100.00%
71.43%
90.00%
100.00%
70,00%
89.50%
8
100.00%
71.3%%
90.00%
98.04%
71.43%
89.50%
16
100.00%
72.86%
90.50%
94.44%
72.86%
89.005
All
16
75.00%
68.57%
81.00%
75.00%
68.37%
81.00%
75.36%
74.29%
82.50%
74.29%
74.29%
82.00%
Unidirectional
18
74.29%
74.29%
82.00%
19
73.24%
74.29%
81.50%
20
73.33%
78.57%
82.50%
19
73.97%
77.14%
82.50%
71.43%
78.57%
81 $1.50%
72.97%
77.14%
82.00%
68.67%
81.43%
68.35%
77.14%
79.50%
failed to synthesize instance: Others
application type mismatch: Others
unknown identifier: Hallucination
invalid pattern, constructor constant marked with Others
invalid pattern variable, must be atomic: Others
unexpected end of input: Others
unexpected token: Others
invalid coercion notation, expected type is not known: Others
cannot coerce function: Others
16

---

Published as a conference paper at ICLR 2025
Table Human evaluation results. RA is the main method; RA -R is the ablation removing depen- dency retrieval; RA +R is the ablation using oracle dependencies; TP. TN, FP, FN are the number of true-positives, negatteg,,,,,, false-posiie posstiposssees and felse-ne of BEq, respectively.
BEq
Benchmark
Method
TP
TN
FP
FN
Precision
Recall
Accuracy
RA -R
22
70
0
9
100.00%
70.97%
91.09%
ProofNet
RA
22
67
100.00%
66.67%
89. 99.00%
RA +R
32
57
0
12
100.00%
72.73%
88.12%
RA -R
29
49
100.00%
100.00%
100.00%
Con-NF
RA
55
44
0
100.00%
98.21%
99.00%
RA +R
74
23
3
100.00%
96.10%
97.00%
Table 6: Human-rectified results centering RAutoformalizer ablative experiments. RA the main method; is the ablation removing dependency retrieval; +R is the ablation using oracle dependencies; @ indicates the portion of samples where predictions are equivalent to ground-truths under BEq in one attempt, defined in Eq. 7 Typecheck@) indicates the portion of samples where predictions pass typecheck in attempt, defined in Eq. Human indicates the estimated portion of samples where model predictions pass Human evaluation.
ProofNet
Con-NF
Method
Typecheck@)
BEq @1
Human 1
Typecheck @1
BEq@
Human @
RA-R
52.14%
11.50%
16.00%
8.12%
3.02%
3.02%
RA
57.22%
12.30%
18.88%
20.50%
11.45%
11.48%
RA+R
72.99%
23.26%
31.80%
60.46%
44.85%
46.55%
typeclase instance problem is stuck, it is often due to metavariables: Others
type mismatch: Others
invalid {...} notaticn, expected type is not known: Hallucination
stuck at solving universe constraint: Others
invalid binder annotation, type not class instance: Hallucination
invalid parametric local instance, parameter with type: Others
invalid constructor (...), expected type must be an inductive type Hallucination
over loaedd, errors Others
expected token: Others
ambiguous, possible interpretations Others
don't know how to synthesize placehol der: Others
invalid field, the environment does not contain: Hallucination
invalid (...} notation, expected type is not the form ...): Others
invalid dotted identifier notation, expected type is not of the form (... ...) where is constant: Others
unexpected identifier: Others
(deterministic) timeout 'whnf maximum number heartbeats (200000) has been reached (use 'set_option maxtleartbeats <num> to set the limit): Others
failed to synthesize: Others
failed to prove index is valid, possible solutions:: Others
cannot coerce to sort: Others
invalid argument name: Hallucination
don't t. know how to synthesize implicit argument: Others
invalid projecti on: Hallucination
17

---

Published as a conference paper at ICLR 2025
Table Distribution typecheck errors RAutoformalizer ablative experiments. RA is main method; RA -R is the ablation removing dependency retrieval; RA +R is the ablation using oracle dependencies; Hallu- cination denotes the number of errois caused by hallucination, and Others denotes the number of other errors. Cyan numbers highlights the percentage of errors reduced relative to RA -R.
Net
Con-NF
Method
Hallucination
Others
Hallucination 8902
Others
RA -R
434
1790
14842
RA
320 (-26.27%)
1500 (-16.20%)
5217 (-41.40%)
13386 (-9.81%)
RA +R
65 (-85.02%)
1173 (-34.47%)
1134 (-87.26%)
5882 (-60.37%)
elaboration function has not been implemented: Others
failed to infer binder type: Others
invalid ocourrence: Others
invalid universe level: Others
expected no space before: Others
tactic failed: Others
invalid constructor: Others
missing end of character literal: Others
unused universe parameter: Others
unknown tactic: Hallucination
unsolved goals: Others
(kernel) declaration has metavariabless Others
invalid use of field notation with 10 modifier: Others
invalid (...) netation, structure type expected: Others unexpocted syntax: Others
expected ';' line break: Others
invalid binder name: Hallucination
field of structure: Hallueination
many explicit universe levels: Others
type class instance expected: Others
fields missing: Hallucination
invalid use of explici: universe parameters: Others
is not a structure: Hallucination
don't know how to synthesize placeholder for argument: Others
cannot coerce: Others
unknown universe level: Others
expected structure: Others
has already been declared: Others
simp made no progress: Others
missing cases:: Others
invalid dotted identifier notation, unknown identifiers Hallucination
invalid 'import' command, it must be used in the beginning of the file: Others
(T) must have a function type, not: Others
not a structure: Others
18

---

Published as a conference paper at ICLR 2025
Table 8: Experiment results of sine-tuning--aase basing autoformalization methods reproduced on DeepSeek- -Mthh Base-7B. Cyan numbers in brackets show ablative improvements over bare autoformalizer ("RA R""): Bold numbers emphasize the highest values excluding oracle ("RA +R") results; B k indicates the portion of samples where predictions
equivalent to ground truths under BFq at least once in l attempts, defined in Eq. Typechecked @* indicates the portion of samples where predictions pass typecheck attempts, defined in Eq. D MMA, MMA (Lean), PDA, and LW represent fine-tuning on MMA, MMA's Lean subset, PDA, and Lean-workber workbook, respectively; RA is the main method; RA -R is the ablation removing dependency retrieval; RA +R is the ablation using oracle dependencies.
least
Method
Dypecheck@11
BEq@11
ProsdNet
1xpecheck88:
BEq@8
Typechedz011
Big@11
Con.F
Typecheck8
BEq
MSSA
13.785
1375
31.02%
5.0372
3.33%
7.25%
MMA
17.60%
2.41%
31 31.02%
5.611
3.71%
1,381
7.59%
4.57%
PIAA
14.71
2,14%
27 S45
5.615
4.99%
1,77%
108254
4.4%%
1W
16. 10.
8.568
51.449
10.165
4.99
1,98%
11.19%
2.0g
RA -I
$1.55
"10.06""
'6 7FF
15.2455
$.235
"T22R
12,50%
"4275"
RA
59.36% (8,025))
10.68% (0.00%)
75.44% (6.159)
17.91%
(2.67%)
17.99%
(9.376)
9.68% (6.569)
25.49%
(12.909)
15.30% (11 (1105))
RA +X
72.75%
(33379)
25300
(12396)
K3,095
(7759)
12
(EA2TS)
80.36%
(02.985)
46.02%
75,9015
(500775)
59,006
(33,065)
Examinations Linear Algebra
5%
5%
Abstract Algebra
Analysis
20%
46%
6%
Number Theory
17%
Topology
Figure 3: Disciplines distribution in the benchmark for human evaluation.
A.3 EXPERIMENT RESULTS DEEPSEEK-MATH-BBNE
also evaluate all fine-tuning-hasse methods using DeepSeek-Math Maatsek--Mt-----Be----- (Shao et al. (0224) as the base
model and the training recipe shown in Appendix A.8 The results are in Table which demonstrate consistent (and even clearer) advantage of our methods over all haselines, and the ablative improvement of Dependency Retrieval.
A.4 HUMAN EVALUATION FOR BEQ
Human Equivalence Benchmark. We use an early version RAutoformalizer with oracle dependency (RA +R) and OpenAl ol -preview to predict formal statements for all samples in ProofNet (Azerbayev et al. 2023) benchmark. RAutoformalizer uses greedy decoding, while ol-preview uses temperature decoding with default hyperparameters from OpenAl. Generated statements are then filtered by typecheck and deduplicated by string matching. Then uniformly sample 100 statement pairs from each model's generation, invite human experts from diverse backgrounds label them as "equivalent" or "inequivalent" resulting our Human Equivalence Benchmark. In total, 4 experts, one from formal verification and three from computer science participate in the labeling. They first separately evaluate the equivalence between formal statements, and discuss in round- table to reach agreement for each sample. The distribution of disciplines in this benchmark is visualized in Figure
Failure Case Analysis. Our BEq reaches 100% precision, thus there are no false positives. For false negatives, we analyze ze them in detail and find roughly error patterns:
Semantic gaps between informal mathematics and formal verification. out of 19 false negatives stem from it. Some subtle differences in informal mathematics may result in large differences be- tween formalizations. As illustrated in Figure 4 formalization P and Q are identical in semantics, but they are formalized under different bases, one by subtype and the other by set. Another example is Figure [S] where model-generated proof fails due to subtle but fatal difference in the underlying types.
Transformation function failure. 10 out of 19 false negatives stem from it. Proving unidirectional definitional implication is novel task, hence the prohibitive lack of supervised data makes it im- possible fine-tune capable model. Our implementation utilizes 5-shot prompted 20B model,
19

---

Published as a conference paper at ICLR 2025
Informal Statement Show that there are infinitely many primes congruent modulo 6.
Formalization P
theorem SP :
Infinite (p Nat.Primen // = -1 12DD 61)
==
sorry
Formalization Q
theorem sQ
Set Set.Infinite ip Nat.Prime p p
sorry
Figure 4: Failure case of BEq; small semantic gap for natural language mathematics might be huge for formal verifier
Informal Statement
ring in which 2 for every a R. Prove that is commutative.
Formalization
theorem SP [R Type*) [Ring R]
(h V R, 3 ==xx
Commiing R
sorry
Formalization Q theorem sQ IR Type u 11 [Ring R (h (x : R), x 3 x) (x R) (y :y:R) R)
x -
sorry
Failed Proof
have conm exercise 4.2.5 have b_ == h comm.mul comm x bx
cype mismatch h_xy
has type
@HMul.hMul R R. R (@instHMal Babbblaallledsssocmmmmrrig.....l y y Prop
expected to have type
@iMul. R R R (einstHMul R Natthttalllenkkeeodkknngg.a.al
x Prop
Figure 5: Failure case of BEq: imperceptible differences in type are intolerable in Lean.
which is relatively weak and fails to generate proper transformation for more complex scenarios, illustrated Figure alnd and Figure
Success Case Analysis. Due to its symbolic nature, BEq can easily find fundamental differences between formalizations that misleading for human expert. We demonstrate examples in Figure and Figure
A.5 VISUALIZATION OF BLEU DISTRIBUTION
The distribution of BLEU scores hetween formal statement pairs from the Human Equivalence Benchmark visualized in Figure T0 along with the precision, recall, and accuracy different thresholds.
20

---

Published as a conference paper at ICLR 2025
Informal Statement Prove that group of order pq. where p and & prime, is simple.
Formalization P
theorem SP (G Type*) [Group G] printype G] (p q N)
(hp : Prime p) (hq Prime q) (hG card p.q)
IsSimpleGroup G = False
sorry
Formalization Q
theorem
(p q : N)
[bp Nat.Prine p)
(hq Nat.Prime
(G Type _ [Grcup G] [Fintype s]
(hG Fintype.card
p q)
: IeSimpleGroup :=
sorry
Equivalence Proofs
sp ~B T R)
have hpp Frime p by exact Nat.prine_iff.we hp
have hqq Prime q := by exact Nat.prine_iffe fetiimme_fffmppp
exact sP hpp hqq
8Q ~B T(sp|sq,R) R)
have hpp Nat.Prime :- by exact Nat.prime_iff.mmm hp have hqg Nat.Prime
by exact Nat Nat.prime_iff.mmg hq exact sQ p q hpp hqg G hG
Figure 6: Failure case of BEq: transformation function fails to generate the transformation.
Informal Statement Assume that f: I2 satisfies (S()) s(=) =/2 for all 1, Prove that is constant.
Formalization
theorem SP (f
+ R 1f El
R
(hf V x Y,
& 1x
2)
3 A sorry
Formalization Q theorem sQ (E
(:: xxxxx , 18t xl < -t- xl 21
(x R ly RD E 1=
sorry Equivalence Proofs T(sQ/sp, R)
have he := sQ hf use 0 ext X
exact he 0
8Q ~g T(sp|sq, R)
have he sP
cases' he with c have hx : f
:- by exact congrFun he x
have by f
= by exact congrFun he y
zw [hx, hyl
Figure Failure case of BEq: transformation function fails to generate transformation.
21

---

Published as a conference paper at ICLR 2025
Informal Statement Show that (12)) is algehraic number.
Formalization
theorem sP IsAlgebraic Q (sin (pi/12)) :=
sorry
Formalization Q
theorem sQ IsAlgebraic Q (Real.sin (Real.p)) .p//12)) sorry
Figure 8: Success case of BEq: These two formalizations are not equivalent. Note that pi in Formalization is an implicit argument of an arbitrary real number, instead of n.
Informal Statement Prove that + 30x 15x + 62 120 is irreducible in Z[x].
Formalization
theorem sP Irreducible
(X"6 30+X"5 15+x"3 6+X Polynomial Z)
sorry
Formalization Q
theorem sQ
(f : Polynomial & :- X + 30xX 15xX"3 + 6+X 120)
Irreducible :=
sorry
Figure 9: Success case of BEq: These formalizations not equivalent. Note that in Formalization means f is of type Polynomial with default parameter X" 6+300xx #+300x"5-155x 3+6xx-120, instead of f-X 6+30+xx0*0 (6+30**"--15+3 5-16+=33+6+6660000 3+6+x-1200
Polynomial Z ==x
1.0
Precision, Recall, Accuracy
Probability Density
0.75
0.5
Equivalent Inequivaient
0.25
Pracision
Accuracy
BLEU
Figure 10: Distribution of BLEU in the benchmark and precision, recall, accuracy different BLEU thresholds.
A.6 CASE STUDY OF BM25 RETRIEVAL.
Formally, BM25 (Rohertson et al. 2009) can be defined as follows:
(k2 1)f(q, d)
BM25(d, Σ IDF(g,, D)
f(g. d) + (1 - b (lante (Tant J E
Lendd
1=1
N - 11 E d/d E D) + 0.5
IDF( D) log((
+1)
lig: E dld e D) + 0.5
22

---

Published as a conference paper at ICLR 2025
Query
Suppose that f holomorphic in an open Q. Prove that if Re(f) is constant, then is constant. Ground-truth Document
Function 'Set' maps given type to proposition, which means that for cach element of that type, it determines whether that element belongs to the set. A set collection of elements of some type Irrelevant Document
a function from complex manifold to complex normed space 'F' holomorphic on preconnected, compact, and open set 'U", and 'a' and "b' are points in 'U", then 'fa = f Irrelevant Document 2
If function 'f' from topological space 'X' to type 'Y' is locally constant, then for any point 'x' in 'X', there exists an open set 'U' containing 'x' such that 'f" is constant U..
Figure 11: Failure case of BM25: BM25 prefers semantic similarity to logical dependency.
where q (g:)==1 query with n tokens :1n...:9:: D {d: a document collection with N documents ... dx, kn and b are hyperparameters; IDF(g, D) the inverse document frequency of token in document D.
As Figure 11 shows, BM25 prefers "semantic similarity" "logical dependency" during retrieval. We focus on keywords, holomorphic, set, and constant the query. The query depends on defimition of "Set", but the frequencies two keywords holomorphic
constant are in the definition of "Set". Instead, the first irrelevant document shares similar frequency of set and holomorphic, while the second irrelevant one is similar in set and constant. Subsequently, both irrelevant documents have higher BM25 scores than the ground-truth.
A.7 PROMPT TEMPLATES
A.7.1 PROMPT TEMPLATE OF BEQ
Given two Lean 4 theorems, please prove 'thm_0' with 'thm_P', You only the following tactics: (ALLOWED TACTICS) 'thm_P" should used at least once the proof. DO NOT add any extra explanation. Here are some examples:
Input:
import Mathlib
open Topology Filter Real Carplex Topologicallpacee Finset open scoped BigOperators noncomputable section
theorem thm_P Anot sorry
texists (x Rat), x
theorem thm_ Rat
\neq 12 :- by
Output: exact (hot_exists.mp thm_P)
Input:
import Mathlib
open Fintype Subgroup Set Polynomial Ideal
open scoped Bi Big0perators
noncorputable section
23

---

Published as a conference paper at ICLR 2025
theorem thmnP (p q r Nat) (G : Type+) [Group G]
[Fintype G] (hpor < aand < z)
(hpqr1 p.Prime Vand q.Prime aand r.Prime) (hG card G p.ger)
Nonempty (Sylow p Vor Nonempty (8ylow Nor Nonempty (Sylow G)
sorry
theorem thi__Q (p Nat lq Nat ir Nat iG Type u_l (Group [Fintype G] (hp
Nat.Prime (hg : Nat.Prime q) (hr Nat.Prime (hpq q) thar : q x)
kintype.card G - P * r) :Nonempty (Sylow G) Nor Nonempty (Sylow q G)
for Nonempty (Sylow G) :- by
Output:
exact thm iAnd hpg her) (Ando hp (And,intro ha hr)) hG
Input::
import Mathlib
open Fintype Complex Polynomial LinearMap FiniteDimensionsl Modale Module. open scoped BigOperators
theorem thm Type*) [AddCamSrooup [Field F]
V) : (S T) .Eigenvalues = (T 8) Eiiennaluess :=
[Module Vj (S
sorry
theorem thm_Q Type v/ IV Type w| [Field K] [AddComrrouu V) [Module K V] (S
Mndule.Rnd V) (T Module.End K V) :Module. Rnd. Rigevvall, (S T)
Riggge (T 8) : by
Out:pt:
exact 8thn P K V
Input:
import Mathlib
open Function Fintype Subgroup Ideal Polynomial Submodule Zegrtd open scoped BigQperators
noncorputable section
theorem thmn
(p Nat) (hp Nat.Prime p) (h Vexists Nat, 2 rr 1)
\exists Nat), 2 (2 k) + +1: sorry
theorem thm_Q (p Nat (hp Nat.Frine p) (h Nexists (r ; Nat ) p = 2 - +1 :exxsss (k Nat ), p
:= by
Output:
exact 8thnuP p hp
24

---

Published as a conference paper at ICLR 2025
Input: import Mathlib
open Fintype Set Real Ideal Polynomial open scoped BigOperators
noncarputable section
theorem thm_P (G Type*) J [Group G]
[Fintype G] (hG2 Even (card G))
Vexists G), Vneq \and al-1 sorry
theorem thm_ (G Typo*) [Croup G] [Fintype G] (h Fintypo.card G $ 0) Vexists
Vneg 1 \and al-1 == by
Output:
have hG Even (card G) 1= by exact?
exact thm hG
According task description and examples, given the following two Lean theorens, please prove "thm__0' with Pt.
Input:
(CHMS TO EVALUMEE)
Output:
To apply this template. MALLONED.IACTIC should be replaced the list of allowed tactics and (TOMS_TOO be replaced to the two statements to evaluate.
A.7.2 PROMPT TEMPLATE OF LLM GRADER
Backtranslation Template
Given a Lean 4 theorer, please webrieflye and **oonsiselye* explain it in natural Language in one 1ine.
Bere are same examples:
Code:
theorem pottnam_194____ (E Real Vinp Real)
(hé : Continuous f \and \forall \alpha 0, Tendsto (fun n Nat \napsto é (n )alpha)) atTop ('nhds 0))
(Tendsto f atTop (Anhds 0)) sorry
Surmarization: Suppose f R R continuous and every a 0, lim f(na) 0.
Prove that lim f(x
Code:
theorem putnam 1968____968_p2
[Group G]
(hG Pinite
25

---

Published as a conference paper at ICLR 2025
Set
(hA : A.ncard caat (Rat)/2) Mfcarl g G, Nexists x Min A lexists y Ain & 9 = == by sorry
Summarization: Let G be finite group (with miltiplicative operation), and A be than half of G's elements. Prove that every
subset of that contains element of G can be expressed as the product of two elements A.
Code:
...
theorem putnam_2022_25
(p Nat]
(hp Nat.Prime p \and
(E Nat := la : Nat Vinp (ZMod p) | \forall n Nat, n \neg aand (n 2)
+ a (n 1) (.ncar))
ff lequiv 0 [MDD 5] Yor lequiv 2 [MDD 5] i= sorry
Surmarization:
p be prime number greater than 5. Let f() denote the number infinite sequences 01, 02,02,00 03.... such that an 21.2....,P }} and
0odn42 = + 0n+1 (mod p) for all n 2 1. Prove that f(p) is congruent to or 2 (mod 5).
Please +bbriefly** and *oonssiseyy** explain the following theorem in one liner Code:
(CHM_CODE)
Surmerization:
To apply this template, [THM_CORR] should he replaced to the formal statement to informalize.
Equivalence Determination Template
Please check following two math problans is same different? Please consider each statement in two problems, they are different if any statement different. Please point out any differences you found, Please reply **sane** or vxdifferents in the final sentence with bold format.
Problem 1: (THM_])
Problem em 2: (THM_2)
To apply this template, [THM_1} and (THM_1) should be replaced to the informalizations of the two formal statements evaluate. Notably, when Majority Voting is adopted, it is recommended to randomize the order of the two statements in multiple attempts.
A.7.3 PROMPT TEMPLATE OF ICL AUTOFORMALIZATION
Please translate mathenatical propositions into Lean 4 theorems. Wathhibb is the only allowed import.
DO NOT add any imports into the translation, and DO NOT try to prove the theoren, ONLY translate it.
Here same examples:
Math Proposition:
Suppose f:R - R is continuous and for every a > O, limp-400 f(na) = 0. Prove that lim f(x) 0.
..
Lean Theorem:
theorem exercise se
(f Real Vimpl ies Real)
26

---

Published as a conference paper at ICLR 2025
(hé Continuous Vand \forall \alpha Tendsto (fun Nat 'napsto Nalpha)) ) atTop ('nhnds 0))
: (Tendsto f atTop ( (Anhds 0)) :
sorry
Math Proposition:
Let G be a finite group (with miliipicative operation), anc A be subset of G that contains more than half of G' elements. Prove that every element of G can be expressed as the product of two elements of A.
Lean Theorem:
theorem exercise
[Group
(hG Finite G)
(A : Set G)
(hA A.ncard > (Ntt.card Rat)/2)
\forrll g G, lexists Vin A, texists y Vin A --xxy
sorry "..
Math Proposition:
Let p be a prime number greater than 5. Let f(p) denote the number of infinite sequences d1, 0a,... such that on E .1, 1} and 0,an+2 1 + ac+1 (mod p) for all 2 1. Prove that f(p) is congruent to or (mod 5).
...
Lean Theorem:
theorem excrcise
(p Nat)
(hp Nat.Prime p \and p >
(f : Nat :- (a Nat Virplies (?Mod p) Aforall n Nat, a n \neq 0 aand a n * (n + 2) = 1 + (n + (ncarr))))
ff lequiv 0 [MO 5] \or Nequiv 2 [M00 5]
sorry ...
Please translate the following proposition: Math Proposition:
//
(INEGRMALLSIMT)
Lean Theorem:
To apply this template, (INFORMAL.STMT) should be replaced to the informal statement to autoformalize. Equivalence Determination Template
Please check following two math problems is same different? Please consider each statement in two problems, they are different it any statement different. Pleass point out any fffrenncs you found. Please reply **sane** or +ddi fferent** in the final sentence with bold format.
Problem 1:
(THM
Problem 2: (THM
To apply this template, {CHM_1} and [THM_1 should be replaced to the informaiizations of the two formal statements evaluate. Notably, when Majority Voting is adopted, it is recommended to randomize the order of the statements in multiple attempts.
27

---

Published as a conference paper at ICLR 2025
A.8 FINE-TUNING DETAILS
Dependency Retriever. We fine-tune dependency retriever based BGE-M3 eChen (023) with FlagEm-
bedding library. Query string is identical to informalizations of theorems. The composition of document strings is as follows:
F+IF: Formal
Explanation::f.attt
F: Formal Declaration: :ec1
where decl and f_stmt represents formal declarations and pesimmalliaaiionsssseeeetttt Both clipped 1536 characters most before composition.
We follow the default hyperparameters of FlagEmbedding, which are as follows:
Learning Rate: 5 10-6
Warmup Ratio: 0.1
Weight Decay: 0.01
Precision: fp16
Train Epochs: 6
Gradient Accumulation Steps: 32
Per Device Train Batch Size: 2
Training Devices: 8
Dataloader Drop Last: True
Normalized: True
Temperature: 0.02
Query Max Length: 1024
Passage Max Length: 1024
Training Group Size: 4
Hard Negative Size: 2
Negatives Cross Device: False
Query Instruction For Retrieval: None
Inbatch Negative: False
RAutoformalizer:
RAutoformalizer and all fine-tuning experiments are fine-tuned from InterL.M2-Math
Base-7B (Ying al. 2024b) using XTuner (Contributors) 2023) and the following hyperparameters:
Max Sequence Length: 8192
Variable-length Attention: True
Batch size:
Gradient Accumulation: 4
Training Devices: 8
Train Epochs:
Optimizer: AdamW with learning rate 2 20 = (0.9, 0.999), weight decay 0, maximal gradient norm 1, warpup ratio 0.03 and float 16 mixed precision training.
Learning Rate Scheduler: Warmup using LinearL.R with start factor 10-6 then train using CosineAn- nealingLR with Mami 0.0.
A.9 OPEN-SOURCE LIBRARIES
For reproducibility, all relevant open-source projects are summarized in Table Special thanks to the authors of these excellent projects.
28

---

Published as a conference paper at ICLR 2025
Table 9: Versions open-source projects used this project.
Name
Githab Link
Version
FlagEmmeddding
https: Miggithie hib por PlasCpen /P 1 Pribediing
7608Cab8521 4455544155557727777744444466666
Xtuner
0.1.23
/ 7414505 rempproye V.n.--hii hiibe
Mathiib
at:ps. C.t.bb......................... E Sanprover Foroooo-ooooo Connin Cy.n
REPL
aEtes wwthhttthhhhh.. com/ Canprover conm Eyvty pp
Doc-Gen
ProafNet-leamd
LeanDojo
I1123:7791
Con-NF
Bctp5:/7 34 Att.io. res.prever Coarilai CV/COL-BE BC
Table 10: Experiment results augmenting in-context learning methods by dependency retrieval. Bold num- bers emphasize the highest values excluding oracle results; BEq@k indicates the portion of samples where predictions are equivalent to ground truths under BEq at least once in le attempts, defined in Eq. 7 T@k in- dicates the portion of samples where predictions pass typecheck least once in it attempts, defined in Eq. 9 ICL represents in-context learning using 3-shot demonstrations; ICL+RA represents in-context learning ing 3-shot demonstrations, augmented by dependency retriever trained Sec. 4.2 ICL+RA represents in-
context learning using 3-shot demonstrations, augmented with ground-truth dependencies; D-2.5 denotes using
Deepseck-V2.
Benchmark
ProofNet
Con-NF
LLM
Method
T@1
Beq @ 1
T@8
Beq@8
T@1
Beq@1 1
T@8
Beq@8
ICL
43.58%
7.22%
66.31%
12.83%
9.78%
1.46%
20.71%
4.16%
GPT-4o
ICL+RA
46.52%
6.95%
77.01% %
13.37% %
22.79%
6.66%
50.57%
12.59 %
ICL+RA (+R)
58.56%
17.38%
81.28%
29.14%
54.84%
38.40%
75.75%
54.11
ICL.
40.37%
9.89%
51.07%
10.96%
9.37%
2.81 %
16.23%
4.27% %
D-2.5
ICL+RA
43.32%
6.42%
58.82%
10.96%
9.37%
1.87%
15.19%
3.12%
ICL+RA (+R)
61.50%
17.91%
72.99%
20.32%
48.18%
32.36%
62.02%
41.94%
A.10 10 EXPERIMENT AUGMENTING ICL METHODS DEPENDENCY RETRIEVAL
The performance augmenting ICL (in-context learning) methods with canarddecyyrttiievllaagmmmmettt shown in Table [T]]
For GPT-4o, the results meet our expectations: RA consistently improves all metrics all benchmarks (except BEq @ 1 on ProofNet), and RA(+R) shows the potential of dependency retrieval.
However, for Deepseek- V2.5, RA doesn't work well. We hypothesize this might be because the instruction- noise in retrieved dependencies
following long-context capabilities Deepseek- V2.5 are limited, degrades autoformalization. But RA (+R) shows significantly better performance than expected.
29


Reviews:
[{"review_id": "Rymn30qgMC", "reviewer": "Reviewer_pPF9", "summary": "The work attempts to address two important gaps in statement autoformalization currently: 1) absence of an automated and accurate evaluation metric for autoformalization; 2) models' agnosia of contextural information during autoformalization, inducing severe hallucination of formal definitions and theorems. \n\n- To address 1), the paper proposes BEq (Bidirectional Extended Definitional Equivalence), a neural-symbolic metric that determines the equivalence between an autoformalized statement and the ground-truth formal statement by taking advantage of Lean 4 features.\n\n- For 2), the paper introduces RAutoformalizer (Retrieval-augmented Autoformalizer) which (a) uses dependency retrieval to find relevant formal objects from libraries, then (b) proposes topological informalization to structure training data. RAutoformalizer significantly improves performance on ProofNet and the out-of-distribution benchmark Con-NF.\n\n- The authors also synthesized Con-NF, a new benchmark with 961 informal-formal statement pairs, by informalizing from a frontier mathematical library that has not been adopted before, to evaluate OOD generalization capabilities in autoformalization.", "questions": "1. I am not clear what the notion ‘transformation primitives’ in line 215/ line 221 means: in some in autoformalization research, 'transformation' is sometimes an equivalent of 'autoformzalization', but here apparently it is not the case. Does 'all transformation primitives' here refer to all functions defined in Definitional Equality in Lean 4 (e.g., α-conversion, η-expansion etc.)? The confusion about transformation T continued regarding how LLMs are prompted to implement the transformation function T. The paper would be clearer and more reproducible if the authors provide more accurate declarations of each symbol and notion, as well as the exact prompt template they use for few-shot prompting. Another example of ambiguous symbol usage is ‘s\\_P’ versus ‘theorem P’. \n \n2. Following the question about T, I wonder how the authors derive the Unidirectional Definitional Implication (2) from equation (1) by explaining how transformations can occur between a formal statement and a proof goal in Lean 4 around line 225-236. Is the T in (2) an approximation of T in (1), based on the fact we can switch between proof goals and formal statements in Lean 4?  \n\n3. A recent work called FormalAlign introduces a metric for autoformalization evaluation to address the first limitation in this paper by training a *formal-informal statement alignment* evaluator, rather than focusing on *formal-formal equivalence*: [https://arxiv.org/pdf/2410.10135v1](https://arxiv.org/pdf/2410.10135v1). The authors are suggested to discuss this in the related works section and ideally include a comparison with this metric as a baseline.  \n\n4. Typo and grammatical issue: \n   - Line 097 (‘A baseline is build’); line 199 (‘with how humans instincts.’); \n   - In Table 3, in the Con-NF / Typecheck@1↑ column, LW achieves the highest (28.10%) but RA is marked bold (20.50%). If there is indeed a mistake here, the body text should also be adjusted.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. **Originality**:\n\n- BEq is one of the pioneer metrics for evaluating autoformalization different from existing ones. It is built based on human intuitions about statement equivalence and implemented using Lean 4 features, thus having the potential advantage of being more interpretable and predictable (although not elaborated in the paper), if provided with high-quality ground-truth formal statements.\n- The paper also formulates the problem of autoformalization as bidirectional definitional equivalence. This approach differs from previous formulation efforts (e.g., https://arxiv.org/pdf/2410.04194).\n- RAutoformalizer introduces a typological informalization and dependency retrieval pipeline to reduce context agnosia. The Con-NF benchmark also emphasizes the importance of OOD evaluation for future researchers.\n\n2. **Quality**: \n  - Uses both in-distribution (ProofNet) and out-of-distribution (Con-NF) testing; \n  - Includes ablation studies to validate technical choices; \n  - Provides training details for RAutoformalizer.\n\n3. **Clarity**: The motivations are coherent. The writing and illustrations are mostly clear. However, improvements can be made regarding the formal problem setup when introducing BEq as well as the implementation procedures.\n\n4. **Significance**: The paper aims to address two important problems in autoformalization that are both relevant and challenging. The significance is very valid, although the applicability of BEq is compromised as mentioned later.", "weaknesses": "BEq:\n\n1. **Limited applicability**: \n   - A limitation of BEq for evaluating autoformalization is it focuses on the matching between ground-truth formal statement and the autoformalized statement, hence it always requires ground-truth formal statements. In other words, BEq disregards the informal statement input and relies heavily on the *quality* of the ground-truth formal statement that the human experts provided. It also relies on the tactic features of Lean 4. This greatly hurts the applicability of the metric to many larger-scale informal-formal language benchmarks without expert-annotated ground-truths or written in other formal languages. It also makes it questionable whether BEq can even be directly used on synthesized benchmarks like Con-NF where the informal-formal statement pairs are not checked by human experts.\n   - The weakness should be also be mentioned in the theoretical constraint of the problem formulation in section 3. Fundamentally, BEq distills the problem of evaluating autoformalizatiom into the problem of evaluating 'Definitional Equality' between two formal statements in Lean 4. However, this overlooks two constraining assumptions that should be ensured about the autoformalization benchmarks: (1) the ground-truth formal statements are aligned with the informal statements; (2) the ground-truth formal statements are successfully type-checked. \n\n2. **Limited robustness**: There are few experiment details about how human evaluation are conducted such as the expert numbers, competence levels, annotation protocols, annotation processes, agreement rates, etc. Considering that evaluating formal statements is a challenging task that frequently induces high disagreement rates among experts in practice, the rigor of BEq and Table 1 results is in question.  \n3. **Unclear experimental setting missing material support**: The paper also did not provide concrete experimental details about Beq such as (a) the few-shot prompt for LLMs on transformation T (line 261), (b) the few-shot prompt for LLMs graders on formal statement matching (line 303). The authors cited Ying et al. (2024a) and stated they used a stronger few-shot setting of their prompt, but I did not find the corresponding prompt template in the reference. It is difficult to review the exact implementation process and experimental details, hurting the robustness and reproducibility of the work.  \n4. **Writing clarity issue**: The formal setup of BEq is unclear. Please check point 1 and 2 in Questions for details.\n\nRAutoformalizer:\n\n5. **Benchmark construction rigor**: The OOD benchmark Con-NF is constructed by prompting LLMs to typologically informalize the formal objects in a new library. There is little quality control as to the informalization quality. This is especially concerning since we are synthesizing data from a new library that LLMs are probably never exposed to. Two issues may arise from this that have been noted in previous research: \n   - There could be a great discrepancy between the informal statements in Con-NF that serve as autoformalization input and the groundtruth formal statements from the first place.\n   - In LLM informalization via prompting, a common issue is that the informalized output often still includes the names of the formal tactics or variables verbatim, because there does not exist a corresponding concept in the natural language that can be used to easily translate the complex functions or relationships in formal language. In more extreme cases, some theorems are not even translatable formally or informally. The paper did not report any filtering procedures or quality checks regarding the potential benchmark issues above.\n\n6. **No human evaluation in comparing RAutoformalizer**: Table 3 compares the autoformalization performance of RA with other baselines on ProofNet and OOD Con-NF using two automated metrics: Typecheck and BEq. Because the robustness of BEq itself is limited as discussed above, the significance of the table results is compromised unless human evaluations are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "18tjQdWRgJ", "reviewer": "Reviewer_fV7x", "summary": "The paper addresses a few key challenges in the problem of LLM-based AutoFormalization. Firstly, they introduce BEq - a new metric to determine equivalence of formal statements -allowing for a better quantification of success for the task. Secondly, they propose RAutoformalizer a new autoformalization technique that retrieves dependent formal objects using informal statements. Lastly, to quantify out of distribution generalization, they propose a new benchmark for the task - Con-NF. They empirically demonstrate that BEq provides expert level performance on determining semantic equivalence of formal statements and then report significantly better results using RAutoformalizer compared to existing SoTA approaches.", "questions": "Please refer to the Weakness section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well-written, easy to follow and well-organized.\n2. The paper presents various key contributions to the Autoformalizer literature in the form of BEq and RAutoformalizer - which can have a great impact research in this area significantly. \n3. The BEq metric solves a key challenge in determining equivalence and is a promising alternative to contemporary techniques. I forsee more future work in the literature using this as an importatnt metric of success. \n4. The RAutoformalizer technique outperforms current SoTA Autoformalization techniques significantly.", "weaknesses": "While I don't see any major weakness in the paper,there are a few minor concerns that maybe worth looking at: \n1. BEq as a metric is quite promising, it is not very clear how does it fundamentally align with human judgements. It may be interesting to go a bit deeper into this in the paper. \n2. It is not very clear what kinds of benefits the retrieval component in RAutoformalizer is bringing and what are existing gaps from the ground truth retrieval setup. \n3. The ICL baselines are a bit too week. It would have been interesting to see how frontier models like DeepSeek and GPT-4o perform with the retrievals provided to them in-context as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3p1RMwIY3Y", "reviewer": "Reviewer_YeoP", "summary": "This paper aims to improve existing statement autoformalization paradigms by two key limitations: 1. absence of faithful and universal automated evaluation for autoformalization results; 2 agnosia of contextural information. To this end, the paper trains a retriever to retrieve dependent information and a autoformalizer to attempt the proof.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe design of the training protocol seems reasonable, like the formal data in existing library is parsed into informal sentences and stored in topological order. The retrieval first and generation next works well for autoformalization.\n2.\tThe experiment result shows that the newly proposed BEq method surpasses baselines by a landslide.\n3.\tA novel benchmark, Con-NF is built from frontier mathematical researches.", "weaknesses": "Some certain details can be included to make the paper more self-contained. Like the reason why the dense embedding is sufficient for the dependency retrieval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Td2YwHEVD", "reviewer": "Reviewer_9PuK", "summary": "The authors present three contributions to the field of autoformalization. First, a new evaluation metric (BEq) which better aligns with human evaluation compared to classic Machine Translation metrics. This is done by extending the concept of definitional equivalence, which can be overly strict in certain cases and not allow enough flexibility to accept all possible correct answers. Second, the authors introduce a retrieval-augmented autoformalization method which retrieves necessary dependencies for a theorem before formalization. Finally, the authors introduce a new task (dependency retrieval) and associated benchmark (Con-NF) to help measure performance for research-level autoformalization.", "questions": "Have you tested different model sizes for the BEq metric? If so, what performance drop-off did you see with smaller models? An analysis of different failure cases for these models would also be very insightful, and could offer avenues for improvement.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors work on an important problem in autoformalization, namely the lack of flexible automatic metrics. Their approach is well-explained and well-motivated, and I believe it can be of significant use to the community. BEq significantly improves over other metrics in terms of human-alignment. I believe this is the strongest contribution of the paper. \n\nSimilarly, the dependency retrieval task is a fundamental task in autoformalization, since otherwise we expect LLMs to be fluent in all necessary context, which is impossible for research-level mathematics which are introduced after the LLM was trained. Thus, the introduction of the task together with a benchmark and baseline approaches is a strong contribution.\n\nOverall I think this paper is a solid contribution to the community and is well-supported by the experiments.", "weaknesses": "While the BEq metric improves on previous metrics, one of its shortcomings is that it requires gold label formal statements (compared to, for example, type checking, which only requires the input data). Both training data and benchmarks are lacking in this area, making BEq less impactful overall. \n\nAdditionally, BEq requires a 20b parameter model to execute, severely restricting access to the metric for researchers without the necessary compute. A significant improvement would be reducing this model size, even to a 7b size which would make inference possible on much smaller devices and with improved efficiency. I didn't notice any performance comparison between various model sizes, which could improve the stance of this model choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C0evzzDLfg", "reviewer": "Reviewer_6DKY", "summary": "This paper improves the statement autoformalization by identifying and solving two key problems. First, they proposed BEq, a better LLM-based way to check if autoformalizations are correct by combining symbolic rules, and the human evaluation proves it's promising. Second, they created RAutoformalizer, a RAG approach, which looks up relevant dependent objects for auto-formalization. Also, authors annotated an OOD benchmark for further evaluation. The experiments show their methods are better than baselines.", "questions": "## Introduction:\n1. Why is dependency retrieval essential in the current context, and how does it address existing limitations?\n2. What evidence supports the claims about β-reduction and type checking limitations? Can authors provide citations or examples?\n3. How does BEq construction overcome the \"non-determinant and highly prompt-dependent\" limitations of LLM graders when it also uses LLMs and few-shot prompting?\n\n## BEq Evaluator:\n1. How does BEq's few-shot prompting actually reduce human design efforts compared to previous approaches?\n2. Why choose InternLM2-Math-Plus-20B when: DeepSeek V2.5 shows better performance in majority voting? \n3. What makes its math pretraining superior? Why not implement BEq with other models? \n4. Can BEq work effectively across different math LLMs?\n\n## Human Equivalence Benchmark:\n1. How were ground truth pairs created and validated in details?\n2. What are the expert qualifications, payment structures, and annotation guidelines?\n3. How was sampling bias controlled in terms of the following aspects:\n   - LLM-generated examples?\n   - non-uniform distribution in Figure 3?\n   - O1 bias control mechanisms to make sure the quality of evaluation?\n\n## Method:\n1. How effectively do RAG methods specifically reduce hallucinations in depth?\n2. Can authors quantify the improvement in Table 3 attributed to hallucination reduction for \"non-existing dependencies\" especially?\n\n## Con-NF Questions:\n1. What are the construction details and distributions of Con-NF?\n2. How do ProofNet and Con-NF differ in terms of discipline, problem distributions and theorem or dependencies?\n3. Why create new datasets instead of using existing ones like [1]? What are exact features considered as OOD comparing to exisitng benchmarks so that authors have to spend efforts in annotating a new benchmark?\n\n## Experimental Questions:\n1. How would results differ with multiple LLM backbones?\n2. Why weren't standard reasoning baselines (COT, TOT, Self-Consistency) included? Comparing with such basic reasoning mehtods is crucial.\n3. How does this work differ from ReProver [1]?\n4. Does the framework only work effectively with InterLLM2-math? More implementations of different families of LLMs with BEqs are required.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper propose a new LLM-based evaluator BEq for statement auto-formalization, which make it more flexible as human evalution.\n2) Authors designed a RAG-based auto-formalization framework by Dependency Retrieval\n3) Authors annotate a new benchmark to evaluate OOD performance of their method.\n4) Experiments show their fine-tuned retriever and whole RAG system more effective than baselines.", "weaknesses": "In general, this is a well-structured and written papers. But I still have some concerns about each components of contributions.\n\n### Introduction Narrative:\n1) In the Line 90-99, it would be more clear if authors can first state why they propose **dependency retrieval**. Why this new task is essential in the context of current methods.\n2) In Line 65-75, it would be more helpful for authors to provide more contexts like citations to introduce statements, i.e., \n    - The discussion of β-reduction and type checking limitations (Lines 65-75) needs citations or examples demonstrating these issues\n    - The statement about type checking being \"too weak\" requires empirical evidence or references to prior work.\n3) Authors state some disadvantages of \"LLM graders\". Could authors provide more citations stating this? \n    - The paper critiques LLM graders as \"non-determinant and highly dependent on prompts\" but doesn't sufficiently distinguish how BEq (which also uses LLMs) overcomes these limitations.\n    - The few-shot prompting used in BEq seems to face similar prompt-design challenges that the authors criticize.\n    - While Table 1 shows improved performance, more analysis is needed to demonstrate that BEq fundamentally addresses the cited limitations of LLM graders raised in the introduction.\n\n### BEq Evaluator:\n1) How few-shot prompting design in BEq pipeline can release human designed efforts of previous works as mentioned in the line 66-68 (Introduction)?\n2) It seems that the BEq is implemented by a specific math LLM, InternLM2-Math-Plus-20B. Choice of InternLM2-Math-Plus-20B needs justification:\n   - Is its math pretraining superior to alternatives?\n   - Table 1 shows DeepSeek V2.5 outperforms InternLM2-Math-Plus-20B in majority voting\n   - Why not implement BEq with DeepSeek V2.5? Is model size a constraint?\nAuthors need to demonstrate BEq pipeline works across multiple math LLMs to show BEq doesn't only work in specific models, which supports claims about not requiring extensive efforts in specific prompt designs (lines 66-68)\n3) In the **Human Equivalence Benchmark**, There are some details requiring clarification to convince readers for fairness of BEq:\n\n   - The mechanism for creating \"ground truths\": whether through expert annotation, rule-based generation, or other techniques - remains unclear. If experts are involved, the potential overlap between pair creators and equivalence evaluators raises concerns about evaluation independence. The justification for using only 200 cases appears insufficient given Con-NF's larger scale of 900+ cases. Moreover, the validation of ground truth existence in ProofNet needs more rigorous documentation.\n   - This part suffers from insufficient documentation of expert qualifications and procedures. The paper fails to specify whether experts were students, authors, or third-party experts, and their required expertise level. Critical operational details are missing, including payment structures, annotation guidelines, and the workflow for equivalence labels. The process for handling cases with multiple valid ground-truths remains ambiguous, as does the distinction between \"expert label equivalence\" and \"ground-truths.\"\n   - The benchmark shows concerning signs of potential sampling bias. The non-uniform sampling evident in Figure 3 requires better explanation, and the paper doesn't adequately address how bias was controlled in LLM-generated examples. The O1 bias control mechanism and prompting strategies need clearer documentation. Additionally, the absence of label distribution information makes it difficult to evaluate the fairness of BEq.\n\nThese limitations significantly impact the trust of implementing BEq based on specific LLM for such task.\n\n4) Potential biased benchmarking: I think one of the biggest concern along the entire paper is that it seems the specific LLM InterLM-math conducts the whole evaluation, method backbones. The family of Large Language Models (LLMs) trained on similar datasets inevitably shares inherent biases, which affects evaluation consistency. Did BEq, driven by InterLM-math, gain an unfair advantage in Authoformalization performance if it leverages the same InterLM-math framework? I think a more direct solution to resolve this concern is to show results that BEq implemented by different math LLMs.\n\n### Method:\n\nWhile Section 4 identifies \"nonexistent definitions in library\" as a main source of hallucinations and proposes RAG-based solutions, it requires empirical validation of improvement. Table 3 shows overall performance gains but doesn't isolate how much comes from reducing such hallucinations. The paper needs quantitative analysis or visualization demonstrating the specific impact of RAG methods on how effectiveness of hallucination reduction.\n\n### Benchmark:\n1) It's nice to see that the authors have annotated a new benchmark, Con-NF, to explore the effectiveness of the OOD capabilities of the developed methods. However, there are no details about benchmark construction or distributions. For example, how was the annotation performed, and what are the topic or problem distributions of the new datasets? Since the authors mention this is an OOD dataset, it would be better to compare the distributions between ProofNet and Con-NF in terms of objectives and disciplines to illustrate how they differ as OOD datasets. I think it’s insufficient to simply mention that one is for under-graduate level and the other for research level, as there is a possibility that they share similar foundational theorems or objects and dependencies or overlapped.\n\n2) Why did the authors choose to create new datasets from scratch instead of utilizing existing ones, such as [1]? It would be more effective for the authors to first specify the features they wish to capture that are not present in existing datasets, thus justifying the need for this curation. If there are existing datasets that already contain the desired OOD features, the reason for annotating a new dataset is less convincing, especially given the lack of detailed descriptions regarding its construction. \n\n### Experiments:\nAs mentioned, just one backbone math LLM InterLM2-math is not enough. Given a global view of this paper, the metrics are designed specifically by authors and InterLM2-math, the benchmark is annotated by authors, and models in methods are chosen and fine-tuned by the same specific models, the fairness is quite questionable. More datasets and LLM backbones are required in evaluation.\n\nAlso more baseline methods and implementations are required. For example:\n- Basic reasoning baselines like COT, TOT, Self-Consistency should be compared.\n-  [1] implemented ReProver, a RAG method for automated theorem proving in Lean. Differences with workflows in this paper and [1] should be explained and compared. The novelty of this paper appears limited compared to [1] if the performance comparison and analysis are not clarified.\n- I don't think it's fair to conduct all fine-tuned methods on one specific model InterLLM2-math, instead, I concern that whether such framework only works for such LLM especially the metric of BEqs is implemented by the same family of LLM (InterLLM also...).\n\n[1] LeanDojo: LeanDojo: Theorem Proving with Retrieval-Augmented Language Models, NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)