Multi-language Diversity Benefits Autoformalization
Authors: Albert Q. Jiang
Wenda Li
Mateja Jamnik
Venue: NeurIPS 2024 poster (2024)
Keywords: AI for math
Autoformalization
Large language models
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Autoformalization is the task of translating natural language materials into machine-verifiable formalisations. Progress in autoformalization research is hindered by the lack of a sizeable dataset consisting of informal-formal pairs expressing the same essence. Existing methods tend to circumvent this challenge by manually curating small corpora or using few-shot learning with large language models. But these methods suffer from data scarcity and formal language acquisition difficulty. In this work, we create mma, a large, flexible, multi-language, and multi-domain dataset of informal-formal pairs, by using a language model to translate in the reverse direction, that is, from formal mathematical statements into corresponding informal ones. Experiments show that language models fine-tuned on mma can produce up to $29-31$\% of statements acceptable with minimal corrections on the miniF2F and ProofNet benchmarks, up from $0$\% with the base model. We demonstrate that fine-tuning on multi-language formal data results in more capable autoformalization models even on single-language tasks.

Multi-language Diversity Benefits Autoformalization
Albert Q. Jiang University of Cambridge qj2130cam. ac ac.uk
Wenda Li University Edinburgh wenda. li@ed.@@dd... ac uk
Mateja Jamnik University of Cambridge
mateja. jamikkel.. cam ac. uk
Abstract
Autoformalization is the task of translating natural language materials into machine- verifiable formalisations. Progress in autoformalization research is hindered by the lack of a sizeable dataset consisting of informal-fonm pairs expressing the same essence. Existing methods tend circumvent this challenge by manually curating small corpora or using few-shot learning with large language models. But these methods suffer from data scarcity and formal language acquisition difficulty. In this work, we create MMA, large, flexible, multi-language, and multi-domain dataset of iiformal-formall pairs, by using language model translate in the reverse direction, that is, from formal mathematical statements into corresponding informal ones. Experiments show that language models fine-tuned on MMA can produce up to 29 31% of statements acceptable with minimal corrections on the miniF2F and ProofNet benchmarks, up from 0% with the base model. We demonstrate that fine-tuning on multi-language formal data results in more capable autoformalization models even on language tasks.
Introduction
Formal mathematics refers to mathematical content that is represented in formal language that be mechanically checked by a computer. Practitioners express mathematics in formal languages integrated into proof assistants like HOL Light (Harrison 1996), Isabelle [Paulson] 1994]. Coq [Barras et al. 1999). and Lean (de Moura et al. 2015). Antoformalization is the task of translating natural language materials into verifiable formalisation An ideal autoformalization engine can reduce the excessive cost for modern mathematical results to be verified [Ball] 2012, Scholze and Stix 2018).
opens up the vast amount of mathematics expressed in natural language to automated reasoning research fields that rely on formal languages, like automated theorem proving [Wu et al., 2022).
The hope of automatically translating informal mathematics into formally verifiable content is as old as formal mathematics (Whitehead and Russell] throughs in neural networks and Neural Machine Translation (NMT) enabled autoformalization be learned [Wang et al. 2020 Wu et al., 2022 Jiang et al. 20236]. NMT methods typically require large parallel dataset, that is, a dataset consisting of pairs sequences expressing the same meaning in both the source and the target language. The most challenging part of autoformalization research constructing such a parallel dataset in natural and formal language, satisfying two conditions simultaneously: (1) the natural language component is close to how mathematics is actually written; and (2) the number of datapoints large enough for the data-hungry machine learning methods. This is hard, because manually translating informal mathematical content into a formal language is only doable by highly trained experts in both mathematics and computer science, hence costly.
Only very recently, the break-
In this work, we address the lack of a parallel dataset by leveraging a state-of-the-art Large Language Model (LLM), GPT-4 [OpenAl 2023 :: we used translate the two largest formal corpora, Archive of Formal Proofs in the language of Isabelle, and mathlib4 in the language of Lean4, into natural language. This process was enabled by the key observations that informalisation is much easier than formalisation, and a powerful LLM can produce diverse natural language outputs. As a result, we
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Example parallel pairs from MMA.
Isabelle statement
GPT-4 informalisation
lemma eint__m _att__inus__l_ assumes " :eint) c"
The lemma named "cint "eint_minus__ _emuus_eeaaaa assumes that an extended integer "b" is less than another extended integer "c".. It then shows that the result of "c" subtracted by "b" is greater than zero
shows "c 0"
lemma closed_ superdiagonal:
The sct of all pairs of elements (x, y) such that x greater than equal is closed set in the context of linearly ordered topology.
"closed {(x,y) y. 2 (y:: (laio[(drrerrrtpolllo
Lean4 statement
GPT-4 informalisation
theorem norm_eq_one _tw_e______pppppeee ee__ne CC (m: N) (h: 1) (hn 0):
For complex number and natural number if to the power of equals and is equal to 0, then the norm of is equal to
1
theorem mul did_vvdmlll follifffleff {a be N) (ha: 0 :):aa a+c bbb
For any three natural numbers b, and c, where a is greater than O, times b divides times if and only if b divides
created a parallel dataset of 332K informal-formal pairs, which we refer to the MMA (Multi-language Mathematical Autoformalization) dataset. To the best of our knowledge, this is the first dataset of natural-formal language aligned data with more than one formal language. The only similar work was that of Azerbayev et al. [2023], which has only one formal language (Lean3) and is smaller than our dataset. Four examples of MMA are shown in Table
We fine-tuned two open-source LLMs, LLaMA-33B (Tourron al. 2023 and Mistral 7B (Jang et al. 2023a], on MMA to generate corresponding formal expressions given the informal ones. The trained model was then evaluated on two autoformalization benchmarks, miniF2F and ProofNet. Manual inspection of 50 problems for each model from each benchmark showed that after fine-tuning, the models could produce or minimal correction, whereas the raw model produced 0%. We also fine-tuned two identical models on the Isabelle and the Lean4 components of MMA separately for the same number of steps. Their autoformalization performances are significantly weaker than the model trained on multi-language data, demonstrating that parallel data containing multiple formal languages is crucial for autoformalization training.
31% % formal statements on the benchmarks that require no
Contributions:
We informalise all formal statements from the Archive of Formal Proofs and mathlib4, creating MMA, a dataset of informal-formal pairs. This is the first natural-form language aligned dataset containing multiple formal languages.
We train the first language models that can autoformalize to multiple languages in the zero-shot setting, and manually evaluate them on two autoformalization benchmarks.
We verify that: (1) language models trained on MMA acquire strong autoformalization abilities; and (2) language models trained on MMA have greater autoformalization performance than those trained on sangle-language partitions of it with the same computational budget.
We release the fine-tuned models for inference. We also release the MMA dataset for people train their autoformalization models on, and to enrich MMA with more domains and languages.
Improving autoformalization ability of models has the potential of translating copious digital repositories of informal human knowledge into formal languages of reasoning tools, and thus presents opportunity to formally verify human informal arguments and solutions. High quality datasets such as MMA and autoformalization models like ours pave the way towards this goa
'The MMA dataset and the fine-tuned models available from the official repository: MMA

---

2 Related Work
Autoformalization Datasets. Wang al. 2018 2020 manually aligned a small parallel dataset and generated a larger parallel dataset with a from Mizar to I Manual alignment is almost as expensive as formalising mathematics anew. Moreover, unlike generative neural informalisation tools (e.g., GPT4), symbolic informalisation tools such as Naproche [Cramer et al., 2009) result in natural language content that lacks the inherent diversity and flexibility in expression: they are rigid and not natural-language-lkks Finally, symbolic informalisation tools are hard to design and implement. They also differ a lot for different formal languages, hence the approach is not scalable for multiple formal languages.
informalisation tool [Bancerek] 2006]
Wu et al. [022] sought to eliminate altogether the need for a parallel dataset by leveraging the in- context learning ability of LLMs: they provided a couple of parallel examples, and asked the LLMs to find a formal counterpart for the informal problem (limited to high-school algebra or number theory). This approach is very effective when the test domain is limited. But when there are many test domains, finding the correct parallel examples becomes difficult: the LLM invents syntactically incorrect segments when it does not know the formal syntax for certain concepts [Wu et al.. [022] Case Study
3]. Liu et al. [2233 and Huang al. [2024) both utilised autoformalization to create aligned informal- formal pairs of data that are verified either manually or mechanically (for proofs), but did not perform
large-scale synthesis of corresponding informal-forma. theorem statements, et et al. (2024) provides a more detailed survey on autoformalization datasets. In summary, there is no existing method, like the one we propose here, that is scalable both in terms of formal languages and mathematical domains.
tack-traaslation. In natural language machine translation literature, the quality of translation heavily depends on the quality of the parallel data between two languages. However, for all but a few language pairs (e.g., en-fr), such parallel data is rare and hard to curate [Guzmán et al. 2019]. Back-translation one of the most effective methods to improve translation quality (Sennrich et al.) 2016, Artetxe et al., 2018) in this setting, which is similar to ours. Back- translation uses an existing target-to-souree model to turn ground-truth target sequences into noisy source sequences. Then, it bootstraps a source- tource-to-target model to reconstruct the ground-truth target from the noisy source.
Usually, the back-translation process is practised in both directions of translation, that is, from source to target and from target to source, and is iterated until convergence. When back-translatio practised in one direction only (because the model from target to source is called through an API and not trainable, for example), this process is referred to as "distilled back-translation". Azerbayey AAzeraiinnnnneeaaaaaaaaa al. 2023) used OpenAl's 's Codex [Chen et al., 2021] model to perform distilled back translation to improve their own model's 's autoformalization capabilities. MMA differs from their dataset mainly in that MMA contains data from multiple formal languages and has four times as many datapoints.
Language Models for Executable Programs and Reasoning. Since OpenAl's Codex [Chen et al. 202T). multiple L.MSs have been trained for code completion and infilling that stem from natural language Yuu et al.] 2018 AAsstin et al. [021]] Fried et al.) 2023). Related is also the research on
natural language mathematical and logical reasoning [Cobbe et al. [202] Lewkowyez al., 2022 Shi et al. 2022] that demonstrates that LLMs can comprehend mathematics and produce reasoning chains in natural language degree. Interestingly, distillation from larger, more capable models can effectively boost the reasoning ability of smaller models [Fu et al. 2023). However, none of these works trained language models for the task of autoformalization, which is the gap that our work fills.
3 Dataset
As established above, there is no existing parallel corpus that satisfies the following crucial criteria for autoformalization model training:
1. The informal data is diverse and flexible, similar to natural mathematical communication.
2. The size suitable for neural model training (2 100K datapoints).
Informalisation. In this work, we use a powerful neural model (GPT4) to generate informal data from existing formal libraries (informalisation) to create a high-quality parallel corpus. We argue, both analytically and empirically, that informalisation is an easier task than formalisation. Hence, our approach of leveraging the power and flexibility of language models for informalisation indeed produces parallel corpus that satisfies both of the criteria above.
3

---

Formal languages have two vital characteristics that distinguish them from natural languages: (1) pre- cision and (2) syntactic rigidity. By precision we mean that every piece of information must be explicitly and precisely expressed and formalised; whereas in natural language, pieces of information are often left implicit or ambiguous. For example, one may write in natural language "Two roots of the equation x.² 3x 2 = 0, I1 and 22,, sum up to 3. meaning the two distinct roots have a sum of 3. Expressed formally, one must also write 21 o I2 to make the statement prov- able. Hence, the information in the formal statement is always sufficient for the informal statement to be inferred, while the reverse is not always true. By syntactic rigidity of formal languages we mean that formal grammars are usually much stricter than natural grammars, permitting less choice and diversity when expressing the essence of a piece of information.
Wu et al. 2022 found that 76% of 38 high-school mathematical problems informalised by OpenAl's Codex model were "more-or-less correct". Azerbayev et al. [023] did more comprehensive study on 371 university-level problems and discovered that the same model has 62.3% informalisation aceuracy, while its formalisation accuracy is 13.4%. Empirically, informalisation has a much higher chance of being completely correct than formalisation.
Curation Process. Lean4 and Isabelle are two of the most popular proof systems for formalis- ing mathematics, with by far the largest formal proof repositories: Isabelle's Archive of Formal Proofs (AFP) and Lean4's mathlib4. They total over million lines of code as of May 2024. this paper. we consider the languages of these two systems due to their sizes and their popularity within the mathematical community, although the curation process can be easily extended other proof languages as well. In neural translation systems, similar languages tend to have similar performances as source or target languages [Lample and Conneau 2019 Roziere et al.) 2020]. Given this fact and cost constraints (see Section 7 for the curation cost), we only use the languages of Lean4 and Isabelle as target languages in this paper, and expect conclusions reached with them to generalise to similar proof languages. Lean4 and Isabelle cover a wide range of topics, from advanced mathematics to software, hardware, and cryptography verification. We use Portal to Isabelle (Jiang et al. 202T) to extract 244K theorem statements, and the LeanDojo (Yang et al. 2023 library to extract 8SK theorem statements. Isabelle AFP articles are under either a BSD-style license (a modified 3-clause BSD license) the GNU LGPL license. Mathlib4 is under Apache 2.0 license. The derived informal statements fall under licenses identical to their formal counterparts.
We choose the most generally performant language model available to us, GPT-4 [OpenA| 2023
to informalise the statements, since its ability with code and natural language is superior to that of Codex [Chen et al. 2021]. which was used by previous works on autoformalization with LLMs (Wu Azerbayev et al. 2023]. Existing works on informalisation [Wu et al. 2022 Azerbayev
ct al.
2022
et al.
2023 typically use few-shot prompting to generate good informal statements. Our informali- sation targets all available formalised content, going beyond high-school and undergraduate-lever mathematical exercises. But targeting such wide range of domains means that acquiring high-quality parallel pairs for every datapoint is challenging and expensive. Hence, instead of manually curating aligned pairs for every mathematical domain, we used an instruction prompting approach [Ouyang et al. 2022]. adopting the instruction prompt below for informalisations, with the text in curly brackets replaced by the individual datapoint content:
Statement in natural language:
Laagrra__annggggssttttteeeee Translate the statement in natural language to [Isabelle[Lean):
For all informalisations, we generated a maximum of 512 tokens from GPT-4 with greedy sam- pling (i.c., temperature = 0.0 in the OpenAl API). The responses received from this informalisation process often begin with "The lemma states that", which is mechanical and does not impact the meaning of the sentence. We remove such phrases and capitalise the remaining sentence.
Statistics. In Table 2 (top) we give the relevant statistics our MMA dataset, including the number of datapoints for each library and the statement lengths in characters for each language.
Analysis. Since formal statements are precise and rooted in exact underlying definitions and complex contexts, the LLM informalisation process may sometimes fail to capture this precision. It might overlook or loosen crucial elements of the formal information, or introduce incorrect details (hal- lucination): this is limitation of our work. To calibrate the extent of this limitation and further

---

Table (top) Statistics of MMA. (bottom) Categorisation of errois in 200 MMA informalisations.
AFP
mathlib4
Datapoints
214238
88536
Length (chars)
Informal
Isabelle
Informal
Lean4
Mean
340.0
166.0
288.5
107.8
Median
291
125
268
93
Min
95
7
98
21
Max
1546
24331
1258
989
Error type
Isabelle
Lean4
None Hallucination
81
67
2
6
Misunderstanding concept
11
18
Incorrect assumption
2
9
Incorrect conclusion
6
Incorrect type
4
8
characterise the dataset, we conducted a qualitative study on 200 statement pairs from the MMA dataset, that we detail below.
We randomly selected 100 Isabelle and 100 Lean4 datapoints from MMA, and manually examined each pair of informal-fone formal statements. We rated each pair along the following axes:
Correctness (whether the informalisation is completely correct)
Hallucination (whether the informalisation contains content not in the formal statement)
Misunderstanding concept (taking one concept in the formal statement for different one)
Incorrectly translating assumption
Incorrectly translating conclusion
Incorrectly translating type
We found that 67 of the 100 Lean statements are informalised correctly, and 81 of the 100 Isabelle statements are informalised correctly. The overall correctness rate is 74%. We estimate the total correctness rate of the MMA dataset to be similar. Wu et al. [022] found that even when only 25.3% of the autoformalization statements are completely correct, downstream theorem proving applications were still able to benefit drastically from the parallel dataset. Hence, we expect our MMA dataset, which is 3x more accurate, to be of great usefulness for the community.
In Table 2 (bottom) we present the types of errors out of the 200 randomly selected informalisa- tions. Note that one informalisation can potentially have multiple errors. We notice that the most common mistake made by GPT-4 in informalising is "Misunderstanding concept", which happens in 14.5% (29/200) of the translations. This is either because there is an inherent ambiguity in the formal expression and the context not enough to determine it, or that the language model is not able to determine the appropriate concept. Spotting these errors requires significant amount of expertise in both mathematics and formal languages. Designing an automatic filter to remove incorrect informalisations seems to be highly non-trivial. We leave improving the informalising language model, such that it produces more accurate translations, for future work.
Case Study. We study the informalisation examples from Table 3 of the 4 are correct, but when informalising the lemma pontt_minu_lee GPT-4 interprets the type "eint" be extended integers, which are usually defined as normal integers extended with negative and positive infinities. This translation is sensible, but not entirely correct: "eint" introduced in a theory of p-adic numbers to represent the codomain for the p-adic valuation this means that it only extends integers with positive infinity, which serves as a maximal element the order (i.e., the valuation of 0). Therefore, important to note that while we use state-of-the-an LLM (GPT-4) to perform the informalisations, the resulting MMA dataset is not perfect: rather than ground truth, informalisations in MMA should be treated as noisy approximations of it.
5

---

4 Experiment
To validate that MMA is a useful dataset for models to gain autoformalization abilities, we train two models from the LLaMA family and the Mistral family on a series of MMA data partitions. We manually evaluate the resulting models on two downstream benchmarks: miniF2F [Zheng et al.
2022] and ProofNet [Azerbayev et al.) 2023]. consisting of high-school mathematical competition and undergraduate--evv mathematical exercise problems respectively.
Experimental Details. We take LLaMA [Touvron et al. 2023] 33B (under the LLaMA license) and Mistral [Jiang et al.) 2023a] 7B (under an Apache license) as the base models, for they were the most performant open- weights model that we could fine-tune at the time of experimenting. We deliberately choose two models of different sizes and families to show that the improvement brought by MMA dataset is not sensitive to model size or family. For fine-tuning, we use the cross-entropy loss with the loss on the input masked out. We use the EasyLM [Geng 2023] software framework on TPUv4-64, with 32 megacores. We parallelise the model across 16 devices, and use local batch size of 8 sequences, with each sequence having a maximum of 512 tokens. We use the Adam W optimiser [oohhhhlovv and Hutter 2019], perform 5000 linear warmup steps with peak learning rate of 3 x 10 and then decay the learning rate with a cosine schedule for 35000 steps to 3 x 10 Preliminary experiments suggest that the final checkpoints of models are the strongest ones, use those to represent fine-tuning runs.
Fine-tuning Data Regimes. We trained the models for the same number of training steps to generate formal statements given their informal counterparts, on different partitions of MMA: Isabelle + Lean4; Isabelle only; Lean4 only. For each datapoint, we used a prompt format identical to the one in Section but with reversed input/output languages, and instructed the model to translate the statement in natural language to Isabelle or Lean accordingly. There are 88K informal-formal pairs of Lean4 data in one epoch of MMA, while for Isabelle there are 244K, 3 times as many. To reflect these proportions fairly, we fine-tuned the jointly trained model for 3.3 epochs, the Isabelle only model was tunet for 4.4 epochs, and the Lean4 only model was fine-tuned for 13.2 epochs.
It is possible that the ratio between data of the two formal languages influences the models' perfor- mances and a sweep of experiments over this ratio is potentially valuable. However, since fine-tuning the LLaMA model costs $2885 by TPU pricing, we are constrained by our budget and unable to perform this sweep.
5 Results
In this section, we analyse the performance of the trained models and their formalisation of realistic mathematical problems from high-school competitions and undergraduate--evel courses.
Loss and Accuracy. In Figure [ we plot the loss and the token accuracy with teacher-foreing Goyaa iet al. [0166 for the LLaMA model, on the Isabelle and the Lean4 validation sets for all 3 models. That is, we assess whether the ground truth token has the highest likelihood assuming every preceding token was predicted correctly. The figure illustrates that fine-tuning on MMA with one or both formal languages can drastically improve the language model's autoformalization capability, boosting their final validation token accuracies to above 90%. Comparing different fine-tuning regimes, we find that for the first 20000 steps, joint tin-ttnnng has higher validation loss than fine- fine-tuning on one formal language only. Afterwards, the single-language fine-tuning validation loss starts to increase while the joint fine-tuning one starts to plateau. At 40000 steps, joint fine-tuning's validation loss ~0.15 lower on the Isabelle validation set and ~0.1 lower on the I Lean4 validation set, respectively. The joint mme-tuning's 's final token accuracy on Isabelle's validation set is 1% higher than single-language fine-tuning, and 0.7% lower on Lean4's 's validation set. This 0.7% accuracy drop is likely because the single-language fine-tuning has seen 4 times more Lean4 material than the joint fine-tuning. We emphasise that the jointly fine-tuned model has seen 3/4 Isabelle and 1/4 Lean4 tokens of the languala models, and conclude that fine-tuning with multiple formal languages is much more data-efficient than with single- formal- formal-language language autoformalization data. We note that both loss and accuracy are proxy metrics of autoformalization capabilities, and in the rest of this section, we will examine autoformalization metrics that are better proxies, albeit more costly to evaluate.
Syntactic Correctness. In addition to monitoring automated training metrics such as validation loss and token accuracy, we used each model to formalise problems randomly chosen from two

---

Metrics on the Isabelle Validation Set
1.00
1.2
Token accuracy
0.8
0.6
0.4
0.85
10000
20000
30000
40000
10000
20000 Training steps
30000
40000
Training steps
Fine-tuned Isabelle only
Fine-tuned on Isabelle Lean4
Metrics on the Lean4 Validation Set
1.00
1.50
1.25
0.95
Token accuracy
Loss
1.00
0.90
0.75
0.50
0.85
0.25
10000
20000
30000
40000
0
10000
20000 Training steps
30000
40000
Training steps
Fine-tuned on Lean4 only
Fine-tuned on Isabelle + Lean4
Figure 1: The Isabelle and Lean4 validation loss and token accuracy of various models fine-tuned on different data regimes, represented by curves of different colours: Green Isabelle data only; Orange is the mixture of Isabelle and Lean4 data; and Purple is Lean4 data only. Fine-tuning both languages yields lower validation loss at the end of the training than fine-tuning on one.
benchmarks: miniF2F [Zheng et al. 2022 and ProofNet [Azerbayev et al. 2023 miniF2F is suite of 488 high-school competition mathematical problems multiple formal languages, and Jiang et al. 2023b) collected their ground truth informal counterparts. ProofNet has 371 self-contained undergraduate--eve mathematical exercise problems from analysis to abstract algebra with natural and formal descriptions. Moreover, the theme of these benchmarks makes train-test contamination less likely, since is rare that exercise problems get formalised and accepted by major formal librarics. In our evaluations, we randomly selected 50 problems from miniF2F and 50 from ProofNett
We tested if the generated formalisations are syntactically correct by the formal language (if they "compile"). The base models do not produce anything that compiles in Isabelle or Lean4 on the two benchmarks we used. The models fine-tuned on Isabelle generate 36% and 30% of Isabelle statements that compile on miF and ProofNet respectively, while the jointly fine-tuned model generates 24% and 18% respectively. An important caveat with the Isabelle language is that there can be variables in the statements with type annotation, and the statements can still be deemed syntactically correct. We observed that such statements generated by the model fine-tuned on Isabelle only are responsible for the high compilation rate, which effectivvly shows that while the compilation rate caps the proportion of completely correct formalisations, does not fully capture how good/useful the formalisations are. 14% and 6% of the formalisations generated by the model fine-tuned on Lean4 compile on miniF2F and ProofNet respectively. The jointly fine-tuned model has higher compilation rate miniF2F (20%) and slightly lower one ProofNet (4%) for Lean4 statements. Next, we go into how much assistance the model generations can offer to the actual formalisation practice miniF2F and ProofNet benchmarks.
Formalisation Quality. For the task autoformalization, the final and most important metrie is the quality of the formalisations generated. For each model, we inspect the 100 formalisations for:
7

---

LLaMA Autoformalizing Isabelle
Mistral Autoformalizin Isabelle
100
fine-tuned
100
fine-tuned
Fine-tuned on Isabelle only
Fine-tune on Isabelle only
80
Fine-tuned Isabelle Lean4
Fine-tuned Isabelle Leand
Counts
60
Counts
40
40
20
20
0
(4) >e from scratch
(3) Significant
(2) Moderate
(1) Trivial
(0) None
(4) >= from scratch
(3) Significant
(2) Moderate
(1) Trivial
(0) None
Correction Effort (lower is better)
Correction Effort (lower better)
LLaMA Autoformalizing Lean4
Mistral Autoformalizing Lean4
100
Not fine-tuned
100
Not fine-tuned
Fine-tuned Lean4 only
Fine-tuned Leand only
80
Fine-tuned on Isabelle + Lean4
80
Fine-tuned on Isabelle + Lean4
Counts
60
Counts
40
20
(4) >> from
(3) Significant
(2) Moderate
(2) Trivia
(0) None
(4) >us from scratch
(3) Significant
(2) Moderate
(1) Trivial
(0) None
scratch
Correction Effort (lower better)
Correction Effort (lower better)
Figure 2: The effort level takes to correct 100 model-generated formalisations into acceptable forms in Isabelle (top) and Lean4 (bottom) with LLaMA (left) and Mistral (right). The blue bars represent the models that are not fine-tuned; the green bars represent models fine-tuned on Isabelle data only; the purple bars represent models fine-tuned represent models fine-tuned on both Isabelle and Lean4 data. Generally, the models fine-tuned on both languages produce outputs that require less effort to correct than models fine-tuned on one.
Lean4 data only; and the orange bars
(1) whether they are completely correct formalisations: and (2) the amount of effort required to correct the formalisations. Two experts in Isabelle and Lean4 formal languages evaluated the formalisations, blind to which model generated them. The amount of effort is rated on a Likert scale from 0 to 4, with 0 meaning "no correction required" and 4 meaning "requiring similar more effort correct than formalising from scratch"".
Previous work autoformalization Wt al. 2022 Azerbayev et al. 2023 typically only considered the correcteess/ncorrettees and find even slightly incorrect formalisations useful to complete their task. This suggests that the evaluation metrics should be more nuanced [Collins et al., 2024). Therefore, in this work we instead spectrum based on the assistance they offer to humans. The manual
the formalisations. But humans often work interactively with LLMs
put each formalisation inspections were performed by two expert-level formal proof assistant users, who had no information about which model produced the formalisations. The evaluations are in the Supplementary Material.
In Figure 2 we plot histograms the effort level it takes human expert to correct model-generated formalisations in Isabelle and Lean4. We define formalisations that have correction effort levels 0 (none) or (trivial) as "acceptable with minimal corrections" We can see that models not fine-tuned cannot autoformalize to Isabelle and Lean4 at all: the vast majority of their formalisations require

---

Table 3: The average effort levels (lower is better) and their 95% confidence intervals of model- generated formalisations of the 200 evaluation samples.
Autoformalizing to Isabelle
Average effort level
95% confidence interval
Basc models
[4 44]
Fine-tuned on Isabelle
2.785
[2.622 - 2.9481 [2.251 2.579]
Fine-tuned on Isabelle Lean4
2.415
Autoformalizing to Lean4
Base models
3.995
[3,985 4.005]
Fine-tuned on Lean4
2.67
[2.501 - 228399
Fine-tuned on Isabelle Lean4
2.495
[2.318 2.672]
correction effort similar to or larger than that of formalising from scratch. The models fine-tuned on Isabelle data or Lean4 data perform significantly better: for the LLaMA models, they generate 6% and 10% of formalisations acceptable with minimal corrections for Isabelle and Lean4, respectively. For the Mistral models, they generate 20% and 26% of Isabelle and Lean4 statements, respectively, that acceptable with minimal corrections. The models fine-tuned on both Isabelle and Lean4 are even better in terms of assistance provided to human experts. 18% of LLaMA's Isabelle formalisations and 16% of its Lean4 formalisations are acceptable with minimal corrections, even though the model has seen fewer Isabelle tokens than the model fine-tuned on Isabelle only, and fewer Lean4 tokens than the mdoel fine-tur tuned on Lean4 only. For Mistral, the numbers are 29% and 31%. respectively. This suggests that there is considerable transfer between data in different formal languages, which benefits autoformalization, evidenced by the fact that the jointly fine-tuned models have superior autoformalization abilities in two formal languages with the same computational cost as the models fine-tuned on zero or one language. We further note that there is considerable discrepancy between the direct examination of autoformalization (Figure and the metrics of loss, accuracy, and syntactic correctness (Figure DT This highlights the unreliability of the proxy metrics.
Comparison with Few-Shot Prompting. Prior works on autoformalization have made heavy of few-shot prompting. Here, we contrast the autoformalization quality of models with few- shot prompting and fine-tuning, It was found that the Codex model with few-shot prompting can correctly autoformalize 13-16% of ProofNet theorems [Azerbayev al. 2023] and 25.3% of MATH [Hendrycks et al. [222T] theorems (which are much simpler than miniF2F and ProofNet). Our best autoformalization models with zero-shot fine-tuning can formalise 22% miniF2F and 12% on ProofNet that require none or trivial corrections (see Figure 2 which are similar or better than previous models, despite being much smaller (Mistral 7B instead of Codex). We use two benchmarks purposefully built for autoformalization as per standard, instead of MATH. Therefore, we think fine-tuning is promising approach to specialise and improve models for autoformalization.
Statistical Significance. We now investigate whether the improvement in models' autoformalization ability with the MMA dataset is statistically significant. In Table 3 we display the average effort level to correct outputs of models trained on each data mixture, and the 95% confidence interval estimated based on the 200 (100 from LLaMA and 100 from Mistral) evaluation samples. We see that for autoformalizing to Isabelle, fine-tuning the models on Isabelle and Lean4 gives outputs that are strictly better than just Isabelle, since the former has confidence interval entirely the left of the latter. Both are significantly better than the base models. For autoformalizing to Lean4, we see that fine-tuning with one or languages on the MMA dataset are both significantly better than not fine-tuning, Fine-tuning on both languages results in a smaller average effort level to correct Lean4 autoformalization outputs.
Discussion and Limitations
Data Contamination. Since the base LLaMA model we chose was pre-trained partially on data from the internet and GitHub, naturally we need to ask the question: "Has the LLM seen the evaluation materials during its pre-training phase and therefore the result is invalidated?". To answer this, we closely inspected the generations by the raw model and examined if any of them were repeating the ground truth formalisation. Our investigation found that in none of the cases did the base model

---

generate anything resembling the ground truth: most of its generations when instructed to translate statement from natural language to Isabelle or Lean4 either LaЛІ or Python code. Interestingly, one of its generations is a ITTEX code listing (the complete generation is in Appendix ( that looks like Isabelle code, but is ultimately not even syntactically correct. The code listing is followed by comments mentioning famous Isabelle AFP contributor. We hypothesise that this is caused by
the model having noisily memorised arXiv papers containing Isabelle content. Our investigation concludes that data contamination is not serious issue in our case.
Evaluation. Evaluating autoformalization is difficult: language models are very capable of generating formal statements that are syntactically correct, but do not express the meaning of the informal statements, as we have seen in Section Hence, there is no easy and reliable way to automatically assess the quality of formalisations generated by machine learning models. Two fairly reliable approaches to indirectly assess the quality of the generated formal statements exist: Wu et al. [2022] showed that autoformalizations can improve automated theorem proving models via expert iteration, illustrating that the autoformalizations are non-trivial; Diang al. [022] proposed to consider statements that can be proven and serve as lemmas for other theorems as good formal statements. However, these approaches require the use of automated theorem proving, which is expensive to set up. In our work, we manually evaluated formalisations on 100 randomly sampled formalisations for each of the 12 -nffrenie language pairs, and analysed the amount of effort needed to correct the outputs in Section 5 If we had more resources to inspect all generated formalisations, this could reduce the sampling variance and make our assessment more robust.
Continuously Pretrained Models for Mathematics. There are models that are continuously pretrained on mathematical materials from base models such as Llemma [Azerbayev et al., [202]] and DeepSeekMath (Shao et al.] 2024). They demonstrate significant improvements mathematical problem solving over the base models and can serve as better starting points for fine-
informal
tuning models. We did not experiment with them since they were published after our experiments.
7 Conclusion
In this paper, we constructed MMA, large, flexible, multi-language, and multi-domain dataset of informal-formal pairs. We demonstrated that language models can acquire superior autoformalization abilities by training MMA, and its use of multiple languages improves sample efficiency and final performance for autoformalization. We are convinced that MMA can very effectively benefit the theorem proving and AI for maths community by two facts: (1) the analytical fact that MMA's estimated correctness rate is 3 times higher than the parallel autoformalization data used by Wu et al. [022] which was very helpful; and (2) the empirical fact that fine-tuning language models on MMA make them significantly better autoformalization models. We release MMA for public exploration.
We sampled only one informalisation from GPT-4 for each of the 332K formal statements, which costs roughly US$3500 based on OpenAl's commercial pricing. If we had more resources, we would further boost the diversity of the informal statements by sampling more than one informal statement for each formal statement, and could extend to more formal libraries such as Isabelle's standard library, and more languages such as HOL Light and Coq.
In unsupervised machine translation literature, troc-ttranllttin typically uses the same model to translate in both directions [Sennrich et al. 2016 Lample et al., 2018). and iterates until the perfor- mance saturates. We were unable to do this, because GPT-4, the model we used for informalisation due its strong performance, is proprietary. The possibility of examining the full potential for iterated back-translation hinges on the existence of an open-source language model that is generally performant in both natural and formal languages. Since state-of-the-ar open models appear at great frequency, we leave the work of unifying and iterating language models for informalisation and autoformalization for the future with great hope.
Acknowledgement
We thank Fabian Gloeckle and Katherine M. Collins for useful discussions and feedback. AQU acknowledges the support of the Peterhouse Graduate Studentship.
10

---

References
M. Artetxe, G. Labaka, E. Agirre, and K. Cho. Unsupervised neural machine translation. In International Conference Learning Representations, 2018. URL https:://pperreiiev.. net/ Forum?id-SS2ogebaY
J. Austin, A. Odena, M. Nye. M. Bosma, H. Michalewski, D. Dohan, E. Jiang, J. Cai. M. Terry, Q. V. Le, and C. Sutton. Program synthesis with large language models, ArXiv, abs/2108.07732. 2021. URL https://appl www.peissaaaaanccccnlssaaaea
Arg/CorpusID 237 142385
7. Azerbayev, B. Piotrowski, H. Schoelkopf, E. W. Ayers, D. Radev, and J. Avigad. Proofnet: Autoformalizing and formally proving undergraduate-level mathematics. CoRR, abs/2302. 124333 2023. doi: 10.48550arriv.2302222522 12433
URL https: 1twp://.....55////// 2302
Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. M. McAleer, A. Q. Jiang, Deng, S. Biderman, and S. Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL Btt.htttps//openrrvviee
Ball. Proof claimed for deep connection between primes. Nature. 10, 2012.
G. Bancerek. Automatic translation in formalized mathematics. Mechanized Mathematics and Its Applications, 192)11 2006.
B. Barras, S. Boutin, C. Cornes, Courant, Y. Coscoy, D. Delahaye, D. de Rauglaudre, J.-C. Filliåtre, E. Giménez, H. Herbelin, et al. The coq proof assistant reference manual. INRIA, version, 6(11), 1999.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. de Oliveira Pinto, J. Kaplan, H. Edwards, Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, Gray, N. Ryder, M. Pavlov, Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss W. H. Guss,
A. Nichol, Paino, N. Tezak, Tang, Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, Murati, K. Mayer, P. Welinder, B. McGrew, Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.033744 2021. wwtp://////aa/.....////// 03374
URL
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110.1411 2021.
K. M. Collins, A. Q. Jiang, S. Frieder, L. Wong, M. Zilka, U. Bhatt, T. Lukasiewicz, Y. Wu, J. B. Tenenbaum, W. Hart, T. Gowers, W. Li, A. Weller, and M. Jamnik. Evaluating language models for mathematics through interactions. Proceedings of the National Academy of Sciences, 121, 2024. doi: 10.1073/pnas 2t..ppaaa...112222......
M. Cramer, B. Fisseni, Koepke, D. Kühlwein, B. Schröder, and J. Veldman. The naproche project controlled natural language proof checking of mathematical texts. In International Workshop Controlled Natural Language, pages 170-186. Springer, 2009.
L. M. de Moura, S. Kong, J. Avigad, F. van Doorn, and J. von Raumer. The lean theorem prover (system description). In A. Felty and A. Middeldorp, editors, Automated Deduction CADE-25 25th International Conference on Automated Deduction, Berlin, Germany, August 1-7, 2015, Pro- ceedings, volume 9195 of Lecture Notes in Computer Science, pages 378-388, Springer, 2015. doi: 26. URL htthhttts://ot 100.///700007783333 6-
26
D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL httpsttps. //tprr/vveeeeeeeieeeeee net/ Forum?id=hhwpp 1bM6EL

---

Y. Fu. H. Peng, L. Ou, A. Sabharwal, and T. Khot. Specializing smaller language models towards multi-step reasoning. In Krause, E. Brunskill, K. Cho, B. Engelhardt, editors, International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10421-10430. PMLR, 2023.
S. Sabato, and Scarlett,
mlr. prwss/ wwtps//v22////2 v202//u23dd2222 fu23d. html
X. Geng. Easylm: A simple and scalable training framework for large language models, March 2023. URL https: // tuub. com/young gereiggengggee Manng/
A. Goyal, A. Lamb, Y. Zhang, S. Zhang, A. C. Courville, and Y. Bengio. Professor forcing: A new algorithm for training recurrent networks. In D. D. Lee, M. Sugiyama, U. von Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016. Barcelona, Spain, pages 4601-4609, 2016. URL http:http////prreeeeiii http://roceedng.........pp.pp..
eeeurips.cc/papee.
E. Guzmán, P. Chen, M. Ott, J. M. Pino, G. Lample, P. Koehn, V. Chaudhary, and M. Ranzato. Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. CoRR,
2019. http::ttt:::/ttttt org/ abs/ 01382
J. Harrison. HOL light: A tutorial introduction. In M. K. Srivas and A. J. Camilleri, editors, Formal Methods in Computer-Aided Design, First International Conference, FMCAD '96, Palo Alto, California, USA, November 6-8, 1996, Proceedings. volume 1 1166 of Lecture Notes in Computer Science, pages 669- Springer, 1996. doi: 10.1007/BBP600318844 URLhttps https 10oii.ogg/1... 100.0/007000000 BBF0031184
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and Stein- hardt. Measuring mathematical problem solving with the MATH dataset. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021. December 2021, virtual. 2021. URL httpps//dattasotss benchatrtss--bbbecerreea proceediigsssoooo neurips httpippp.ccppppr///p -bataaaaa1777aa111aaaaaattt... round2. html
Y. Huang, X. Lin, Z. Liu, Q. Cao, H. Xin, H. Wang, Z. Li, L. Song, and X. Liang. MUSTARD: mastering uniform synthesis of theorem and proof data. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttp::/
net wwt.:eeooooriddddiiiiieeeee
A. Q. Jiang, W. Li, J. M. Han, and Y. Wu. Lisa: Language models of isabelle proofs. In Artificial Intelligence and Theorem Proving, September 5-10, 2021, Proceedings, Paper 17, 2021. URL http:///aitp-confereeute
17.p:/
A. Q. Jiang, W. Li, and M. Jamnik. Learning plausible and useful conjectures. AITP, 2022. URL tttp::/aaipp-cofffeeetee org/ httg//222/bbstraacc..oo 2aarrp_2222_pppeee_ 19. 1ff
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample. L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang. T. Lacroix, and W. E. Sayed. Mistral 7b, CoRR, abs/2310.06825,, 2023a. doi: 10.48550/ httpht
ARXIV.2310.0622.
org/ 10. 48540/550////// ar.550//a/Xvv... 06825
A. Q. Jiang, S. Welleck, J. P. Zhou, T. Lacroix, J. Liu, W. Li, M. Jamnik, G. Lample, and Y. Wu. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. 2023b. URL
G. Lample and A. Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019.
G. Lample, A. Conneau, L. Denoyer, and M. Ranzato. Unsupervised machine translation using mono- lingual corpora only. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver; BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL htttps://openrevieee http://wwww..eeefooorrreeaaaaae AZ
12

---

A. Lewkowyez, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, Schlag, T. Gutman-Sole al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 15:3843-38577 2022.
Z. Li, J. Sun, L. Murphy, Q. Su. Z. Li, X. Zhang, K. Yang, and X. Si. A survey on deep learning for theorem proving. CoRR, abs/2404.0999 099999 2024. doi: 1044850/ARXXVV24444009999 URL https://doi org/10. 10. 48550/ arXiv 2404. 09939
C. Liu, J. Shen, H. Xin, Z. Liu, Yuan. H. Wang, W. Ju. C. Zheng, Y. Yin, L. Li, M. Zhang, and Liu. FIMO: A challenge formal dataset for automated theorem proving, CoRR, abs/2309. 2023, doi: 10.48550ARRXIV2300000222
org////0...../00.. URL https:///// http://doo.orggg.////o// org/10. 48550/arXiv 2309, 04295
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net. 2019.
OpenAL. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
URL https://ap:
semanticscholar. org/ :oopusID::275::::
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, Welinder, P. F. Christiano. J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022. URL http: http::/pappers. mipppprrrs.ppspp.pmmo...p. http://spppppccccpepeeeeeccaeeae
L. C. Paulson. Isabelle A Generic Theorem Prover (with a contribution by Nipkow), volume 828 of Lecture Notes in Computer Science. Springer, 1994. ISBN 3-54--58244444 doi: 10.1007/ BFb0030541, URL URLhttpss: https:
1ttg//10.1007/BP0000044
B. Roziere, M.-A. Lachaux, L.. Chanussot, and G. Lample. Unsupervised translation of programming languages. Advances in neural information processing systems, 1 2000.
Scholze and J. Stix. Why abe is still conjecture. URL http://www. kurims. kyoto-u, ac. jp/motizuk//S22088.000 pdf. 2018.
R. Sennrich, B. Haddow, and A. Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics. ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Pa- pers. The Association for Computer Linguistics, 2016. doi: 10.18653/v1/p16-1000 1009. URL https://doi org/10/ 1111.11111111111111110
Z. Shao, P. Wang, Q. Zhu, R. Xu, Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi: 10448550ARXXI.244220.333 03300
URL htttps://doi org/10. 48550/ http///iivvvv.. .402
Shi, M. Suzgun, M. Freitag, Wang, Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, et al. Language models are multilingual chain-of-thouge reasoners. arXiv preprint arXiv:2210.03057, 2022.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023. doi: 1044550/arxxv22200.11911 2302. 13971
URL htttps://doo.....
Q. Wang, C. Kaliszyk, and Urban. First experiments with neural translation of informal to formal mathematics. In F. Rabe, W. M. Farmer, G. O. Passmore, and A. Youssef, editors, Intelligent Computer Mathematics 1 1th International Conference, CICM 2018, Hagenberg, Austria, August 13-17, 2018, Proceedings, volume 1 11006 of Lecture Notes in Computer Science, pages 255-270. Springer, 2018. doi: 10.11077978-333199-000 12-46 3- 978-3-319-980 90812-4-12222
URL htttps://daat http///...o.......//
13

---

Q. Wang. C. E. Brown, C. Kaliszyk, and J. Urban. Exploration of neural machine translation in autoformalization of mathematics mizar. In J. Blanchette and C. Hritcu, editors, Proceedings of the 9th ACM SIGPLAN International Conference New Orleans, LA, USA, January 20-21, 2020, pages 85-98, ACM, 2020. doi: 10. 14/11352377885. 3373827. URL pettps://doi.oom 10. 1145/ /141/3172222 387 3373827
Certified Programs and Proofs, CPP 2020,
A. N. Whitehead and B. Russell. Principia Mathematica. Cambridge University Press, 1925-1927.
Y. Wu, A. Q. Jiang. W. Li, M. N. Rabe, C. Staats, M. Jamnik, and C. Szegedy. Autoformalization with large language models. In NeurIPS, 2022. URL http mips www.: cc/ faapperrfflles files/
erence httl
K. Yang, A. M. Swope, A. Gu, R. Chalamala, Song. S. Yu, S. Godil, R. J. Prenger, and A. Anandkumar. Leandojo: Theorem proving with retrieval-augmenter lan- guage models. 2023. URL http://papest mips cc/ wwwpcpppppppaeeeeeeeeeeeeeeeeeee files/ htterr/eeeeeeeeeee//eee httpppappp/pppaaaapeeee and Bedd__Bnncmarksss html
T. Yu, R. Zhang, K.-C. Yang, M. Yasunaga, D. Wang. Z. Li, J. Ma. I. Z. Li. Q. Yao, S. Roman, Z. Zhang, and D. R. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. ArXiv, abs/1809.nm 809.08887, 2018. URL https
Irr..apuurrrr/CCC :tt 52815560
K. Zheng, M. Han, and S. Polu. minif2f: cross-system benchmark for formal olympiad-leve mathematics. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL attps://openrreiew.rr forumiidd=92PeguFFTT
14

---

A Additional Autoformalization Case Studies
In this section, we present some additional autoformalization examples.
In Figure 3] we display an informal statement from ProofNet, the reference ground truth Isabelle formalisation, and the formalisation attempts by 3 models with different finettuning data. Here analyse their autoformalization behaviours. We first note that none of the 3 model formalisations completely correct; each is inaccurate in its own way. The base LLaMA model does output anything resembling Isabelle code, but rather a regurgitation of the original informal statement followed by repetitions of the prompt. The model fine-tuned on Isabelle only and the model jointly fine-tuned on Isabelle and Lean4 both correctly translate the main assertion of the problem, but are wrong with the assumptions. The model fine-tuned on Isabelle only ignores the assumption that p is an odd number, introduces an irrelevant structure "poly_mod", mod", and imposes an equality constraint between and the partially applied predicate "residuc "residuc_primroot p". The imposed equality causes compiling error. The model jointly fine-tuned on Isabelle and Lean4 data does not state the correct assumptions that p is an odd prime, but gives the lemma name "let_p. _iet___oot odd prime" instead. This behaviour is likely due to the model not recognising the first sentence of the informal statement as an assumption and interprets as the lemma name. In this example, both fine-tuned models give significantly better formalisations than the base model, while the model fine-tuned on multi-language formal data makes fewer mistakes.
Informal statement:
Let p be an odd prime. Show that is a primitive root modulo iff (!=-1)/9 # 1(p) for all prime divisors g of - 1.
Ground truth Isabelle:
lemma exercise_4_8:
assumes "prime p" "odd p"
shows tresidue_primrooe factors (p 1). [a ((p 1) div q) 1] (mod p))
{ (4' q E
Base LLaMA model: Let p be an odd prime. Show that a primitive root modulo p iff (P-11//a * (p) for all prime divisors q of - 1.
The statement in natural language is theorem. The statement Isabelle is theorem.
The statement Isabelle is definition.
The statement in natural language is a lemma.
Fine-tuned Isabelle only:
1 lemaa (in poly_mod) "prime p"
defines "a residuu_pprimron p" shows "residue_ "residue_ppimmoot
+ (A q E
(p 1). [a ((p 1) div q) # 1] (mod p))*
Fine-tuned Isabelle Lean4:
lemma gett_p_od_ppriee
shows "residue prrriduueppimmrot P
(V q E pwwpp__fiitttoee factors (p 1). [a ((p 1) div q) # 1] (mod p))"
Figure 3: Case study autoformalization example: Ireland-Rosad exercise_44 from Prt We display the input informal statement, the reference ground truth formal statement, and formalisation attempts by the base LLaMA model, the model fine-tuned on Isabelle data only, and the model jointly fine-tuned on Isabelle and Lean4.
15

---

In Figure 4 the model attempts to autoformalize the statement that the remainder of 2003 divided mentioned previously, the base LLaMA model perform poorly: it repeats rephrased
by 11 version of the informal statement without attempting to formalise it in Isabelle. The model jointly fine-tuned on Isabelle and Lean4 finishes the task nicely. Despite not being identical to the ground truth (e.g., it treats the mentioned numbers integers rather than natural number), it should still be accepted as a perfect answer. In contrast, the output from the model fine-tuned on Isabelle only is unexpectedly bad it does not make sense either syntactically semantically.
Informal statement:
What is the remainder when 2003 is divided by 11? Show that it is
Ground truth in Isabelle:
theorem mathd_ mathd__umberther www.eneteeor_gee....
"2003 mod 11 = (1: :nat)
Base LLaMA model:
"The remainder of 2003 divided by 11 is 1. The remainder of 2003 divided by 11 is 1.
The remainder 2003 divided by 11 is 1. The remainder of 2003 divided by 11 is 1.
The remainder 2003 divided by 11 is
The remainder of 2003 divided by 11 is 1. The remainder of 2003 divided by 11 is 1. The remainder of 2003 divided by 11 is
The remainder of 2003 divided by 11 is
Fine-tuned on Isabelle only:
1emma. r_2003 [simp] "eval r_2003 [e] \<down>= 1"
Fine-tuned on Isabelle + Lean4:
lemma
(2003:: int) mod 11 1
Figure 4: Case study autoformalization example: mathd taaatd__nneeethor_.966 from miniF2F.
16

---

Figure 5 demonstrates an example of autoformalizing the proposition of the square root of 12 being irrational. The jointly fine-tuned model tackles the task correctly: uses the predicate rat æ to encode the rationality of which is totally valid; the ground truth uses equivalent encoding: E Q. Neither the base L.LaMA model nor the model fine-tuned Isabelle only does satisfactory job.
Informal statement:
Prove that there no rational number whose square is 12.
Ground truth Isabelle:
theorem exercise_1_2: assumes
Q. x 12" shows False
Base LLaMA model:
Theorem square_of sanare_of_12_s_nnae 1tt___i__ntttraiiin -1/x)~2 12"
"x 12"
"x 1/12"
"I 1/12" by (simp add: square_od _f of_one_divide bt_pu_yr___________e _ataare_f__oee_
"I = 12" by (simp add: equare_of_one wtt__diviie__yyyyeeee _nn))
"I = 12"
(simp add: square_of _ate__on______ iee__diiieebb__eeeeeeeeee
one)
"I 12" by (simp add: square_of_ _ouare_of_one_eeee
"I 12" by (simp add: sqgerrot...m.dddvee..e....rrrrr.
"I = 12" by (simp add: square_of_ ctwoooooooooovveee
_one)
"I = 12" by (simp add: oquare_of_one_e wtt.:/oww..vivieeeb.euuuurrrre
Fine-tumed on Isahelle only:
lemma
" \<lbaacc>\<acuue>>> 12\<rbrace> \<acute>N
:= 2 <<aute>> <abuaace><acuut>| 24brrbre
Fine-tuned on Isabelle Lean4:
lemma
3 x. is_rat A x = 12)"
Figure Case study autoformalization example: Rudin
exercise_ 1_2 from ProofNet.
17

---

In Figure 6, the model is prompted to formally state that 10a & 28a + for any real number The jointly fine-tuned model handles the autoformalization task nicely by correctly constraining the variable to be real number using the keyword 'fixes'. The base LLaMA model rephrases the informal statement and keeps repeating the prompts. The model fine-tuned on Isabelle only attempts to formalise this problem using limits in mathematical analysis, which is beyond our comprehension.
Informal statement:
For any real number a, show that 10a < 28a² + 1.
Ground truth Isabelle:
theorem algebra_binnomeddkeeee wtt.aa/oooooaueeeeoooo.oa.aaaaa. fixes :: real
shows "10 I 28 * a + 1"
Base LLaMA model:
For all a: real, 10 <= 28 * +
Translate the statement natural language Isabelle: For all real, 10 <= 28 a² +1
Translate the statement in natural language to Isabelle: For all a real, 10 <= a - 1.
Translate the statement in natural language to Isabelle: For all real, 10 28
Translate the statement in natural language Isabelle: For a: real
Fine-tuned on Isabelle only:
Lemma
fixes :: real
assumes "a 2 1
shows "(A n. pour / (n (1 pour (2 n))))
/ (1 powr n))"
"(A m. (1 a powr n)) n)
VoLangoongJongggghhtttroooo 1 / (1 + a powr (2 * n))"
Fine-tuned on Isabelle Lean4:
lemma. @hoo__0_le__2________1__
fixes :real
shows "10
Figure 6: Case study autoformalization example: algebra_.inommegdiieeeeeooeeeee binomeglaacrinng_.0l1/88aag from miniF2F.
18

---

Show there exists a point æ of S such that Figure presents a non-trivial problem in point set topology. It is problem from the ProofNet corpus, which originates from an exercise the classic topology textbook by James Munkres. The jointly fine-tuned model correctly encodes S using the term 'sphere 0 1' in Isabelle and manages function from one set to
to identify the predicate continuous, _aap to encode the continuity another. Nevertheless, it makes a couple of common mistakes, such as neglecting to convert the open set 'sphere to a topology and not recognising the inherent Euclidean topology on I. The model fine-tuned on Isabelle realises the topology problem but attempts to construct topology using random definitions. The base LLaMA model, as usual, reiterates the informal statement without attempting formalise it. Overall, for this example the jointly fine-tur tunettned model performs slightly better than the one fine-tuned on Isabelle only, while both outperform the base LLaMA model by large margin.
Informal statement:
Let f S¹ + R be continuous map. f(x) = f(-x).
Ground truth in Isabelle:
theorem exercise_22 24_2:
fixes a::"\a::mttric_ apace + real"
assumes "continuous_ map sto__of_ett (sphere 1)) euclidean f" shows "3 x sphere 1. f x (-x)"
Base LLaMA model:
Let f : S + R be a continuous map. Show there exists a point x of S such that f(x) f(-x). Translate the statement in Isabelle to mathematical statement: Let f +R be continuous map. Show there exists a point S' such that f(x) = f(-x). Translate the statement in Isabelle
mathematical statement in the language of theory: Let f 5 + R be continuous map. Show there exists point æ of St such that [(x) f(-w). Translate the statement in the language of set theory
mathematical statement in the language of set theory with the axiom of choice: Let f S + E be continuous map. Show there exists a point x of S such that f(x) f(-z). Translate the
statement in the language of set theory with
Fine-tuned on Isabelle only:
lemma cont contimuus,.om map_ 1emmam:
fixes f :: *"* a: ttg.ooooooccccsspppc..... + *b: >b: "b::t2_space"* "continuous, map (subtopology (powertop_real UNIV)
@ttandard_simplex p)) g" shows "3 x. x E S f x & (- x)"
Fine-tuned on Isabelle Lean4:
lemma. Daatinoous_map-sphhee :tt.mmpsspperreeiiistt:::: exists:
fixes "'a: ireal lereeallnomee_wteeeeee vector "b: caarral_normed_eetttoe assumes "continuous _mpp (sphere 0 g"
shows x. x f f(-x)
Figure 7: Case study autoformalization example: Munkres exercise _ from Proofliet
19

---

Figure refers to problem from the Putnam Competition, collected by the ProofNet dataset. The proposition is mainly about showing that specially defined function fn has no root in the closed unit disk on the complex plane. The jointly fine-tuned model nicely identifies the Isabelle definition proots_coumt to count the number of roots of polynomial within domain. Nevertheless, it fails to define the function fn in the informal statement nor correctly encode the unit disk (= C |e]' s 1). Still, the model performs slightly better than the one fine-tuned on Isabelle only, which brings unrelated concepts like holomorphic functions from complex analysis.
Informal statement:
Let be a positive integer, and let (n( = n + - 1)a+ (n 22)²² + ......2 +2- n- Prove that fn has no roots in the closed unit disk {= E C: | 11.
Ground truth in Isabelle:
theorem srercise_2018_bee
fixes m:at and f:: :n:"nat 3 complex complex"
assumes "n >
n z ( n. (n-
shows AA i > 0. z. cmod z S - f
0"
Base LLaMA model: Let n be a positive integer, and let (z) = + (n- 1)2+ (n 1--22 2 +... - n-
Prove 13. Translate the statement in Isabelle to formal language: Let n be a positive integer, and let fn(z) = n + (n 1)2+ (n 10-22% 2) +... n-1 Prove that fn has roots in the closed unit disk {2 E C: | 1). Translate the statement in formal language to formal language with quantifiers: Let n positive integer, and let (2) 2 E C, if 2 & 1, then (2)
that fn has roots in the closed unit disk {= e C: |2|
(n 1)2+ (n 2)
nー Prove that for all
Fine-tuned on Isabelle only:
Lemm ww_proots_in_uni.__e
fixes f::" = complex" and c::copp
defines "n zorder z" and "f, f n"
"open s" "z E s"
"f
defines "S {z. z E
f 0}"
defines "I = (A n. (f,) / (f n
shows
(V n. (f n) holomorphic_on 8) A
(V
E
- n) z S)
n z. 2 E A Afz
- (f n) 2 2 A
(A 2 s"
Fine-tuned on Isabelle Lean4:
lemma no so_rrots__rr wootsttiittiinn... wwt.wihhnnnetttt
fixes n::int
assumes "n>0" and "f * (A z. n)
shows "proots_count (f z) {z C R. < 1} 0"
Figure 8: Case study of the autoformalization example Putnam Pro
exercise_201s from
20

---

B An Attempted Formalisation from the Base LLaMA Model
Informal statement:
Which of the following is the value of Vlog, 6 + loga 6?
(
(B) v /log 6 (C) (D) 1ogg3 3 + logs 2
(E)
logg 6 + v /log3 6
Show that is (D) V Vlog 3 /logs 2.
Attempted formalisation by the base model: \begin code)] | theory sqrt log_2. 2_3_6
imports Main
begin
lemma "sqrt(log +og = sqrlll 23+ log_3 2)" proof
have "sqrt(log_2 2 26 log 6) sqri(log 26 + log_3 6)"
by (simp add: sqrt_deff
also have "sqrt(log_2
log_3 6) = sqrt(log_2 __ + log_322 2)"
by (simp add: log add def) finally show ?hhesis by simp
qed end (end(code)
The above code working.
Comment: What is the question?
Comment: @ @ManuelEberl have edited the question.
Comment: don't understand what you are trying do. What the question?
Comment: ManuelEberl have edited the question.
Comment: don't 't understand what
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and the introduction clearly state the claims made in this paper, and include list of contributions made in this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Section 6 discussed aspects of the work that are subject limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We have no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the full description of the experiments, the code, the dataset used, and the resulting models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: Our dataset is open- as mentioned Section public repository.
The code used from
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpollliiggiddee/ooooooe http:///ddd//oeSuuubnnn ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We carefully detail these in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the 95% confidence intervals for our main results and analyse them in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detail the resources used in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pw.//.ccoo....o/ooo////aa
Answer: [Yes]
Justification:
We do conform to the code of ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work advances formal mathematics, which has no societal impacts as far as we can see.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We clearly indicate the licenses for the datasets in Section B and the models in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The new assets derivatives of the LLaMA and Mistral models, whose original documentations apply.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "8BjEogjarZ", "reviewer": "Reviewer_ctnj", "summary": "This paper introduces MMA, a large-scale dataset consisting of informal-formal pairs of mathematical statements (i.e., parallel autoformalization data) in two types of formal languages, Isabelle and Lean4. The dataset encompasses statements from multiple domains and exhibits high quality. It was constructed using a back-translation method, converting from a formal language to an informal language, to improve data quality.", "questions": "1.\tIs it fair to control the number of training steps across different set of training data? Why not control for the size of the training dataset and the number of training epochs instead?\n2.\tIn line 262-265, \"it does not fully capture how good/useful the formalisations are\", why are the formalizations considered not good/useful as long as they lack type annotations? What proportion of the statements are of this type? If these are removed, what is the proportion of syntactically correct statements generated by models that have been fine-tuned singly or jointly?", "limitations": "The metrics “loss” and “token accuracy” on the validation set are limited, as stated above. This could affect the correctness of all analyses and the corresponding conclusions.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper constructs a large-scale, high-quality, and diverse dataset for autoformalization.\n2.\tThe paper analyzes the benefit of training on multiple formal languages to improve the performance of single-language autoformalization.\n3.\tThe motivation for the method and the explanation for the data quality are written clearly.", "weaknesses": "The experiments and the corresponding analysis are not robust enough.\n1.\tThe metrics “loss” and “token accuracy” used on the validation set are limited, as there could be multiple correct autoformalization results beyond the given reference. One piece of evidence for the limitation of these metrics is that in Figure 1, the model fine-tuned on Lean4 alone shows a higher loss, suggesting worse performance, yet its token accuracy is actually better.\n2.\tIn line 242-243, the paper claims, “This 0.7% accuracy drop is likely because the single-language fine-tuning has seen 4 times more Lean4 material than the joint fine-tuning”. However, another potential explanation could be that the joint fine-tuning training data contains more Isabelle than Lean4, potentially leading to a deterioration in Lean4 performance. The authors should investigate this possibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback and the time they invested in reviewing our paper. We address specific points raised by the reviewer:\n\n> **The experiments and the corresponding analysis are not robust enough.**\n> **1. The metrics “loss” and “token accuracy” used on the validation set are limited, as there could be multiple correct autoformalization results beyond the given reference. One piece of evidence for the limitation of these metrics is that in Figure 1, the model fine-tuned on Lean4 alone shows a higher loss, suggesting worse performance, yet its token accuracy is actually better.**\n\nWe fully agree with the reviewer that the “loss” and “token accuracy” metrics do not show the full picture. We point out in line 270-271 that the final and most important metric for the task of autoformalization is that of the formalisation quality. For this metric, we measured it by having human evaluations of the difficulty/effort it takes to correct the machine-generated formalisations.\nWe still wish to include the “loss” and “token accuracy” metrics for two reasons: (1) These two metrics are fairly standard in language model pre-training and fine-tuning; (2) We think by showing the discrepancy between these two metrics with the “ground truth” formalisation quality metric, we highlight their unreliability and emphasise the importance of the human evaluation. We will make the latter point more pronounced in the updated version of the paper.\n\n\n> **2. In line 242-243, the paper claims, “This 0.7% accuracy drop is likely because the single-language fine-tuning has seen 4 times more Lean4 material than the joint fine-tuning”. However, another potential explanation could be that the joint fine-tuning training data contains more Isabelle than Lean4, potentially leading to a deterioration in Lean4 performance. The authors should investigate this possibility.**\n\nWe fully acknowledge and apologise for the lack of precision in our statement in line 242-243. We will update the paper to show both possible hypotheses: that the two fine-tuning runs have very different Lean4 data quantity, or that the two fine-tuning runs have very different Isabelle-Lean4 data proportions. Due to the limitation on computational resources, we are unfortunately unable to run a sweep of data proportion to confirm/refute the hypothesis, but have to leave it as future work.\n\n\n> **Is it fair to control the number of training steps across different set of training data? Why not control for the size of the training dataset and the number of training epochs instead?**\n\nThis is a very good question! Controlling the total number of training steps and the number of training epochs have their respective advantages: controlling the number of training steps guarantees that the trained models go through an equal number of updates, and controlling the number of training epochs guarantees that the datapoints have been seen by the models for an equal number of times.\n\nIn our preliminary experiments, models keep getting stronger as training progresses. In our current experiments, the joint fine-tuning consists of 3.3 epochs for both Isabelle and Lean4, while the Isabelle-only and the Lean4-only fine-tuning consist of 4.4 epochs and 13.2 epochs of the respective datasets (line 226-227). If we control the number of training epochs for the individual fine-tuning experiments, we will end up with training runs lasting 75% and 25% as long as current ones respectively. This will result in two significantly weaker models. Since the stronger models fine-tuned on single languages are worse than the jointly fine-tuned model, we have reasons to believe that their weaker versions are even worse. Therefore, this will not change our conclusion that multi-language training considerably benefits autoformalization capabilities.\n\n\n> **In line 262-265, \"it does not fully capture how good/useful the formalisations are\", why are the formalizations considered not good/useful as long as they lack type annotations? What proportion of the statements are of this type? If these are removed, what is the proportion of syntactically correct statements generated by models that have been fine-tuned singly or jointly?**\n\nAs a design choice, open variables in theorem statements in Isabelle will be implicitly universally quantified: when properdivisor_sum is not a built-in definition, lemma \"properdivisor_sum 18 = 21\" is interpreted as lemma \"!!properdivisor_sum. properdivisor_sum 18 = 21\", where !! is meta-level forall. In contrast, in Lean4, variables need to be explicitly quantified within the statement (or declared earlier in the theory file). Due to this design discrepancy, the Isabelle syntax checker is more tolerant, allowing more false statements, whereas Lean’s syntax checker is stricter (at the cost of slightly more verbose statements).\n\nOn a rough inspection (because inspecting very carefully takes as much time as re-doing the evaluations), we find that 19% of the Isabelle formalisations by the Llama model have this issue. So having this removed will reduce the syntactic correctness of the model fine-tuned on Isabelle alone to 14%, while the jointly fine-tuned model rarely has this issue and maintains a 21% syntactic correctness. In the updated version of the paper, we will do a closer inspection and report the corresponding figures.\n\n> **The metrics “loss” and “token accuracy” on the validation set are limited, as stated above. This could affect the correctness of all analyses and the corresponding conclusions.**\n\nWe thank the reviewer for pointing this out! We will make it clearer that these two metrics have limited correlation with the model quality in the updated version, as discussed above.\n\nWe would like to thank the reviewer for their valuable feedback, which has greatly helped us improve our paper. Given the improvements we've made, we kindly request the reviewer to reconsider their score, or give indication for further improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kIYzNBb27L", "reviewer": "Reviewer_zhuU", "summary": "The authors use backtranslation to create a large (~332k samples) dataset of formal-informal statement pairs in the Lean4 and Isabelle formal proving languages: they take formal samples from a Lean4 and an Isabelle proof library, and ask GPT4 to restate them informally. They use this dataset to finetune two open source LLMs (Llama 33B and Mistral 7B) on the task of autoformalisation: converting informal statements to formal ones, in Lean4 and in Isabelle, comparing the result of this finetuning to the initial state of the LLMs and ablating on training with both languages as opposed to just the target languages.\n\nThe authors invest time and manually evaluate the performance of their trained models on 50 samples from each of two small benchmarks, proofnet and miniF2F (total 100 per evaluated model), to draw their conclusions.\n\nFor both languages, the fine tuned models outperform the base models on autoformalisation. For Isabelle, they find that training with the combined dataset (containing both Lean4 and Isabelle samples) is better than the relevant monolingual, for Lean4 the difference is slightly less clear. \n\nI note the mixed dataset contains mostly Isabelle statements: 244k/332k, and personally wonder if this relates to the found results - the influence of this ratio is not explored. This makes sense however: evaluating the correctness of autoformalised statements is non trivial, and easily automated metrics such as training loss, or per-token accuracy in a teacher forcing setting, are not immediately indicative of correct formalisation.", "questions": "Clarifications, nits, comments:\n1. Are the MMA statements disjoint from those of ProofNet and MiniF2F?\n2. line 62 \"strong\" autoformalisation: subjective\n3. section 2 \"rule based informalisation\" vs \"symbolic informalisation\" it is not clear to me if these are the same or different. If the same, choose one term, if different, clarify difference.\n4. \"Manual alignment\" also unclear\n5. line 86: appreciate the reference to specific case study, but would appreciate some lines on what \"syntactically incorrect segments\" are directly in this paper for completeness.\n6. line 93: some lines on how there is a setting where there is a target-to-source model, but not the other way around, would be appreciated. Speaking of which: given all this dataset is based on GPT4 backtranslations: how does GPT4 perform at autoformalisation?\n7. line 95: \"usually, the back translation process is...\" - would appreciate a/some sources on this!\n8. line 99 what language is Azerbayev et al 2023 on?\n9. line 116 \"high quality\": subjective, not clear by what measure this is said\n10. line 119 \"satisfies both of the criteria\": did you check for diversity of the informal statements? how did you measure it?\n11. line 142 do you mean \"generation\" (not curation) cost?\n12. nice to have: line 148 does gpt4 add further licenses? what is the license of the combined dataset (is it just \"both\", or do they merge)?\n13. table 2: what's that massive isabelle statement (24331 characters)? is it an anomaly? what happened there?\n14. line 178: what is misunderstanding a concept - can you give a concrete example? similarly line 181 with \"type\".\n15. line 191 vs lines 168 and 126: are formal expressions complete and precise or inherently ambiguous? soften/clarify statements to resolve this apparent contradiction\n16. line 205: \"train\" -> \"finetune\"\n17. line 208: what languages are miniF2F and ProofNet in? comes up again in line 250\n18. nice to have: would be nice to see metrics on lean when fine tuned on just isabelle, and vice versa. correctness evaluation probably too labour intensive, but loss/token-accuracy curves maybe. but maybe they'll just be zero. i recognise this data may not be obtainable now that models have already been trained (don't know how hard/expensive it is to train the models)\n19. line 227: number of epochs doesn't seem to quite line up with equalising number of samples seen/steps made, maybe because you are giving rounded numbers? doesn't particularly matter but strange\n20. line 235: figure shows loss and per-token accuracy, which are not necessarily indicative of autoformalization capability, rephrase.\n21. fig 1: not so clear what the per token accuracy of the models are in first 2k train steps, would appreciate knowing - maybe can have a line on this in body of paper if dont want to distort images\n22. line 243: it has seen lean material 4x more, but it has not seen 4x more lean material (i.e. it is seeing same material)\n23. line 242-243: unconvincing/invalid interpretation: if this were the explanation, we would see a similar (but weaker) drop for Isabelle, but instead, Isabelle sees a gain.\n24. lines 245-246: lean4 results dont support this\n25. line 258: \"... generate 36% and 30% of Isabelle statements that compile...\" unclear, rephrase\n26. lines 256-269: seems compilation rate is discussed for only one model (Llama?). Would prefer to have information on both models. If not, at least clarify which model is being discussed here.\n26. lines 262-265: need elaboration - I imagine you have support for this claim, but need to see it. Are you saying that a lot of the generated Isabelle statements type checked, but were incorrect? Give numbers\n27. lines 265-267: more mixed results regarding whether it is better to train with mixed or monolingual data. I wish the paper was more up front on this inconclusivity, it appears to have a slight \"joint-is-better\" narrative, that is not critical to its contribution. \n28. line 296: \"mdoel\"\n29. line 308 again insufficiently up front about inconclusive results regarding mono/multi lingual data for lean4 (choosing to focus only on the comparison to not fine tuning at all)\n30. line 309-310 don't particularly like this statement when the confidence intervals overlap (table 3), can be up front\n31. need more of a gap under table 3\n32. line 320: did fine tuning surface any memorised info? would have been interesting to check\n33. line 340: call to \"sample efficiency\" - define and clarify how this relates to findings?\n34. last 2 paragraphs of conclusion belong more in \"limitations\" in my opinion\n\n35. what are the (math) domains of the data used to create MMA (within AFP and mathlib4), and the domains of miniF2F and proofnet, and how do they relate to each other? I.e., are miniF2F and proofnet in- or out-of- distribution for the MMA training set, with respect to type of math considered? What about specifically the subsets of miniF2F and proofnet that were used for the evaluations in this work?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Provide dataset that will be useful for fine tuning work on autoformalisation, and greater than previously available datasets\n2. Method for generating dataset is straightforward: can be extended in future with additional languages and samples, or remade with better models (for better backtranslation)\n3. Demonstrate the positive value of dataset for finetuning models for autoformalisation (and carry out the non trivial evaluation needed to do this)\n4. Some investigation of whether it is better to use multi- or mono-lingual data for fine tuning autoformalisation models\n5. Clearly written\n6. While this is in the appendix - autoformalisation case studies are interesting for understanding what is happening. Similarly, in main paper, the analysis of 200 samples from the generated dataset (table 2) helps get a sense of its quality", "weaknesses": "1. Dataset is automatically generated by an LLM, and no filters were applied or considered. Naturally, some of the LLM's informalisations will be incorrect (manually inspecting 200 samples from the dataset, the authors find 52 with errors (table 2)). I wonder if further manual inspection would have raised any obvious failure cases that could be automatically flagged and filtered from the full dataset. \n2. I would have liked an investigation of impact of the [ratio of different languages in the training set] on the [performance of the fine tuned model]. Specifically because the models fine tuned on the full dataset did better on Isabelle than when trained only on Isabelle, whereas for Lean this wasnt as clear, despite Isabelle having the larger dataset, I wonder if the issue for Lean was that it was 'drowned out' by the amount of Isabelle samples in the data.\n3. I don't know how open NeurIPS specifically is to dataset rather than method contributions, and this seems mostly a dataset contribution. I do think it is a valuable one though.\n4. The dataset is based on GPT4 backtranslations, and the evaluations are of open source models before and after fine tuning with these backtranslations. I wonder, is this actually just a distillation of GPT4 capacities into open models, or does this dataset actually advance autoformalisation? A comparison with GPT4 autoformalisation ability (eg just on proofnet and minif2f) would help settle this question.\n5. Majority of improvement in token accuracy and loss of the models happens in first few k steps (fig 1). Would have been nice to do an evaluation of the models (ie on the benchmarks) at that stage, to see if remainder of steps is necessary\n6. *Important!* Missing baselines/discussion: the question of whether fine tuning (as done in this work) is at all the correct approach for autoformalisation is not raised. Given that there is existing work on using few-shot learning for autoformalisation, the question is relevant and cannot be ignored. The few shot alternative should be properly discussed. At minimum, the existing reported results (Azerbayev at al, Wu et al) should be recalled and compared to in this paper (to the extent the comparison is possible, see next point). (Ideally, the models would have been evaluated in a few shot setting on the same benchmark-subset as used in the paper, though I recognise this is a labour intensive request.)\n7. *Important!* Nonstandard benchmarking: The models in this paper are evaluated on the (mixed) subsets of two different benchmarks, proofnet and minif2f. This makes it hard to compare the results to those reported in other autoformalisation works, which use the benchmarks in whole and separately from each other. I hope the authors can at least update the report to list the disaggregated results on the two benchmarks. Independently, please list exactly which samples were taken from the two datasets for the combined one, to allow accurate comparisons to these results in the future. The use of only a small subset of the main benchmarks (50 samples each) also weakens the significance of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback and the time they invested in reviewing our paper. We address specific points raised by the reviewer:\n> **Dataset is automatically ...**\n\n- We thank the reviewer for pointing out that there could be potential improvements to the dataset filtering process. We will include the complete informalisation evaluation results in the updated version for completeness.\n- We did not find any obvious failure cases that can be easily filtered out. Table 2 Right gives a breakdown of the errors made by GPT4 during the informalisation process. It can be seen that the most frequent errors are caused by the misunderstanding of formal concepts, e.g., not correctly understanding the context of inverse operations. We find that in general, spotting these errors requires a fair amount of domain expertise in mathematics and formal languages, and believe it to be difficult to find an automatic filter that does the same job.\n- We will update the paper to say this explicitly and let the readers judge after seeing the informalisations made.\n\n> **I would have liked an investigation of impact of ...**\n\nWe agree with the reviewer that this is a valuable experiment to run. However, we are constrained by computational budget and unable to run the said experiments. To note, our main experiment with llama 30B took 14 hours on a TPUv4-64 to run, which by [TPU pricing](https://cloud.google.com/tpu/pricing) costs $2885. We are unable to do a sweep experiment that controls the Isabelle:Lean4 ratio and evaluate them accordingly. We recognise this as a limitation of the work and will update the paper to say so accordingly.\n\n> **I don't know how open ...**\n\nWe think the methodology (back-translation + fine-tuning) can be expanded to other formal languages and datasets than Isabelle AFP and Lean4 mathlib4. The finding that multi-language diversity can make the resulting models stronger and more robust is a contribution in addition to the methodology. Hence we think the paper has more value beyond the dataset.\n\n> **The dataset is based on GPT4 backtranslations, ...**\n\nWe think a direct comparison of our fine-tuned models with GPT4 has a confounding factor: model representation capability. The GPT4 model is rumoured to have 1.76T parameters while the models we fine-tuned have 30B and 7B parameters respectively. There is a 60-250x difference in model size and GPT4 thus has a significant advantage in terms of its representation capabilities.\n\nWe think the fairest comparison would be to fine-tune a GPT4-level model on its own backtranslations, and see the model performance before and after the fine-tuning. It is impossible to do so for GPT4, but becomes more realistic as new open-weight models start to catch up and surpass GPT4’s performance. We leave this for future work as it still requires a significant amount of computational resources to adapt an open-weight model like Mistral Large 2 or Llama 3.1 405B for the autoformalization task.\n\n> **Majority of improvement ...**\n\nWe performed some preliminary experiments on the intermediate checkpoints without performing the full suite of evaluation because it is expensive to do so. We find that the model’s autoformalization quality continues to improve as training progresses. We will update the paper to say so explicitly.\n\n> **Important! Missing baselines/discussion: ...**\n\nWe agree with the reviewer that few-shot prompting should be compared as a competing technique. We will compare the results from Azerbayev et al. and Wu et al. below:\n- Azerbayev et al. (in Lean3): Codex with few-shot prompting can correctly autoformalize 13-16% of ProofNet theorems. ProofGPT 1.3B and 6.7B have 0 accuracies with few-shot prompting on the same dataset. ProofGPT-1.3B can correctly autoformalize 3.2% of ProofNet theorems after distilled backtranslation (similar to our methodology).\n- Wu et al. (in Isabelle): Codex with few-shot prompting can correctly autoformalize 38 out of 150 problems (25.3%) from MATH, which is of much lower difficulty than miniF2F and ProofNet. \n- For our best models without few-shot prompting, we can achieve 22% correctness on miniF2F, which is similar to the Wu et al. result but on a much harder dataset. We can achieve 12% correctness on ProofNet theorems, which is similar to Azerbayev et. al result but with a much much smaller model (Mistral 7B instead of Codex).\n- In view of the above comparisons, we think fine-tuning presents a very promising approach, which is compatible with few-shot prompting as well. But for simplicity, in this paper, we only consider the vanilla case of zero-shot fine-tuning. We will update the paper with these comparisons.\n\n> **Important! Nonstandard benchmarking: ...**\n\n- We note that only ProofNet (Azerbayev et al.) uses the complete dataset for comparison. The earlier work examining autoformalization (Wu et al.) used a subset of 150 MATH problems for evaluating autoformalization quality.\n- We will provide the disaggregated results on the two benchmarks. We already include the exact statements and evaluation results of the two subsets in the supplementary materials.\n\nWe thank the reviewer for the very careful reading and feedback! Due to space limitations, here we will answer some questions. And for the rest of the points, we will modify the updated paper accordingly.\n\n1. Yes\n3. They are the same. We will update both to “symbolic informalisation”.\n8. Lean3\n10. Since the formal statements come from diverse mathematical fields, we expect the informal corresponding statements to also be diverse. We examined the informal statements and found it to be so.\n12. GPT4 outputs are not constrained by licenses, only subject to Terms of Service by OpenAI.\n\nWe would like to thank the reviewer for their valuable feedback, which has greatly helped us improve our paper. Given the improvements we've made, we kindly request the reviewer to reconsider their score, or give indication for further improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Kux8N4Yt1", "reviewer": "Reviewer_ByiA", "summary": "The paper presents a dataset MMA and a model trained on the same. The dataset comprises informal-formal pairs of theorem which are generated using LLM. The experiment shows decent results on the autoformalization tasks of miniF2F and ProofNet benchmarks. The authors also claim that autoformalization performance improves when trained on multi-lingual data. These characteristics generalize over different sizes of models, supporting the hypothesis that multi-lingual training is good for autoformalization.", "questions": "1. Can you separate Table 2 (Right) into various Error Types per language? Right now all the 200 MMA formalizations from the samples are presented together without the distinction between Lean vs Isabelle.\n\n2. The authors talk about few-shot prompting vs instruction prompting on Page 4. I understand the budget reasons, however, I would like to see some case studies showing the qualitative differences between the two strategies (for a small subset of data) in this domain.\n\n3. In Figure 2 (top), the bar plot for \"Llama Autoformalization Isabelle\" shows almost no correct formalization for the Isabelle-only model. I would like to the examples of those autoformalization that were possible because of the Isabelle + Lean model. A qualitative analysis of when this type of transfer is effective will be appreciated.", "limitations": "The authors do try to discuss some limitations, however, I would like to see more. I'm interested in seeing the resemblance of the test and train data since it was generated by GPT-4, and there is a potential chance of data leakage. Since the generation of informalization cannot be controlled, this risk always exists and hence just a comparison with the baseline is not sufficient.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors present a strong case for how and why multi-lingual training is beneficial for autoformalization. This indicates the need for further studies on how transfer learning helps in training across different formal languages. We need to understand if it is only applicable to Lean and Isabelle or is applicable to other languages like Coq as well.\n\n\n2. Adequate human studies have been conducted to test different aspects of the training data generated. The authors also try to thoroughly test the statistical significance of their findings which is great.\n\n\n3. I appreciate the effort put in to manually verify a set of autoformalizations generated during the evaluations.", "weaknesses": "1. Even though, authors try manually to sample from the data, the sample size is often small (which is understandable). However, I want to see how the numbers change with varying sample sizes. For example, instead of just one experiment with 100 sample size, the same experiment with varying sample sizes of 50, 75, and 100 showing similar trends will be appreciated.\n\n2. Authors make a key observation that informalization is much easier than formalization, however, I don't find enough justification for this in the paper. Some studies are mentioned on Page 4, but these observations may not generalize well on the dataset in question.\n\n3. The authors accept that the GPT-4 informalizations should be considered as a noisy approximation. This actually is a big limitation. Since not every informalization is verified, there is a potential chance of leakage of the test data itself. Since informalization is generated by GPT-4, it might as well generate informalization for theorems other than the formal statement in question (maybe something which was similar to the formal statement). If such a similar formal statement is present in the test data, then this potentially can help the model perform better on test data (this can be considered a type of leak in the training data). I can understand that this may not be a common case. However, some case studies about the resemblance between the train and test data should be conducted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback and the time they invested in reviewing our paper. We address specific points raised by the reviewer:\n\n> **Even though, authors try manually to sample from the data, the sample size is often small (which is understandable). However, I want to see how the numbers change with varying sample sizes. For example, instead of just one experiment with 100 sample size, the same experiment with varying sample sizes of 50, 75, and 100 showing similar trends will be appreciated.**\n\n- We appreciate the reviewer for raising this point. We want to first clarify that each model is evaluated on 100 randomly selected examples, with 50 from miniF2F and 50 from ProofNet. Hence, we can randomly select 50, and 75 from these 100, to perform the required analysis. We do exactly this below:\n- We examine the autoformalization quality of the Mistral model fine-tuned on both Isabelle and Lean4, on 50, 75, and 100 random examples respectively. The plots are in Figures 1 and 2 in the general response. We can see that for both languages, the autoformalization quality proportions change very little as we go from 50 to 75 to 100 examples. This validates the sufficiency of having 100 examples as we can project that with more examples (since they are all randomly chosen), the autoformalization quality will not change dramatically, and conclusions drawn based on 100 samples will generalise.\n\n> **Authors make a key observation that informalization is much easier than formalization, however, I don't find enough justification for this in the paper. Some studies are mentioned on Page 4, but these observations may not generalize well on the dataset in question.**\n\nAnalytically, informalisation does not have as stringent syntactical requirements as formalisation, so is more easily accepted. Empirically, Wu et al. and Azerbayev et al. both experimentally observed that informalisation is easier than formalisation on two datasets of different difficulty levels. Therefore, we have reasons to believe that the observation generalises to our dataset.\n\n> **The authors accept that the GPT-4 informalizations should be considered as a noisy approximation. This actually is a big limitation. Since not every informalization is verified, there is a potential ...**\n\nWhen we conducted the taxonomy of the informalisation, we noticed that GPT-4 always closely followed the input it is given and never generated something that is unrelated. In the 200 randomly chosen examples, GPT-4 always closely followed the formal statement. Since the MMA dataset is gathered from AFP and mathlib4, they are disjoint from the miniF2F and ProofNet benchmarks. This is because miniF2F and ProofNet benchmarks are deliberately prevented from becoming entries in AFP and mathlib4 to prevent contamination. Hence, we see no reason to particularly worry about benchmark contamination in our work. We will update the paper by attaching the 200 randomly chosen informalisation examples for the reader to examine as well.\n\n> **Can you separate Table 2 (Right) into various Error Types per language? Right now all the 200 MMA formalizations from the samples are presented together without the distinction between Lean vs Isabelle.**\n\nWe appreciate the reviewer’s suggestion and present the error types per language below:\n| Error type    | Isabelle | Lean4 |\n| -------- | ------- | ------- |\n| None | 81 | 67 |\n| Hallucination | 2 | 6 |\n| Misunderstanding concept | 11 | 18 |\n| Incorrect assumption | 2 | 9 |\n| Incorrect conclusion | 2 | 6 |\n| Incorrect type | 4 | 8 |\nWe will update the paper to use the per-language breakdown.\n\n> **The authors talk about few-shot prompting vs instruction prompting on Page 4. I understand the budget reasons, however, I would like to see some case studies showing the qualitative differences between the two strategies (for a small subset of data) in this domain.**\n\nWe would like to clarify that the reason behind going for instruction prompting instead of few-shot prompting is not cost. As we explain on Page 4, there are a large number of mathematical domains in AFP and mathlib4, making it impractical to manually come up with few-shot prompts for every single domain. Hence, we consider the former strategy not eligible for constructing the MMA dataset. We can further conduct the qualitative case studies if the reviewer deems it necessary.\n\n> **In Figure 2 (top), the bar plot for \"Llama Autoformalization Isabelle\" shows almost no correct formalization for the Isabelle-only model. I would like to the examples of those autoformalization that were possible because of the Isabelle + Lean model. A qualitative analysis of when this type of transfer is effective will be appreciated.**\n\n- We thank the reviewer for this suggestion. We will have one case study in the rebuttal and give a further analysis of the autoformalizations only doable by jointly trained models in the updated paper.\n- Case study 1. \n  - Informal statement: “Find the remainder when $91145 + 91146 + 91147 + 91148$ is divided by 4. Show that it is 2.”\n  - Ground truth: \"theorem mathd_numbertheory_640: \"(91145+91146+91147+91148) mod 4 = (2::nat)\"\n  - Jointly trained model autoformalization: lemma \"(91145 + 91146 + 91147 + 91148) mod 4 = (2::nat)\"\n  - Isabelle-only model autoformalization: \"lemma assumes \"x^4 + 34578*x^3 + 1996*x^2 + 688*x + 2 = 0\" shows \"(9114*x^6 + 334575*x^5 + 496*x^8 + 353*x^7 + 4*x + 2) / 4 = 2\"\n  - One can see that the Isabelle-only model gives a degenerate solution while the jointly trained model manages to correctly formalise it.\n\nWe would like to thank the reviewer for their valuable feedback, which has greatly helped us improve our paper. Given the improvements we've made, we kindly request the reviewer to reconsider their score, or give indication for further improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x7mBxUxEi2", "reviewer": "Reviewer_8WKp", "summary": "This paper introduces MMA, a large multi-language dataset for autoformalizing theorem statements. MMA employs a back-translation approach using GPT-4 to convert two formal corpora (AFP for Isabelle and mathlib for Lean4) into informal-formal pairs. Experiments demonstrate that LLMs such as LLaMA and Mistral, fine-tuned on MMA, achieve significant improvements over base models (from 0% to 29%-32% with no or minimal correction), outperforming their single-language counterparts in autoformalization on the miniF2F and ProofNet benchmarks.", "questions": "* In the discussion, the author mentions plans to expand to other languages like Coq. However, I believe that a significant portion of Coq projects focuses on software verification, which is often challenging for human readability. One success of informalization by GPT-4 is that the formalized concepts are easily understood by GPT-4. Do you think your approach could still be effective for such Coq projects?\n\n* (optional) If applicable, I am curious about the performance of DeepSeekMath or Llemma as the base model to be fine-tuned on MMA. These models have been pre-trained on large-scale informal and formal datasets and exhibit some basic autoformalization abilities.", "limitations": "I appreciate the detailed statement of the limitations in the paper. Moreover, I believe the most significant limitation is the presence of noisy aligned informal-formal pairs, which could be improved further.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is well-written and well-organized.\n* The proposed MMA dataset is one of the largest collections of multi-language informal-formal data pairs, addressing the notable data scarcity in autoformalization. This dataset could be somehow beneficial for future research in this field.\n* Experiments demonstrate that MMA is effective for fine-tuning LLMs for autoformalization, showing that multi-language data can enhance performance for single-language tasks.\n* I appreciate the manual evaluation and the detailed discussion provided in this paper.", "weaknesses": "* I think the main weakness of this paper is that MMA is constructed solely based on zero-shot prompting for informalization by GPT-4, resulting in a dataset that is not perfectly aligned. Table 2 shows that the formalization accuracy is around 75%, indicating significant room for improvement. While the choice of zero-shot prompting may be due to cost considerations given the large scale of the dataset, there are potential ways to improve quality. For example, filtering out obvious errors in informalized statements or using more advanced prompting techniques (e.g., few-shot, self-consistency, or prompt to refine the misunderstanding concept) could help construct a smaller but higher-quality dataset. I am also curious about the effectiveness of fine-tuning with a small high-quality dataset compared to a large but noisy dataset for autoformalization.\n\n* Minor: Missing references: There are some other works that leverage autoformalization to generate informal-formal data [1, 2]. [3] also surveys methods for autoformalization and theorem proving.\n\n[1] FIMO: A Challenge Formal Dataset for Automated Theorem Proving, arXiv preprint 2023\n\n[2] MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data, ICLR 2024\n\n[3] A Survey on Deep Learning for Theorem Proving, arXiv preprint 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback and the time they invested in reviewing our paper. We address specific points raised by the reviewer:\n\n> **I think the main weakness of this paper is that MMA is constructed solely based on zero-shot prompting for informalization by GPT-4, resulting in a dataset that is not perfectly aligned. Table 2 shows that the formalization accuracy is around 75%, indicating significant room for improvement. While the choice of zero-shot prompting may be due to cost considerations given the large scale of the dataset, there are potential ways to improve quality. For example, filtering out obvious errors in informalized statements or using more advanced prompting techniques (e.g., few-shot, self-consistency, or prompt to refine the misunderstanding concept) could help construct a smaller but higher-quality dataset. I am also curious about the effectiveness of fine-tuning with a small high-quality dataset compared to a large but noisy dataset for autoformalization.**\n\n- We thank the reviewer for suggesting the advanced prompting techniques and filtering methods for improving the quality of the MMA dataset. In the paper, we explain that we did not use naive few-shot prompting because the formal data comes from many diverse domains, while few-shot prompting helps the most when the in-context examples are in the same domain as the input (line 156-157). We acknowledge that potentially one could write representative examples in each major category and dynamically retrieve them accordingly. However, considering that there are many diverse mathematical domains present in AFP and mathlib4, we think this approach is very labour-intensive and seems infeasible given our budget and access to formal mathematical expertise.\n- For filtering, as we explain in the informalisation taxonomy section, the informal outputs (even the wrong ones) make sense at first sight and require significant expertise to spot the mistakes. We cannot think of good ways to filter out major mistakes in the dataset at the moment.\n -We gladly accept that there can be other methods to improve the alignment. As this is the first informalisation attempt at such a big scale, we are unsure how good the 75% accuracy is and hope for follow-up works to improve on it. The main reason for using a standard zero-shot prompting is that it provides the simplest methodology and as we are creating the dataset for the first time, it seems the most obvious choice. Improvements are welcome and can surely be made.\n- We would like to stress that the scope of this paper is to introduce a methodology and a dataset at a scale that has not been done before. More improvements are always possible, and this work will hopefully inspire and spark them. Based on these, we think the contributions made are significant enough.\n\n> **Minor: Missing references: There are some other works that leverage autoformalization to generate informal-formal data [1, 2]. [3] also surveys methods for autoformalization and theorem proving.\n[1] FIMO: A Challenge Formal Dataset for Automated Theorem Proving, arXiv preprint 2023\n[2] MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data, ICLR 2024\n[3] A Survey on Deep Learning for Theorem Proving, arXiv preprint 2024**\n\nWe thank the reviewer for pointing out these works. Indeed, the autoformalizaton with feedback approach in FIMO and the joint theorem-proof autoformalization approach in MUSTARD are very nice additions for our context. We will update the paper by citing them.\n\n> **In the discussion, the author mentions plans to expand to other languages like Coq. However, I believe that a significant portion of Coq projects focuses on software verification, which is often challenging for human readability. One success of informalization by GPT-4 is that the formalized concepts are easily understood by GPT-4. Do you think your approach could still be effective for such Coq projects?**\n\nWe think that the approach can still be effective for Coq projects. Our reason for this is that software verification often has a couple of important fixed themes like loop invariance, which allows for easier optimisation than mathematics. Furthermore, there are software and hardware verification theories in the AFP, which we did not find to be more difficult to informalise compared to mathemaitcal theories, from the informalisation taxonomy experiments.\n\n> **(optional) If applicable, I am curious about the performance of DeepSeekMath or Llemma as the base model to be fine-tuned on MMA. These models have been pre-trained on large-scale informal and formal datasets and exhibit some basic autoformalization abilities.**\n\nWe thank the reviewer for pointing this out. Indeed DeepSeekMath, Llemma, or the recent Mathstral (continuous pretraining and fine-tuning based on Mistral 7B, so can be directly compared to Mistral 7B in the paper) can be very interesting subjects of study. As these models were not available when we started the paper, we did not venture to use them. We will update the paper to say that these additional training might make them better candidates for autoformalization models.\n\nWe would like to thank the reviewer for their valuable feedback, which has greatly helped us improve our paper. Given the improvements we've made, we kindly request the reviewer to reconsider their score, or give indication for further improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all reviewers for their keen input, which has significantly raised the quality of this paper. You will find a one-page attachment containing two plots we use to demonstrate our points.

Below, we'd like to address a few common points:

1. "Is the evaluation dataset size (100 per language) enough?" (reviewers zhuU and ByiA)
   - In the pdf attachment, Figures 1 and 2 show the distribution of correction difficulty for the jointly fine-tuned Mistral model for 50, 75, and 100 samples respectively. We can see that the distribution barely changes as the number of samples is increased.
   - This demonstrates empirically that the variance present in the 100 samples is small enough, and that adding more samples will not change the conclusion.
2. "Does advanced prompting work better?" (reviewers ByiA and 8WKp)
   - We did not use naive few-shot prompting because the formal data comes from many diverse domains, while few-shot prompting helps the most when the in-context examples are in the same domain as the input (line 156-157). Considering that there are many diverse mathematical domains present in AFP and mathlib4, we think few-shot prompting is very labour-intensive and seems infeasible given our budget and access to formal mathematical expertise.
   - We construct MMA as a dataset of this scale as a first step towards unlocking better autoformalization capabilities through fine-tuning. We want MMA to inspire and encourage further improvements, through better prompting techniques, while using our methodology.
   - We believe that our contributions are beyond the dataset itself, and warrant a publication.

Given the improvements and explanations we've made, we hope we can convince the reviewers to reconsider their score, or give indication for further improvements.

Meta-review: This paper introduces MMA, a large-scale multi-language dataset for autoformalization, created using a back-translation approach with GPT-4. The dataset's quality and diversity enable improved performance in autoformalization tasks, particularly when training on multiple formal languages. The paper's strengths lie in its well-written presentation, valuable dataset contribution, and insightful experiments. However, there's room for improvement as suggested from reviewers (e.g., addressing the noise introduced by zero-shot prompting.) Overall, the paper provides a significant resource and insights for advancing the field of autoformalization.
Decision: Accept (poster)