Lyra: Orchestrating Dual Correction in Automated Theorem Proving
Authors: Chuanyang Zheng
Haiming Wang
Enze Xie
Zhengying Liu
Jiankai Sun
Huajian Xin
Jianhao Shen
Zhenguo Li
Yu Li
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) present an intriguing avenue for exploration in the field of formal theorem proving. Nevertheless, their full potential, particularly concerning the mitigation of hallucinations and refinement through prover error messages, remains an area that has yet to be thoroughly investigated. To enhance the effectiveness of LLMs in the field, we introduce the Lyra, a new framework that employs two distinct correction mechanisms: Tool Correction (TC) and Conjecture Correction (CC). To implement Tool Correction in the post-processing of formal proofs, we leverage prior knowledge to utilize predefined prover tools (e.g., Sledgehammer) for guiding the replacement of incorrect tools. Tool Correction significantly contributes to mitigating hallucinations, thereby improving the overall accuracy of the proof. In addition, we introduce Conjecture Correction, an error feedback mechanism designed to interact with prover to refine formal proof conjectures with prover error messages. Compared to the previous refinement framework, the proposed Conjecture Correction refines generation with instruction but does not collect paired (generation, error & refinement) prompts. Our method has achieved state-of-the-art (SOTA) performance on both miniF2F validation (48.0% → 55.3%) and test (45.5% → 51.2%). We also present 3 IMO problems solved by Lyra. We believe Tool Correction (post-process for hallucination mitigation) and Conjecture Correction (subgoal adjustment from interaction with the environment) could provide a promising avenue for future research in this field.

Published in Transactions on Machine Learning Research (07/2024)
Lyra: Orchestrating Dual Correction in Automated Theorem Proving
Chuanyang Zheng The Chinese University of Hong Kong
edu.hk
Haiming Wang Sun Yat-Sen University
wanghmm:99@mmaile.yss.......
Enze Xie
aie. rik.ennae@haaees.com
Huawei Nooh's Ark Lab
Zhengying Liu
liczhengyingg@@hawvee.com
Huawei Noah's Ark Lab
Jiankai Sun The Chinese University of Hong Kong
Huajian Xin Sun Yat-Sen University
mulijemail2..sssu.euu..o
Jianhao Shen
shenjianhao22@huavei.com
Huawei Nooh's 's Ark Lab
Zhenguo Li Huawei Noah 's Ark Lab
Li:Zhenggu@huuwee..oo
Yu Li
FguuCcse.cuhe edu.hk
The Chinese University of Hong Kong
Reviewed on OpenReview: https: // openreview. net/ forum? id- Sut 75ko tss
Abstract
Large Language Models (LLMs) present an intriguing avenue for exploration in the field of formal theorem proving. Nevertheless, their full potential, particularly concerning the mitigation of hallucinations and refinement through prover error messages, remains an area that has yet to be thoroughly investigated. To enhance the effectiveness of LLMs in the field, we introduce the Lyra, new framework that employs two distinct correction mechanisms: Tool Correction (TC) and Conjecture Correction (CC). To implement Tool Correction in the post-processing of formal proofs, we leverage prior knowledge to utilize predefined prover tools (the tool is used to prove the connection. The tool can be by simp, Sledgehammer and so on) for guiding the replacement of incorrect tools. Tool Correction significantly contributes to mitigating hallucinations, thereby improving the overall accuracy of the proof. In addition, we introduce Conjecture Correction, error feedback mechanism designed to interact with messages. Compared to the
prover to refine formal proof conjectures with prover previous refinement framework, the proposed Conjecture Correction refines generation with instruction but does not. collect paired (generation, error & refinement) prompts. Therefore, DSP DSP et al. 2023a +Tool Correction + Conjecture Correction = Lyra. Our method has achieved statc-of-the-art (SOTA) performance on both miniF2F validation (48.0% 55.3%) and test (45.5% 51.2 51.2%). Using GPT-4, our Lyra enhances the bascline DSP, achieving improvements the miniF2F validation from 50.4% to 55.3%, and the test from 42.6% to

---

Published in Transactions on Machine Learning Research (07/2024)
51.2%. We also present 3 IMO (International Mathematical Olympiad) problems solved by Lyra. We believe Tool Correction (post-process for hallucination mitigation) and Conjecture Correction (subgoal adjustment from interaction with the environment) could provide a promising avenue for future research in this field.
1
Introduction
Formal proof automation is challenging task that has garnered increased attention in recent years (Bansal
et al.
2019a
Polu & Sutskever] 2020 Lample et al. 2022 Jiang et al. 2022 Wu et al.) 2022 Wang al.
2023c
Unlike other domains where deep learning approaches have shown remarkable success, previous si studss
have
proposed techniques to synt hesize additional formal training data (Wu et al.) 2022 Polu & Sutskever
2020
Han ct al., 2021 Bansal et al.
2019b Polu et al. 2023 Recently, large language models (LLMs)
trained on informal mathematical data have showcased impressive quantitative reasoning abilities Lewkkwycz
et al.
2022 Welleck al., 2022).
Draft, Sketch, and Prove (DSP) (Jiang al.) 2023a] maps informal proofs to formal proof sketches, and uses the sketches to guide an automaled prover by directing its search to easier sub-problems. Following this direction, Subgoal-based Learning (Zhao et al. 2023) replaces the informal proof with subgoal-proof and learns how to optimize subgoal demonstration selection. However, they have not been able to post-process LLM generation or gradually refine previous generations.
In this paper, our goal to develop Lyra, a system that integrates Large Language Models (LLMs) enhance the capabilities of formal theorem proving. There are two major challenges for LLM generation: 1) hallucination mitigation; 2) interaction with the environment. To mitigate LLM hallucination, we propose Tool Correction to leverage prior knowledge and rules to guide incorrect tool replacement. As shown in the observation in Figure Tool Correction, prover fails to prove conjecture = 19 * (x div 19) + 4 because by (simp add: div_mult mult toiittmod_eeee mod__q) generated by LLM cannot prove I = 19 * (xr div 19) + 4 (considered as LLM hallucination), while the conjecture is correct but employed tool simp is not powerful enough. The "simp" the name of one tool. The expression by (simp add: div_mult mult_mmdd__________________m____ _eq) refers to a directive in interactive theorem proving, which suggests an altempl lo simplify the current goal statement using the theorem div mult, mod eq, which usually relates to the relationship between division, multiplication, and modulo operations. Tool Correction employs predefined tools (e.g. sledgehammer, arith) to guide incorrect tool replacement and finally prove the conjecture. We also propose general interaction technique with LLM named Conjecture Corvection. To further improve and modify the conjectures, Conjecture Correction leverages a general framework that can casily integrate foodback from any environment, in this case, the Isabelle prover, to further polish conjectures. We believe the Lyra presents our insights to mitigate LLM hallucination and interact with the environment.
The proposed method significantly outperforms competing approaches in formal theorem-proving tasks, achieving pass rale of 51.2% on the miniF2F test dataset, 5.7% absolute improvement over the previous state-of-the-ans Furthermore, the insights gained from Tool Correction and Conjecture Correction design can be applied to other frameworks that need to interact with the environment. In summary, our contributions are as follows:
We introduce Lyra, a method composed of two components Tool Correction and Conjecture Correction, to guide automated provers with formal proof sketches
Tool Correction employs the predefined tools to replace the incorrect tools to mitigate hallucination, while Conjecture Correction integrates previous formal sketch and prover error messages to refine proof.
We establish new SOTA of 55.3% and 51.2% on miniF2F validation and test, outperform previous best. 7.3% and 5.7% respectively. With GPT-4, our Lyra improves the baseline DSP on both mF validation (50.4% + 55.3%) and test (42.6% + 51.2%). And we newly solve two IMO problems: #K0_1974_p5 p5 and IMO_1981_pp
2

---

Published in Transactions on Machine Learning Research (07 (204)
Formal Proof:
Formal Proof:
Formal Proof:
theozun mathd waggsbrr 275:
thaozam motidd algabara 273:
thecrem nechd algebre, 2755c
Eixees 1: real
fixeea x 1: ceal
fixes x 1: zeal
assumes [1l posr (1 / 432 powe (3
azanames *(ll pome (1 / 411 powe (3
assumes "(22_ pOME (1 411 poiz (3
3)
31 1 5"
31 1
shows "111 posr (1 / 4:1 poie
shows (11 poMe
91) pour 16
shows (11 soME (1 / 4)1 poss
x 2) 121 / 25"
x+ 2) 121 25*
x 21 - 121 / 25*
proof
proot
proof
have (cr-)) 4-
"2m 4--41
"In-21
by Isimg edd: (reuuae
by Istmp add power en arinnc
by 205
god
qud
Incorrect Tool
qed
Correct Tool
Prover Validate
Tool Correction
Formal Proof:
theorem aine 1983__98__pppp
Formal Proof:
theoram a.me F9:
fixes x::real assumes "Dox" "xep1"
fixons x: :real
assumes "Ooxe" "xopp""
shows Vol (19
(sir '233
shows *12 <<le> [19 (x'2
(sin x)
(x rin x)
(x sin x)"
Prover Validate
proof
proof
The La to the fact that trying show equality
Let y ain x.
rewrite the
expression ae 2y'+1 By + define y where "y" x + sin x*
between two expressiioss when they are not
y
necessarily equal.
Error Tactic: (auto
should instead be showing that one is lees than or equal to the other. +)
with "Dex" "xcps" have "y 0" unfolding y_
simp:
(auto simp: pilt_pos_pps sin_gt__rero
power2_ _e________ppp [q_squrre)
(+ Let y -.nn.. We can zewzite the expression
Error Message
9/--1 = +l
Since > 0, and s becsuse
Summary.
as "
D < <,
tactic_ failed
define y where "y =xx sin xt
No have y>>00 D. So can apply [AM--M ]]:
Error Message Detail:
with "D=x"" "xeqpi" have "y > 0" untolding y_def
by simp: milt_poc_ pos atn_st_zzrr)
Failed to apply proof
(* Since > 0, and sili 0 because <
By 2 2//9 J
method using this: 0<
we have y 2- 0.
apply [[MM-MM]::
goal (1 subgoal): 1.9
have "12 kdl 9 y using 'y> 0-
y
yd
[x sinx) x) +4/ (x*
By + 2. 1.. 12
by (aulo simp: real LogL Ler icr adi sii__dvideddistaee wtw.ddissstiissmeeeetti an weightes)
xi== (9 (x* sin
have *12 Vel 9 xxy. 1/77
x/\<>sup>2 + 4) (x"
using yy by (oato zimp: roal eqpt lo iff add dliiida dlsttcillayymattiill weighted)
sin x) At command "by"
The equality holds when
By = y² = +
equality holds
Therefore, the minimam value is 012. This is reached when we have siin = in the original son
By <= 3" { <=>
Conjecture Correction
Therefore, the miniman value is 12.
This is reached when we have of stn -3 the
Diso have "(adonss a 119 + (x iain x3 untoldingg def using y 0" by (auto simp:
1'2)
CrojeectCoonecuuur
also have "yaiot 119 1x"2
original ecnuatiion..
X)1211
4 / Ix sin xi" unfolding y. def using y U" by lauto simps
[ower2__g__ruuarr
finally show 2thesis by auto
Correct Conjecture
add www_aaviiieeitttteee
power2_eg__agnae
ged
finelly show Kheels by auto
qed
Figure 1: Our proposed Lyra framework contains two modules. Tool Correction: employ the predefined tools to replace the incorrect tools and prove the conjectures. The prover fails because by (simp add: div_mult moult_mmd_e___ _eq) generated by LLM cannot prove x = 19 * (a div 19) + 4, which considered as the hallucination. Actually, the conjecture is correct and simple, and the prover fails to prove it because it employs an incorrect tool. Hence, the prover successfully proves the conjecture when employing by sos. Conjecture Correction: We design an interaction framework that integrates previous formal sketch and prover error messages to better sketch generation. The steps with the ATPWit delimiters are generated by an automated prover with Tool Correction.
2 Related Works
Interactive theorem provers. Contemporary mathematical verification systems are centered on interactive theorem provers (ITPs), including Isabelle (Paulson 1994). Lean (de Moura et al. 2015). Coq [Barras et all 1997p, and Metamath (Megill & Wheeler 2019). ITPs lay the groundwork for mathematical definitions and theorems on a robust logical foundation through their core kernels. The validation of each theorem is semellbased and takes place within the ITP. To achieve formal proof, a theorem is initially expressed in the programming language of the ITP and systematically refined into simpler subgoals until it aligns with previously established facts. In this paper, the chosen ATP is Isabelle, known for its potent prover tools, including sledgehammer (Paulson 2010
3

---

Published in Transactions on Machine Learning Research (07/2024)
Machine learning for formal proving. Numerous approaches advocate the integration of machine learning
with contemporary interactive theorem provers (TTPs) YYang & Deng, 2019 Gauthier et al. 2021 They
leverage the recent advancements in language models (Polu & Sutskever 2020 Han et al. 2021 Polu et al.
2023 Jiang et al., 2022, Lample et al. 2022: Mikula et al.) 2023). These techniques recommend actions based
on the current proving state, and the tree search identilies a sequence of correct sleps using act actio provided
by the language model. Potent methods like MCTS (Silver et al., 2018 Wu et al. 2021 Laurent & Platzer or dynamic-tree MCTS (Wang et al., 2023ch are utilized for this purpose. Previous work (Wu al,
2022
2022 has demonstrated the few-shot statement autoformalization capability of LLMs (Chowdhery et al.
2022 In investigating these findings' applicability to proof autoformalization, DSP (Jiang et al.) 2023a
conducted an in-depth analysis using Draft, Sketch, and Proof. Subgoal-Learning (Zhao et al. 2023 further employs a hubbool-ggol-basee
informal proof approach. In an effort to support the open-source communily,
LeanDojo Yangg et al. 2023) created a Lean playground that includes toolkits, data, models, and benchmarks.
While these methods directly use the results generated by LLMs, we adopt a different approach by employing predefined tools to post-process the generations mitigate hallucination, specifically Tool Correction.
Large language model refinement. Calibration studies conducted on LLMs reveal that the probabilistic predictions made by current LLMs are closcly aligned with the actual frequencies of token occurrences. resulting in well-calibrated predictions for specific tasks (Guo et al. 2017 Kadavath et al. 2022 Jiang
et al. 2020 As LLMs exhibit reliable calibration, an inereasing number of research studies emphasize using self-evaluation for verification. For instance, Reflexion (Shinn et al.) 2023) leverages an agent with dynamic
memory and self-reflection capabilities, while Self-Refine (Madaan et al.) 2023 proposes method to generate outputs from LLMs and refine their previously generated outputs based on their own feedback. Taking similar approach, methods like Self-Debug (Chen et al. 2023 and CRITICS (Gou et al. 2023) interact with code interpreters to further debug. The Baldur (First et al. 2023) trains a model to correct and refine the generated proof. In contrast, Frogressive-Hing Prompting (Zheng et al. 2023) iteratively extracts hints from previous LLM's answers as hints for the next answer generation. However, previous works require extensive prompts, including generation prompts and refine prompts. Our approach Conjecture Correction refines generation with instruction but does not collect paired (generation, error & refinement) prompts.
3 Method
This section describes our Lyra for formal proof automation, which leverages Tool Correction and Conjecture Corvection to guide automated formal theorem provers with proof sketches.
3.1 Background: Pipeline of DSP
DSP (Jang et al. 2023a] aims to generate a formal sketch from an informal statement, verifiable by an off- the-shelf automated theorem prover. DSP creates N demonstration examples, denoted as E = E1, Eg, EN, each containing informal/forman components (statements, informal proofs, formal proof). The statement is the problem description, the informal proof is the informal solution, and the formal proof is the proof via formal language. The pipeline of DSP has the following three steps.
Informal proof generation. The informal proof generation utilizes in-context learning. DSP provides the model with a few examples containing both (statement, and informal proof) for informal proof generation. Subsequently, DSP presents a problem statement that needs to be translated and the model then generates the subsequent tokens to produce the desired informal proof.
Formal proof generation. The formal proof generation also utilizes in-context learning. DSP leverages the few-shot learning capabilities of a large language model. Specifically, DSP provides the model with a few example pairs containing (statement, informal proof, formal proof) for a formal proof generation. Subsequently, DSP presents a (statement, informal proof) that needs to be translated. The model then generates the subsequent tokens to produce the desired formal sketch.
Two Formal Proof Generation Scenarios. Given There are two scenarios: one with an existing human informal proof and another where language model generates draft-proof candidates without a human
4

---

Published in Transactions on Machine Learning Research (07/2024)
reference. For LLM-generated informal proof, should first generate informal proof, and then utilize the generated informal proof to generate formal proof.
Prover validation. In the final phase, off-the-shelf automated provers address sketch gaps. These systems create formally valid proofs. DSP framework remains agnostic to prover type (symbolic, neural, hybrid). Successful prover results yield verifiable formal proofs.
3.2 Tool Correction
Tool Correction employs prior knowledge employ predefined tools (e.g. sledgchammer) to guide incorrect tool replacement, as shown in Algorithm | We introduce the Tool Correction as remedy lo alleviate the generation errors stemming from Large Language Models (LLMs). Through empirical observation, it becomes evident that despite the factual accuracy of conjectures, LLMs at times adopt misguided tools that do not withstand validation by theorem provers, as shown in Figure] [
For instance, consider the statement the tactic by (simp add: div_mult dtv_mult_mod_eqq] leading to failure. This is the LLM hallucination, by (simp add: div_mun mult_mod_ees is suited for proving Substituting it with by sos enables the theorem prover to successfully verify x x= 19 * (æ div 19) + 4. Hence, in certain instances, LLM might formulate correct conjectures but employ inappropriate tools, resulting in unsuccessful proof attempts. To address this, Tool Correction leverages predefined tools to enhance the success rate.
19 (x die 19) + 4, where LLM proposes utilize
div b b+a mod but not 19*(x div +9)
The Tool Correction approach entails the validation of given tactic t using Isabelle. If validation succeeds, proceed; if not, Tool Correction intervenes to modify the tactic. Specifically, when tactic is equal to " commencing with "by' "sledgehammer" but the tactic fails, we attempt the application of tooo This toal can be cither: 1) "sledgehammer" or; 2) by tool with tool belonging to the set (auto, simp, blast, fastforce, force, eval, presburger, sos, arith, linarith, auto simp: field simps).
By integrating Tool Correction, we systematicallyy explore the applicability of "sledgehammer" and 11 heuristic tools. If any of these successfully pass the thcorem prover, we progress to the subsequent tactics. However, if proof still fails to prove the lactic after trying all leot fail, the overall proof attempl is deemed unsuccessful.
3.3 Conjecture Correction
Theorem Prover
Prover Validation
Prover FeedBack
Formal Preof:
proof
thecrun wet.hhLalpppboss..000
Initial Round
have az (0=21 "2ia" 22aa"22uun 22-20-"" by add: power2_espppeeeee
disoe real
accumes (11 powr 101 powe 13
Repeat Four Times
x 1) 1/25 3"
sbows (11 poir (1 / 40 posr 16
+ 2) 121
ged
Refine
Proot
Feedback
fella after four times, then regenerate the
initial proof without prover
Figure The figure illustration Conjecture Correction. First, we generate the formal proof without the prover feedback (just like DSP formal proof generation. Then, validate the proof to get the feedback and utilize the feedback to refine previous formal proof. we fail to successfully refine the previous proof four times, then we go back to the initial round, which generates formal proof without feedback. In work, shown in Figure 1 the feedback of prover contains: 1) error tactic; 2) error message summary and 3) error message detail.
For Conjecture Correction, design framework that can easily integrate previous formal sketches and error messages from the prover to improve sketch generation. LLMs, particularly GPT-4, can leverage prior responses or contextual cues for improved output. Nonetheless, integrating feedback into mathematical proving remains a challenge. This stems from two primary factors: 1) diverse theorem provers employ distinct syntax, complicating the design of varied prompts; 2) often require an extensive token count, incurring high computational cost and exceeding model length limits. To address these limitations, Lyra uses Conjecture Correction, offering versatile refinement, pipeline that can transform non-refined framework
5

---

Published in Transactions on Machine Learning Research (07/2024)
into refined one. Compared to the previous refinement framework, such Self-Refing Maadaan et al. 2023 or Self-Debug (Chen et al. 2023), the proposed Conjecture Correction refines generation with instruction, but does not collect paired (generation, error & refinement) prompts. The details are shown in Algorithm 2 and Figure 2
Initial round generation. the initial round generation, we follow the same process as DSP, directly producing informal or formal proofs without prover error messages.
Rectification round. Our approach also involves the use of an LLM (e.g. GPT-4 (Bubeck et al. 2023)) for rectification. In contrast to the initial round generation, the rectification employs the same initial prompt the first round but appends error messages from the prover. As most formal proofs of our prompts begin with proof we add proof - at the end of the instruction so that the LLM response is formal proof.
Reset initial round generation. Since the rectification round builds upon the first round, the quality of Conjecture Correction performance is tied the initial round proof. To ensure that a potentially subpar initial round proof does not negalively affect subsequent proofs, we regenerate the initial round proof at interaction rounds K, 2K, 3K, and so on, refining its generation in the remaining rounds. For example, when working with 200 attempts and sctting K to 5, Conjecture Correction partitions the 200 attempts into 40 patches. Each patch consists of the first proof derived from DSP, followed by four subsequent refined proofs that build upon the previous proof and incorporate the error message provided by the prover.
4 Experiment
4.1 Dataset
In this study, we assess our approach using the miniF2F dataset (Zheng et al. 2021), which is a collection of 488 formal mathematical problems derived from high-school competitions and expressed in three formal languages: Lean (de Moura et al., 2015), HOL-Light (Bansal et al. 2019a), and Isabelle (Paulson 1994). The dataset is divided into validation and test sets, cach containing 244 problems. These problems are sourced from three distinct categories, with 260 problems extracted from the MATH dataset (Hendrycks et al. 2021) ) 160 problems taken from actual high school mathematical competitions (AMC, AIME, and IMO), and 68 problems specially crafted to mirror the difficulty level of the aforementioned competitions.
Evaluation. The objective of our study is to generate formal sketches for the problems in the minil2F dataset. We consider proof valid if and only if (a) it does not have any 'cheating" keywords (sorry and oops) that terminate a proof without completion, and (b) Isabelle must be capable of verifying the corresponding formal statement with the proof.
Implementation details. In our research, we utilized GPT-4 as the Language Model Model (LLM) for generating informal drafts and formal sketches. The temperature of GPT-4 was set to 0.7, with 200 attempts. The details of the baselines are shown in the Appendix.
4.2 Baseline
To evaluate the effectiveness of our proposed methodology, we employed several baseline methods, which follow previous work setting (Jiang et al. 2023a, Zhao et al., 2023
Sledgehammer with heuristics The first bascline Sledgchammer (Paulson 2010), which a proof automation tool in the Isabelle environment. Additionally, we utilized Sledgehammer supplemented with heuristics, which integrates 11 prevalent tactics (i.e., auto, simp, blast, fastforce, force, evapl, presburger, sos, arith, linarith, auto simp: field simps) with Sledgehammer. If all tactics fail, the system employs Sledgehammer (Jiang et al. 2023a). The Magnushammer employs contrastive learning and transformer to retrieval relevant premises.
Language model based methods Thor (Jiang et al., 2022) combines language models with automatic theorem provers to help select premises from a vast library. +eoreexpert (Wu et al., (022) iteration enhances neural theorem prover by training it theorems that have been successfully formalized. Draft, Sketch,
6

---

Published in Transactions on Machine Learning Research (07/2024)
Table 1: Proving success rates on the miniF2F dataset with Isabelle. The table displays the success rates of previous works and the Lyra, using both human and GPT-4 informal proofs. The highest success rates for each set. are highlighted bold.
Success rate
miniF2F-van -alldd
miniF2F-test
Baselines
Sledgehammeer [Paulson
2010
9.9%
10.4%
Sledgehammner heurislics
Jiang al. 2023a
18.0%
20.9%
Thor (Jiang al. 2022
28.3%
29.9%
Thor + expert iteration Wu al.
2022
37.3%
35.2%
Magnushammer Mikula et al 2023
43.9%
39.3%
Draft, Sketch, and Prove (100 attempts)
Jiang al.
2023a
Human informal proof
42.6%
39.3%
540B Minerva informal proof
42.6%
38.9%
(100 attempts) (Zhao al.
2023
48.0%
45.5%
LEGO-Prover (100 attempts) (Wang al.
2023b
Model informal proof
52.4%
45.5%
Human informal proof
55.3%
50.0%
Lyra (Ours)
GPT-4 informal proof (100 attempts)
52.8%
44.2%
GPT-4 informal proof (200 attempts)
54.9%
47.9%
Human informal proof (100 attempts)
52.0%
47.1%
Human informal proof (200 attempts)
55.5 55.3%
51.2%
and Prove (DSP) (Jiang et al., 2023a transforms informal proofs into formal sketches and utilizes these formal sketches to guide an automated prover. Another LLM-based method is Subgoal-Proof Learning (Zhao et al. )023), which incorporates subgoal proof to replace informal proof and proposes a prompt selection framework. Finally, we also compare our method with the LEGO-Prover Wang et al. (2023b).
2023 excluded representative methods such Following previous work (Jiang al. 2023a Zhao et al.
Hyper'Tree Proof Search (HTPS) (Lample et al.] 2022 and GPT-f with expert iteration (Polu et al., 2023), which are implemented using Lean de Moura et al. 2015 different interactive theorem prover. The disparity in tactics and automation between Lean and Isabelle renders them not directly comparable to our method.
4.3 Main Results
Table I presents the distribution of successful formal proofs obtained from the miniF2F dataset using the interactive theorem prover Isabelle. An examination of the results presented in Table [ reveals a conspicuous enhancement in the efficacy of the Sledgehammer automated prover, owing to the integration of 11 supplementary heuristic tactics (Jiang et al.) 2023a).. Noteworthy achievements are also realized through deploying the DSP-based methods (DSP and Subgoal), attaining success rates of 39.3% and 45.5%, respectively on the miniF2F test set.
By harnessing informal proofs generated by GPT-4, our proposed method achieves success rates of 54.9% and 47.9% on the validation and test sets of miniF2F respectively. This performance persists even when the attempt number is set at 100, affirming its robustness. When the attempt number is 100, compared to 540B Minerva informal proof with DSP, our proposed Lyra improves the performance on miniF2F validation set from 42.6% to 52.8% and miniF2F test set from 38.9% to 44.2%. This outcome can be attributed to the Tool Correction and Conjecture Correction.
In instances where human informal proofs are employed, our proposed method demonstrates impressive success rates of 55.3% and 51.2% the validation and test sets of miniF2F. Comparative analysis against.
7

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Table 2: Ablation results on the miniF2F dataset with Isabelle. There are three important conclusions: 1)
GPT-4 is better than Codex for mathematical proving; 2) Tool Correction can consistently improve performance; 3)
Conjecture Correction can improve performance but needs more attempss. Our proposed method degrades to DSP (Jiang et al. 2023a when without Tool Correction and Conjecture Correction.
Attempt
Formal Proof
Informal Proof
TC
CC
minil2F-vvliid
miniF2F-test
Codex
540B Minerva
42.6%
38.9%
GPT-4
GPT-4
X
48.3%
38.9%
100
Codex
Human
X
X
42.6%
39.3%
GPT-4
Human
47.9%
39.7%
GPT-4
GPT-4
52.8%
44.2%
GPTT4
Human
/
52.0%
47.1%
GPT-4
GPT-4
49.5%
40.9%
GPT-4
GPT-4
55.3%
45.0%
GPT-4
GPT-4
48.3%
40.9%
GPT-4
GPT-4
54.9%
47.9%
200
GPT-4
Human
X
50.4%
42.6%
GPT-4
Human
52.8%
45.9%
GPT-4
Human
X
46.7%
43.0%
GPT-4
Human
55.3%
51.2%
DSP reveals improvement. of 12.7% and 11.9% the validation and test sets respectively for miniF2F. Furthermore, when contrasted with the previous state-of-the-art Subgoal-Leariing model, approach showcases advancement of 7.3% and 5.7% on the miniF2F validation and test sets respectively. Though LEGO-Prove et al. (2023b) achieves relatively good performance, we would like to highlight that LEGO-Prover needs more computation resources to build skill library.
The performance human informal proofs surpasses that of GPT-4 generated counterparts, especially on the test set. This substantiates the notion that precision in informal proofs important for generating formal sketches.
4.4 Ablation Study
GPT-4 is better than Codex, especially on miniF2F validation dataset.. In the absence of Tool Correction and Conjecture Correction, our proposed method experiences degradation to DSP. Referring to Table when considering the informal proof generated by LLM (GPT-4 or 540B Minerva), GPT-4 is better than Codex (Chen et al., 2021).. When compared with the deployment of Codex for generating formal sketches, GPT-4 demonstrates improvements of 5.3% and 0.4% the validation and test subsets of miniF2F, respectively, while utilizing the same attempt number 100 and human informal proof. This substantiates the notion that GPT-4 indeed enhances performance.
Tool Correction:
consistently improve performance. As evident from Table 2 and Figure 3 the inclusion of Tool Correction yields enhanced performance. Similarly, when asscssing -PP--4-eeerated informal proofs on the miniF2F test. set, Tool Correction elicits improvements of 4.1% and 7.0% in the absence and presence of Conjecture Correction, respectively. When considering human informal proofs on the miniF2F test set, Tool Correction showcases enhancements of 3.3% and 8.2% in scenarios devoid of and accompanied by Conjecture Correction, respectively. Therefore, regardless of whether the informal sketch is generated by GPT-4 or created manually by human, Tool Correction consistently enhances performance and can further benefit from the addition of Conjecture Correction.
Conjecture Correction: further improves performance, prefers more powerful prover and requires more attempts be convergent. The outcomes presented in Tablegaan illustrated Figure 3 underscore the efficacy of integrating Conjecture Correction, albeit at the expense of requiring an increased number of attempts to achieve convergence. When considering human informal proofs the miniF2F test.

---

Published in Transactions on Machine Learning Research (07/202) 2024)
miniF2F-valid (Human Informal)
miniF2F-test (Human Informal)
120
#Successful Proofs
120
#Successful Proofs
100
100
80
80
w/ TC & w/ CC
w/ w/ CC
60
W/ TC & w/o CC
w/ TC & w/o CC
60
w/o TC w/ CC
w/o TC & w/ CC
40
w/o TC w/o CC
w/o TC & w/o CC
40
50
100
150
200
0
50
100
150
200
# #Autoformalization Attempts Per Problem
# #Autoformalization Attempts Per Problem
miniF2F-valid (GPT-4 Informal)
miniF2F-test (GPT-4 Informal)
140
120
#Successful Proofs
120
#Successful Proofs
100
100
80
w/ TC & w/ CC
60
w/ & w/ CC
60
w/ TC & w/o CC
w/ TC w/o CC
w/o TC & W/ CC
w/o TC & w/ CC
40
W/o TC w/o CC
w/o TC & w/o CC
50
100
150
200
0
50
100
150
200
# #Autoformaization Attempts Per Problem
# #Autoformalization Attempts Per Problem
Figure 3: Number of problems solved miniF2F against the number of autoformalization attempts per problem. On miniF2F validation and test set, we have shown the results of Tool Correction (TC) and Conjecture Corvection (CC) human informal proof and GPT-4 informal proof respectively.
set, Conjecture Correction showcases enhancements of 0.4% and 5.3% in scenarios devoid of and accompanied by Tool Corvection, respectively. This suggests that Conjecture Correction improves proof quality, but needs more powerful prover (e.g. with Tool Correction) to fill the formal gaps. Conjecture Correction needs more attempts to be convergent because Conjecture Correction modifies the initial proof to generate subsequent proofs, which strongly hinges the quality of the initial proof. Specifically, Conjecture Correction partitions the pool of 200 altempts into 40 patches, wherein the first proof originates from DSP, and the subsequent four are based on the initial proof. Furthermore, it's worth noting that, in theory, any problems solvable through DSP remain solvable using our approach, as DSP is equivalent to our initial proof generation without Tool Correction.
Attempt number: Lyra benefits more with attempt number increment. In the absence of Tool Correction and Conjecture Correction, our proposed method reduces to DSP. Within the validation set with human informal proofs, when the number of attempts is escalated from 100 200 (shown in Table 2), the performance of DSP experiences a gain from 47.9% to 50.4%, achieving a 2.5% improvement. Conversely, our proposed approach exhibits a performance improvement from 52.0% to 55.3%, reflecting a more substantial 3.3% enhancement. For the test set, DSP's performance improves from 39.7% to 42.6%, marking a 2.9% increment. In contrast, our method demonstrates an increment from 47.1% 51. 51.2%, indicating a more 4.1% boost. This divergence implics that our proposod approach effeetively surpasscs the performance limitations of DSP, highlighting the polential efficacy of expanding the atlempt number to further enhance performance differences.
The performance of Conjecture Correction under smaller attempt number and no Tool Correc- tion. Initially, a proof is created with feedback from a prover, and Conject Correction is applied up to four times to refine this proof. However, the initial proof of poor quality, further refinement may be ineffective, particularly for difficult problems that require such refinement. While Conjecture Correction can enhance problem-solving for harder issues, its effectivencss diminishes with fewer autoformalization attempts and is compromised when Tool Correction is disabled. It operates by analyzing and correcting errors in
9

---

Published in Transactions on Machine Learning Research (07/2024)
Statement:
Determine possible values of bbb alble bleld numbers.
where a, c, arbitrary positive
Informal Proof (Human)):
Note that 2 = a-b a48 c+d ctd > atbto-d atd-eta + atbiota atbiota = 1. We will now prove that $ can reach any range in between 1 and 2.
Choose any positive number a. some variables such that m. > and & +1 let b ak, am, Piugging this back into the original fraction, we S arukkaa atuktam unFonetut aTeneeut 1-7 + IFFF
al.
E+m+1 1+m=7 The above equation can be further simplified S 2-m z + s-r- Note that
function and that f(m) = m z-m is strictly increasing function. We can now decrease and to make m tend ar arbitrarily close to 1. We see limm-1 + z-m = 2, meaning S can be brought arbitrarily close to 2. Now, set d x and b = for some positive real numbers
Then 9 204y
2n 2v +2v--2="
2v 1 2222 Notice that if we treat the numerator and
zyta a(x)
denominator each quadratic in y. will
by"= where g(x) has degree lower than 2. This taking -im 1,2 which means can be brought arbitrarily close 1. Therefore, done.
Formal Proof:
theoren imo_ 190_1994_p5:
assuzes "a>0" "b>0" "c>0" "d>0" fixes
real
ho (a=a/(a-bed) b/(abtt)) c/(bectd) d/aeo=))
shows *1<s \<and> =<2" proof
(+ Let's correct the proof.
Define the sun +)
let ?8 "a/(atbed) ba(abbtc) c/(biced) d/(a+ctd)"
Shou that +)
(arbbrrdd b/(abbrctd) c/(arbect) d/(abectdd)
using asans by (sinp add: (aridd__immm) [1TWWA]
noreover "78 (atbbrctd) b/abbectd) c/(atbecrd) (atbectd)*
proof
have "a/(arbrd) a/(atbect))" "b/(arbec) b/(atbteed)*
"c/(bee+d) c/(abbtcad)" "d/(aectd) d/(a+bectd)
using assma by (aimp_ail add: aovida_strice siigtssttrtttttttt__eeeeeeeeeett right_ nono) [ATPUR/TT]
then show Pthesis by [АРРWAT[] qed
ultimately have 78" by sinp (АТРИATO
Show that 3 2 (ainp add: divida_ simps) [17РWNT]]
hare "a/(atb) b/(atb) c/(ctd) d/(ctd) using
moreover have "75 a/(atb) b/(atb) c/(ord) </(d)))
proof
have "a/(a-bed) a/(at))" "b/(atbre) < (a+b)) "c/(beerd) c/(ed")" (arcrd) dctdd))
using aseas (simp_all adé: wwwiid_etrict_righteeeo then show ?thosis by auto [TTИANT qed
ult altizately have *78 < 2" by sinp [АТРИRTT
Conclude then show "I<a /<and> a<2" using asena ho "1 < 78" auto [АТРURNT
ged
end
Figure successful formal proof synthesized with human informal proof. With Tool Correction and Conjecture Correction, successfully solve au IMO problem IMO 1974_19..... The steps with the ATPWidhTC delimiters are generated by an automated prover with Tool Correction. We also solve IMD 110_1959_pp _pp with GPT-4 informal proof, which is shown in the Appendix.
factics and conjectures based on detailed feedback from the prover, potentially rectifying issues that DSP (the initial proof generation technique) cannot. The inclusion of Tool Correction, which addresses errors in the tools used by Conjecture Correction, can significantly improve performance by replacing incorrect tool tactics with correct ones, thus mitigating the drawbacks of Conjecture Correction.
4.5 Case Study
We solve another IMO problem IMO_1959, with GPT-4 informal proof, which is also solved via DSP with 540B Minerva. Furthermore, to present the effectiveness of our method, we provide formal sketch of an IMO problem named IM0_1974___ that remains unproven by earlier state-of-the-art methods. As demonstrated in
10

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Figure 4 our Lyra successfully proves IMD _10__994_p3 with Tool Correction and Conjecture Correction. We have shown the interaction details of 1M0_1974_pP and _p1 in the Appendix.
4.6 Further Analyze Tool Correction and Conjecture Correction
Number of fixed wrong steps by Tool Correction To further analyze the effectiveness of Tool Correction, we calculate how many fixed wrong steps by Tool Correction, compared to baseline sledgehammer + houristics.
Proof comes from: the miniF2F validation set (pass rate 55.3%) and test set result. (pass rate 51.2%), with GPT-4 human informal proof, Conjecture Correction and Tool Correction. The definition of proof step: a proof step is regarded as tactic. Calculation protocol: if sledgchammer- to validate the current tactic, then the current, proving process will be terminated and we will turn to the next formal proof validation. Finally, we calculate how many correct tactics/proof steps. On miniF2F-valid, bedgehammmer++huuissee
Tool Correction fails
can help the prover successfully pass 2260 steps. After adding Tool Correction, the number increases to 3486 steps. Therefore, Tool Correction fixes 1226 wrong steps. On miniF2F-tesst Tool Correction fixes 1293 wrong steps
Table 3: The number of wrong fixed steps by Tool Correction.
Dataset
Sledehammer heeritiss
Tool Correction
Number of Fixed Wrong Steps
Finil22Fvvlid
2260
3486
1226
FiniF2F-test
2594
3887
1293
The Tool Correction genoralization To prove the gencralization of Tool Correction, we combine DT- Solver (Wang et al. 2023c with Tool Correction on PISA dataset. (Jiang et al. 2021), which is used to evaluate previous famous works, such as Lisa (Jiang et al. 2021), Thor (Jiang et al. 2022) and Thor + expert. We do not employ Lyra on the PISA datasct becanse: similarly to the experiment setting of DSP and Sub-goal learning, the Lyra needs informal information, such as informal problem and informal proof. To the best of our knowledge, only miniF2F has an Tsabelle dataset that has informal information, while PISA does not. According the experiment results in Table we can observe that Tool Correction significantly
Table 4: The experiment result of DT-Solver with Tool Correction PISA dataset.
Dataset
DT-Solver
DT-Solver + sledgehammer DT-Solver + Tool Correction
PISA
37.0%
37.0%
55.2%
improves the performance, from 37.0 to 55.2%. This proves that Tool Correction is also useful for other datasets, such as the PISA dataset. The Mollehhamme++hurrre heuristics does not work well for DT-Solver on the PISA dataset, bocause DT-Solver hardly uses sledgehammer tactics, while gledgchammer+huurie only works when the given lacties is sledgehammerr. We believe the above experiment can prove that TC is also useful for other datasets but not limited to miniF2F, even for different models. Meanwhile, TC is plug-in-plug-our module, which is casily integrated into other methods.
The potential explanation of why Conjecture Correction works. There are two potential explanations
of why Conjecture Correction works.
One potential explanation: The publicly available Isabelle proof corpus also has examples of error messages. The training dataset of GPT-3 (Brown al. 2020) contains Common Crawl datasets (Raffel et al.) 2020). which contain Stackflow. On the Stackflow, we find examples of error messages. Example Failed to apply initial proof method: using this: [] E ns us_public goal (1 subgoal): 1. A A X. Says A B X & set of list
Example Failed to apply proof method: using this: (y, x) E & (z, y) E r goal (1 subgoal): 1. (z, x) E Example Failed apply initial proof method: using this: n <a n <b goal (1 subgoal): 1. n <a * b. Therefore, the publicly available Isabelle proof corpus may also have examples of error messages.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Another potential explanation: the LLMs may understand Isabelle's error message if they understand English and Isabelle's 8 syntax, but do not have to see Isabelle's error messages before. First, we show what Isabelle's error messages look like. The error message is written in English, such as "Failed to apply proof method using this: <y goal (1 subgoal): 1. * (x * sin x) + 4 / (x sin x) = (9 (x sin x)< + 4) / (x * sin x) At command "by" u. According to the error message, we can find that the error message only contains English words and Isabelle syntax (such as <šup>). Therefore, to understand Isabelle's error message, the proposed LLM may not need to see Isabelle's error message before, but just has to understand English and Isabelle's syntax.
4.7 Relation between performance and the time limit
Given a time limit, the performance of Lyra is related whether allowing parallel process. If allows the parallel procoss, then we keep the performance (55.3% validation and 51.2% on test), if the time limit is larger or equal to 10 minutes. For the relationship between time limit and performance, we can refer to Figure 3 on Page 8, which presents the relationship between the number of attempts and the performance. One attempt takes 2 minutes, if not allow the parallel process.
Table The relation between performance and time limit.
Parallel Process
Time Limit
Method
Time Cost
miniF2F-valid
miniF2F-test
10 mins
DSP
10 mins
32.7%
25.4%
10 mins
Lyra
10 mins 33.6%
28.6%
x
30 mins
DSP
30 mins
10.1%
31.5%
x
30 mins
Lyra
30 mins
40.1%
36.0%
X
400 mins
DSP
400 mins
50.4%
42.6%
X
400 mins
Lyra
400 mins
55.3%
51.2%
10 mins
DSP
2 mins
50.4%
42.6%
10 mins
Lyra
10 mins
55.3%
51.2%
Usually, it takes about 1 to 2 minutes to finish one attempt (we take min/attenmnt here), where each problem is allowed to try 200 attempts(a total of 400 minutes), in our setting. As shown in Table 5 for DSP, the 200 attempts can be processed in parallel. Hence, if allowing the parallel process, the maximum time cost minutes. For Lyra, these 200 altempts are divided into 40 patches, where each patch contains five attempts. The 40 patches can be processed in parallel. Hence, if allowing the parallel process, the maximum time cost is 10 minutes. If not allowing the parallel process, when the time limit is 10 mins, Lyra achieves 33.6% miniF2F-validation and 28.6% miniF2F-test And if not allow the parallel process, when the time limit is 30 mins, Lyra achieves 40.1% FiniF2--vllidatinn and 36.0% miniF2F-test.
5 Conclusion
In this paper, we introduced Lyra, a novel pipeline that takes advantage of Tool Correction and Conjecture Correction. Tool Correction uses existing knowledge to select and apply predefined tools, such the 'sledgehammer" tool, for the purpose of correctly replacing tools that were initially used inappropriately. Conjecture Correction, interacting with the prover environment, integrates previous formal sketch and prover error messages for belter sketch generation. We demonstrated the feasibility and effectiveness of Lyra by reaching state-of- the-tr performance 55.3% and 51.2% on the miniF2F dataset validation and test, respectively, with the Isabelle thcorem prover. Central to our mothod the incorporation of prior knowledge and the development of a comprehensive GPT-4 refinement framework. Our ablations showed that both Tool Correction and Conjecture Correction are critical to the success of Lyra.
12

---

Published in Transactions on Machine Learning Research (07/ (224))
References
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergradnate-levll mathematics. arXiv preprint 2302 120..1144. 12433, 2023.
Kshitij Bansal, Sarah Loos, Markus Rabe, Christian Szegody, and Stewart Wilcox. Holist: An environment for machine learning of higher order logic theorem proving. In International Conference on Machine Learning, pp. 454 463. PMLR, 2019a.
Kshitij Bansal, Markus N Szegedy, Christian anFd Rabe, Sarah M Loos, and Viktor Toman. Learning to reason in large theories without imitation. arXiv preprint 1xX5:1905.10501, 2019b.
Bruno Barras, Samuel Boutin, Cristina Cornes, Judicaäl Courant, Jean-Christophe Filliatre, Eduardo Gimenez, Hugo Herbelin, Gerard Huct, Cesar Munoz, Chetan Murthy, et al. The Coq proof assistant reference manual: Version 6.1. PhD thesis, Inria, 1997.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877 1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Fric Horvitz, Ece Kamar, Peter Loe, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXie preprint arXiv:2202.0066 07646, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. aixiv preprint ar arXio:2107.03371 03374, 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Scbastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXie preprint arXiv:2244.00 023111 2022.
Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris Van Doorn, and Jakob von Raumer. The lean theorem prover (system description). In Automated Deduction-CAl DF-25: 25th International Conference on Automated Deduction, Berlin, Germany, August 1-7, 2015, Proceedings 25, pp. 378 388. Springer, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences, 119(32))c2122343311 2022.
Emily First, Markus N Rabe, Talia Ringer, and Yuriy Brun. Baldur: whole-proof generation and repair with large language models. Xiv preprind arXiv:2303.04910 2023.
Thibault Gauthier, Cezary Kaliszyk, Josef Urban, Ramana Kumar, and Michael Norrish. Tactictoe: learning to prove with tactics. Journal of Aulomaled Reasoning, 65:257-288 2021.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing, 2023.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMTR, 2017.
Jesse Michael Han, Jason Rute, Yuhuai Wu, Edward W Ayers, and Stanislas Polu. Proof artifact co-training for theorem proving with language models. arXiv preprint 2Xiu:2102.00200 2021.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirly- Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin, Haiming Wang, Zhenguo Li, Linqi Song, and Xiaodan Liang, Mustard: Mastering uniform synthesis of theorem and proof data. arXiv preprint arXxv:2402.082 2024.
Albert Qiaochu Jiang, Wenda Li, Jesse Michael Han, and Yuhuai Wu. Lisa: Language models of isabelle proofs. In 6th Conference on Artificial Intelligence and Theorem Proving, pp. 378-392, 2021.
Albert Qiaochu Jiang, Wenda Li, Szymon Tworkowski, Konrad Czechowski, Tomasz Odrzygóádk, Piotr Miloš, Yuhuai Wu, and Mateja Jamnik. Thor: Wiclding hammers to integrate language models and automated theorem provers. Advances in Neural Information Processing Systems, 35:8360 8373, 2022.
Albert Qiaochu Jiang, Sean Welleck, Jin Peng Zhou, Timothee Lacroix, Jiacheng Liu, Wenda Li, Mateja Jamnik, Guillaume Lample, and Yuhuai Wu. Draft., sketch, and prove: Guiding formal theorem provers with informal proofs. In The Eleventh International Conference on Learning Representations, 2023a.
Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T Kwok. Backward reasoning in large language models for verification. arXiv preprint arXio:2308. 07758, 2023b.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-48 433,,,8845 2020.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, 7ac Hadfield-Dodds, Nova DasSarma, Eli Tran-Johnson, al. Language models (mostly) know what they know. arXiv preprint arXin:2207.0522 2022.
Guillaume Tample and François Charton. Deep learning for symbolic mathematics. In International Conference Learning Representations, 2019.
Guillaume Lample, Timothee Lacroix, Marie-Anne Lachaux, Aurelien Rodriguez, Amaury Hayat, Thibaut. Lavril, Gabriel Ebner, and Xavier Martinet. Hypertree proof search for neural theorem proving. Advances in Newral Information Processing Systems, 35:26337 26349, 2022.
Jonathan Laurent and André Platzer. Learning to find proofs and theorems by learning to refine search strategies: The case of loop invariant synthesis. Advances in Neural Information Processing Systems, 35: 4843-4858 2022.
Aitor Lewkowycz, Anders Andreasson, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, el al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-355 3857, 2022.
Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, et al. Fimo: A challenge formal dataset for automated theorem proving. arXio preprint 2rXi::2309.00.22
2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-moda model with robust instruction tuning. arXiv preprint arXiv:2306.1 2023b.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Norman Megill and David A Wheeler. A computer language for mathematical proofs. 2019.
Maciej Mikula, Szymon Antoniak, Szymon Tworkowski, Albert Qiaochu Jiang, Jin Peng Zhou, Christian Szegedy, Lukasz Kuciński, Piotr Miloš, and Yuhuai Wu. Magnushammer: translormer-based approach to premise selection. arXiv preprint arXiv:2303.04/ 04488, 2023.
Lawrence C Paulson. Isabelle: A generic theorem prover. Springer, 1994.
Lawrence C. Paulson. Three years of experience with sledgehammer, practical link between automatic and interactive theorem provers. In Renate A. Schmidt, Stephan Schulz, and Boris Konev (eds.), Proceedings of the 2nd Workshop Practical Aspects of Automated Reasoning, PAAR-2010, Edinburgh, Scotland, UK, July 14, 2010, volume 9 of EPiC Series in Compuling, pp. 1-10. EasyChair, 2010. doi: 10.29007/tnld. URL https://doi. org/10 10/ 29007 /tffd
Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint Xrx::20009. 033993, 2020.
Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and llya Sutskever. Formal mathematics statement curriculum learning. In The Eleventh International Conference on Learning Representations, 2023.
Jianing Qiu, Lin Li, Jiankai Sun, Jiachuan Peng, Peilun Shi, Ruiyang Zhang, Yinzhao Dong, Kyle Lam, Frank P-W Lo, Bo Xiao, et al. Large ai models in health informatics: Applications, challenges, and the future. arXiv preprint ar Xiv:23303 11568, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Reseorch, (11)):5485-553 55511 2020.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflcction. aixiv preprint arXiv:2303. 1 1366, 2023.
David Silver, Thomas Hubert, Julian Schrittwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(64199:1140-111 2018.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
Christian Szegedy. A promising path towards autoformalization and general artificial intelligence. In Intelligent Computer Mathematics: 13th International Conference, CICM 2020, Bertinoro, Italy, July 26-31, 2020, Proceedings 13, pp. 20. Springer, 2020.
Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfci Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. In Conference on Robol Learning, 2023a.
Haiming Wang, Huajian Xin, Chuanyang Zheng, Lin Li, Zhengying Liu, Qingxing Cao, Yinya Huang, Jing Xiong, Han Shi, Enze Xie, et al. Lego-prover: Neural theorem proving with growing librarios. arXiv preprint Xix::23310. 00656, 2023b.
Haiming Wang, Ye Yuan, Zhengying Liu, Jianhao Shen, Yichun Yin, Jing Xiong, Enze Xie, Han Shi, Yujun Li, Lin Li, al. Di-solver: Automated theorem proving with dynamic-tree sampling guided by proof-level value function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12632-12646, 2023c.
Qingxiang Wang, Chad Brown, Cezary Kaliszyk, and Josef Urban. Exploration of neural machine translation in autoformalization of mathematics in mizar. In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs, pp. 85 98, 2020.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Scan Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Natural- prools: Mathematical theorem proving in natural language. arXiv preprint or (rXx:2104.011 2021.
Sean Welleck, Jiacheng Liu, Ximing Lu, Hannanch Hajishirzi, and Yejin Choi. Naturalprover: Grounded mathematical proof generation with language models. Advances in Neural Information Processing Systems, 35:4913-4927, 2022.
F Wiedijk. Formal proof sketches. types for proofs and programs, proceedings of types 2003, Incs 3085, 2003.
Freek Wiedijk. Formal proof-getting started. 2008.
Minchao Wu, Michael Norrish, Christian Walder, and Amir Dezfouli. Tacticzero: Learning to prove theorems from scratch with deep reinforcement learning. Advances in Neural Information Processing Systems, 34: 9330-9342, 2021.
Yuhuai Wu, Albert Qiaochu Jiang, Jimmy Ba, and Roger Grosse. Int: An inequality benchmark for evaluating generalization in theorem proving. arXiv preprinl arXiv:2077.0292 2020.
Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. Autoformalization with large language models. Advances in Neural Information Processing Systems, 35:32353- 32368, 2022.
Zhiheng Xi, Senjie Jin, Yuhao Zhou, Rui Zheng, Songyang Gao, Tao Gui, Qi Zhang, and Xuanjing Huang. Sclf-polish: Enhance reasoning in large language models via problem refinement, 2023.
Kaiyu Yang and Jia Deng. Learning prove theorems via interacting with proof assistants, In International Conference on Machine Learning, pp. 6984-6994. PMLR, 2019.
Kaiyu Yang, Aidan M Swope, Alex Gu, Rahul Chalamala, Poiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkunar. Leandojo: Theorem proving with retrieval-augnnntee language models. arXiv preprint wxXi::2306.16690 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint 2rxi:22309 12284, 2023.
Xucliang Zhao, Wenda Li, and Lingpeng Kong. Decomposing the enigma: Subgoal-based demonstration learning for formal theorem proving. arXiv preprint arXiv:2305.1633 2023.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXin:2304.09997, 2023.
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: a cross-system benchmark for formal olympiad-level mathematics. aixie preprint arxiv21000 000.0000,, 2021.
16

---

Published in Transactions on Machine Learning Research (07 (27/))
Appendix
Algorithm Pseudocode of Tool Correction in Python-like style.
#tactic_ list: list of the tactics of formal proof
#prover: Isabelle Prover
#TCUsage: whether employ Tool Correction
tool hooo_hhursttcs""""a auto', 'by arith', 'by blast', 'by simp', byy fastforce', 'by force', 'by eval', 'by presburger', 'by sos', 'by linarith', 'by (auto simp: field_simps). "sledgehammer") for tactic in tactic_list:
use cee_heuristicc=False
output prover run_tuuuteeeeeeeee cin_tac(tactio) if output['error" is not None:
if TCUsage: Use Tool Correction Not
if tactic. wtt..sstipp..saaattttttcccc start or tactic.strip()== (ttrip))==("")))) use hse_heuristic--ruee
("sledgehammer" tactic) or use ase_heuristi:: for tool_try in tool_heuristics:
output prover run wwt.rruutnttttttotaaeaaaoaa tac(tool_tty) trr) if
is None:
break if output[''rronn is not None:
return "tactic_failed" output if output "'taccicc_statt =- 'no goals'; return "success", output
return "proof incomplete", output
Algorithm Pseudocode Conjecture Correction a Python-like like style.
fround_counte the current round number #prompt_sample:
#prompt_ sample: the prompt and proposed question
response: previous formal proof #error_ info: error information from Isabelle input=[("role] "system", "content": "You Mathematical Proof and Isabelle Proof Assistant. Follow the given A examples and complete proof with Isabelle Proof Assistant"), ["role": "user", "content"; prompt_samplex) cound_ #If False, the instial round, #0therwise, then the Refinement round. #Refinement Bround
expert in \
input. append "assistant", "content"; previous_respon) respon)) append
"user", "content"; "(*The last proof has the following errors from Isabelle Prover. Therefore, An 1) Please Follow the Above Prompt; \nnn \nAn 2) And Utilize the Following Errors to redo \ the last formal proof. \n O. An
proof -An" ffo.fot (error_info))
json_obj openai. ChatCompletion create (messages=input)
17

---

Published in Transactions on Machine Learning Research (07/2024)
A Tool Correction tells us something fundamental about the nature of mathematical proofs
Tool Correction tells us something fundamental about the nature of mathematical proofs: try as many proving tools as we can, so that can improve the performance by reducing the occurrence of "the conjecture is correct, but the proving tools fail to prove the conjecture".
Why does the simple strategy used by TC, i.e., iteratively replacing the tacties in failed proof with one of the 11 tactics from the tool heuristics set, work?
Recall when can we successfully prove a conjecture
The conjecture should be correct, and the proving tools (such as sledgehammer) can prove the conjecture.
Recall when we fail to prove a conjecture: the proving tools fail to prove the conjecture.
Situation 1: The conjecture is incorrect.
Situation 2: Or, the correction is correct, but the proving tool not correct the proving tool cannot prove the conjecture.
Why Observation: prover (such as Tsabelle or Lean) fails to validate a formal proof, which may caused by "the conjecture is correct, but the proving tools fail prove the conjecture". The following an example.
Conjecture: 1*1=1
If the proving tool can not process multiplication operation, then it will fail to prove the conjecture 1*1=1, though the conjecture is correct.
Therefore, to improve the performance, we can reduce the occurrence of "the conjecture is correct,
but the proving tools fail to prove the conjecture"
One solution is Tool Correction: try as many proving tools as we can to prove the conjecture.
B Other Potential tactics besides the sledgehammer + 11 tactics
To improve the effectiveness of TC, we can easily add tactics, including by clarify, by fact, by rule, by erule, by elim, by induction, by algebra and so on.
According to the DSP paper, the 11 tacties can be used to process algebra and number theory problems. And the lactics also have their own advantages and disadvantages. For example, it seems that by simp is not very good at processing natural numbers, shown in Figure in our paper.
As introduced in the DSP paper, the Sledgehammer (Paulson 2010) is relatively special, as it works by flattening the goals encoded in the higher-order logic used by Isabelle/HOI, into other logics (e.g,, first- order logic) which can then be fed into automated theorem provers. The development proving tools is also an important direction of automated thcorem proving. Recently, there has boen a work named Magnushammer Mikula al. 2023), which is more powerful than Sledgehammer.
18

---

Published in Transactions on Machine Learning Research (07 (07/2024)
C IMO 1974 p5 Case
Statement:
Determine all possible values of $ = atd+d + positive numbers.
atdto +
"
btotd +
atota where 6, c, d, are arbitrary
Informal Proof (Human): Note that 2 a+6 atb 24d a.d 8 atbictd a4b4e4d prove that can reach any range between and Choose any positive number a. For
b
a+bboed + atbtoed 1. We will now
variables such that k, 0 and k+ | let b ak, am, and am + atamial ok + aktamtal
d al Plugging this back into the original fraction, we get S ataksa! a-ak
I is a strictly increasing function. We can +++ 14 + 1444m k4m47 + T+m447 The above equation can be further simplified to = 2-m + 2 + m I Note that S is a continuous function and that f(m) m + 2-m now decrease and to make m tend arbitrarily close to 1. We see limm-1 m + 2-m 2, meaning $ can be brought arbitrarily close to Now, set
and b c== y some positive real numbers 2y 482 Notice that if treat the numerator and denominator each y. Then S 2zty 2y+r 2y"+5ep+2r"
2 where g(r) has degree lower than 2. This means laking limy- +5*+ 1, which means can brought: arbitrarily close to Therefore, we are done.
quadratic in will get 1 + g(o)
Formal Proof:
theorem imo_ 1974_p5:
fixes a b c d s :: real
assumes "a>0" "b>0" "c>0" "d>0"
assumes ho (a=a/(a + b/(a+b+c) c/bbctd) + d/ (a+c+d)"
shows "1<g \<and> s<2"
proof
1et "a/(a+b+d) b/ (atbbc) (b+c+)) d/ (a+c+d)
have c0: "a/(a+b) (b/))////////)))) (atb) (c+d) +d/ (c+d) 2" using assms by auto [АРРИииГ
have ci: "a/(atbec+d) +b/ (aatrrrrrddee/ttttttttttaddddddd (a+bb++c+) AТPWИATC
using asams by auto [
have "1 ?S" using asems by (simp add: add add_pos_pos c1) ПАТРИиКТ
moreover have "?S 2" using assms (simp add: add_pos_pos c0) [АТРИитО then show "1<s \<and> s<2" using asams ho by auto [ATPWA*7 ged
Figure 5: IN0_1974_p3 first round.
19

---

d atbtotd = 1. We will now Published in Transactions on Machine Learning Research (07/2024)
Statement:
Determine all possible values of $ atbid + atbte + positive numbers.
broed + atopa where a, 6, c, d, are arbitrary
Informal Proof (Human): Note that = a46 + with cld c+d prove that reach any range between 1 and
6
atbicta atbtetd
6
+ u+btotd +
Choose any positive number a. For some variables such that k, m.m > 0 and k 1, let b = ak,
am, and
am + atamial al. Plugging this back into original fraction, we get S atak+al a-ok +am uktam tal
m 14 + 14k+m k+m+7 + T4m+4T The above equation can further simplified to z-m + 2 + + - z Note that S is a continuous function and that f(m) = m + 2-m now decrease and to make m tend arbitrarily close to 1. We see limm-1 m + 2-m
1 is a strictly increasing function. We can
I 2, meaning 5
can be brought arbitrarily close to Now, set d and b = y for some positive real numbers 282+82=+22"" Notice that if we treat the numerator and denominator each 2y"+5zut2s"
V. Then S as a quadratic in we will get 1 + 2y² 22y+4222 where g(r) has a degree lower than 2. This means taking s(z)
g(z)
lomy-+o 25"+ which means can be brought arbitrarily close to Therefore, we done.
Formal Proof:
theorem imo_ imo_1974_p3 1974_p5:
fixes c s real
assumes "a>0" "b>o" "c>0" "d>0"
assumes "s=a/(atbtd) b/(abbtc) c/ (b+c+d) (a+c+d)" shows "i<s \<and> s<2"
proof
let (a/(atb+d) + b/(atb+c) (b+cc)) d/ (a+cod)"
(* Let'' show that have ci: "a/(atbecdd) + (aabbec+d) (aabb+c+d) + d/ d/(a+b+ctd) = 1" using asams by (simp add: divide divide_simps) have "1 < ?S" proof
note wesssstrttttont [of b d, of ba c of c e cbd, of a d] then have "a/(a+)+)) aaatbtc+t)" "b/(atbtc) b/ b/(abtcctd)" "c/(bd) > c/ (a+b+c+d)" "d/(atctd) > d/(abtc+d)" using asams by (simp_all add: seaidd__srcttrrihhhee then show "1 ?S" using by auto [АТРШММ
[АТРИитГ
qed
(* Let's show that < +)
have c2: "a/(atb) + ba(abb) c/(c+d) d/(ccd) 2" using asams by (simp add: divide_simpa [АТРWиATC have "?S 2"
proof
note add_strict_mone mof a a 0, b ba of c O of d then have "a/(atbta) < a/ (a+b)" "b/(atb+c) < b/(atb) "c/(b+c+d) c/ (c+d)" "d/(at)td) dc(dd))
using asams by (simp_all all add: Aavide_strict sttrcttright__steeetttttttttteet (igght_mono) [АТРИаNГ then show "?s < 2" using c2 by auto [АТРWм]Г]
ged
then show "1<s \<and> s<2" using ho ?S" by auto [АПРИии ged
Figure IMO_1974_p5
second round.
20

---

d atbtotd = 1. We will now Published in Transactions on Machine Learning Research (07/2024)
Statement:
Determine all possible values of $ atbid + atbte + positive numbers.
broed + atopa where a, 6, c, d, are arbitrary
Informal Proof (Human): Note that = a46 + with cld c+d prove that reach any range between 1 and
6
atbicta atbtetd
6
+ u+btotd +
Choose any positive number a. For some variables such that k, m.m > 0 and k 1, let b = ak,
am, and
am al. Plugging this back into original fraction, we get S atak+al a-ok +am uktam tal + atamial m 14 + 144+m k+m+7 + T4m+4T The above equation can further simplified to z-m + 2 + + - - Note that S is a continuous function and that f(m) = m + 2-m now decrease and make m tend arbitrarily close to 1. We see limm-1 m + 2-m
1 is a strictly increasing function. We can
I 2, meaning 5
can be brought arbitrarily close to Now, set d and b = y for some positive real numbers 282+82=+22"" Notice that if we treat the numerator and denominator each 2y"+5zut2s"
V. Then S as a quadratic in we will get 1 + 2y² 22y+ where g(r) has a degree lower than 2. This means taking s(z) limy-+o 25 1, which means can be brought arbitrarily close to Therefore, we done.
g(z)
Formal Proof:
(#statement begin*)
theorem imo_1974_p5: fixes b d real
assumes "a>0" "b>o" "c>0" "d>0" assumes ho "s=a/(atbtd) b/(abtc) c/bbctd) + d/ (atccd)) shows "1<s \<and> s<2" (*statement end*) proof
Let'a
the proof. +)
(* Define the aum S +)
let (a/(a+ + b/ (atbbc) c/(bcct)) d/ (a+c+d)"
(* Show that > 1 *)
have (atbboo)+ b/(atbte+d) (/(ab+c+d) d/ (aatb+c+d)
using asems by (simp add: divide saappsssmsp) ПАРРИитП moreover have proof
a/(atbo++)) b/(atboc+)) c/aabtc+d) d/aaboc+d)"
have "a/(atb+d) > a/(a+b+c+d) "b/(atbtc) b/ (a+bbc+)d)"
"c/(b+c+d) c/(atb+c+d" "d/ > d/ d/(a+be++d""
using assms by (simp_all add: divide_sted wwvide_strict... sttricttrigttttteeetttttttttteet mono) (АТРИмT then show ?thesis by auto [АТРИиNT(] qed
ultimately have "1 ?S" by simp [АТРИиTТ
(* Show that 3 +)
have "a/(a+)) + (a+b)) c/+c)) (c(d)
using assms by (simp add: divide moreover have proof
divide_simps) simps) [АЛРиаиК
(atb) b/(atb) c/+c) (c+d)"
have "a/(atb+d) a/(ab)) "b/(a+btc) (a+b)"
"c/(b+ctd) < c/ " (atcct)) < d/(d))) using asams by deimp_all add: ceaiiddssrrittrrighhte [АТРИаТГ
then show ?thesis by auto [АТРИмAT qed
ultimately have "?S 2" by simp [АТРИиИГ
(* Conclude the proof +)
then show "1<s <<nnd> s<2" using ho "1 ?S" by auto [АТРИиNТ
qed end
Figure IM0_1974_pp
Third round.
21

---

Published in Transactions on Machine Learning Research (07 (07/2024)
D IMO 1959 pl Case
Statement:
Prove that the fraction 2ln+d 14n43
is irreducible for every natural mumber nh
Informal Proof (GPT-4): We can use the Euclidean algorithm to prove this. If the greatest common divisor (ged) the numerator and the denominator 1, then the fraction irreducible. Let's find the ged of 21n + 4 and 14n + 3.
We start by subtracting the smaller number from the larger one. We get: (21n + 4) (14n +3) Tn +1. Next, subtract 7n 1 from 14n + 3. We get: (14n 3) (7n + 7n 2.
Now, we subtract 7n + from 7m + 2. We get: (7m + 2) (7n + =1
Since the greatest common divisor of 21n + 4 and 14n 3 is the fraction 14n+3
21x+4 is irreducible for every
natural number n.
Formal Proof:
theorem smo_1959_pt_ fixes nat shows "ged (21+n 4) (14th 3)
proof
Let's find the ged of 4 14n 3. *)
have hi: "ged (21*n + 4) (14*n + 3) ged ((21*n + 4) - (14+n + 3)) (14tn + 3)"
by (metis add add_ difddiff__n serf__iiveeeeeeee nat ged_ nat le_add2 nat nat_le_linear [АТРииТт subtracting the smaller number from the larger one.
(* We start
We get: (2In 4) (14n + 3) = 7n + 1. *)
also have "... ged (7+n 1) (14+n + 3)" by simp [АПРΝаИТ)]
(* Next, we subtract 7n+11 from 14n + 3. We get: (14n + 3) (7n + 1) = 7m 22 *)
also have "... ged (7+n 1) (7+n 2)"
by (metis ced_difff_ivveree nat gcd_diffi_ms
le_add2 nat Now, subtract 7 from + 2. We get: (7n (7n 1. +)
[АТРии67
also have "... ged (7+n + 1) 1" by simp [47РИиTГ]
Since is the greatest common divisor any number,
find that ged (7+n + 1) - 1. +)
finally show ?thesis by simp [АTРИиTO
ged
Figure 8: IMO_1959_p1- first round.
22

---

Published in Transactions on Machine Learning Research (07/2024)
Statement: Prove that fraction 2ln+d is irreducible every natural mumber n. 14m+3
Informal Proof (GPT-4):
We the Euclidean algorithm prove this. If the greatest common divisor (ged) of the numerator and the denominator 1, then the fraction irreducible. Let's find the ged of 21n and 14n 3.
We start by subtracting the smaller number from the larger one. We get: (21n + (14n + + 7n +1 Next, subtract 7n from 14n + 3. We get: (14n
(7n + 1) 7n
Now, we subtract 7n + from 7n + 2. We get: (7n + 2) (7n 1) 1.
Since the greatest common divisor of 21n + 4 and
is the fraction 21n44 is irreducible for every 14x+3
natural number
Formal Proof:
theorem imo_1950 10__1999_ppp_ fixes nat
shows "ged (21#n 4) (14+n + 3) = 1"
proof
(* Let's find the ged of 21n + 4 and 14n + 3. *)
have h1: "ged (21*n
(14*n
ged (14#n 3) ((21#n mod (14*n 3))
using acd_red_nat' _at by blast [АТРWидТ(]
Ne compute modulo operation
get: (21n 4)mod((41)) + 3) + *)
also have .... ged (14+n + 3) (7+n + 1)"
by (ent righti neutral add_Suc L____uu_rrte Suc_right add aad_lessD1 ged. commute mult 2 mult_Sut Suc rigattteetttttalll
numeral_Bite BittalBBitttt numeral Bit1 numeral_eq_iit [АТРИадТ(] (* Then again compute the GCD with the modulo operation. also have "... ged (7+n 1) ((14tn 3) mod (7+n 1))
mult mult_less___ 1eaee__canccllllll_eeee__ mult_numerge semiring_norm(?7 (76) semiring_nozen (7)m) semiring.norm(B)
using ged_ree red nat by blast [АПРWыИТ(]
(* We compute the modulo operation and get: (14n + 3)mod( 7n + = n + *) ged (7+n 1) (n 2)"
also have
by (ant add_2_eq_Suc add_lesalD ged. commute mult_2 mult fut__uuc mult mult_les______elll la__llss__aa_e Less_cancel1 semiring_nore (8)m)))) [АТРИиАТ(]
mult_Suc
semiring. Baaiin_oommm)) (76) semiring_normt (* Then we again compute the GCD with the modulo operation. +)
also have "... ged using ged_ ged_red_mmt rer_red__a__ by blast [АТРыаиТ
2) ((7*n + 1) mod 2))
Ne compute the modulo operation also have "... ged (n + 2) 1"
get: (7n + 1)mod(( + 2) *)
by (ent add. commute add add_lessDi ged. commute mult mult_less____ set__eess_cancell wwwtarinnnnorm((.
semiring_nore norm (76)
semiring_norm(8)) [АЛРИиИ(
(* ged of any number with 1 is 1. +)
also have "... 1" by simp [АТРИмТ( finally show ?thesis [АТРииАТ ged
Figure IMD_1959_p1: second round.
23

---

Published in Transactions on Machine Learning Research (07/2024)
Statement: Prove that fraction 2ln+d is irreducible every natural mumber n. 14m+3
Informal Proof (GPT-4):
We the Euclidean algorithm prove this. If the greatest common divisor (ged) of the numerator and the denominator 1, then the fraction is irreducible. Let's find the ged of 21n and 14n 3.
We start by subtracting the smaller number from the larger one. We get: (21m 31m+44 (14n + + = 7n +1 Next, subtract 7n from 14n + 3. We get: (14n
(7n + 1) 7n
Now, we subtract 7n + 1 from 7n + 2. We get: (7n + 2) (7n 1) = 1.
Since the greatest common divisor of 21n + 4 and 14n is the fraction 21n44 is irreducible for every 14x+3
natural number
Formal Proof:
theorem 1mo_1959____ 1959_p1:
fixes nat
shows "ged (21#n 4) (14*n + 3) =
theorem imo_ sso_1959_p1: fixes n nat shows "ged (21#n 4) (14tn 3)
proof
Let's
ged of 21n + 4 14n + 3. *)
have h1: "ged (21*n + 4) (14*n + 3) god (14#n 3) ((21*n + 4) mod (14+n + 3))
using gcd_red red nat by blast [АТРиит((]
(* We compute modulo operation and get: (21n 1)mod(11n + 3) 7n also have "... ged (14+n 3) (7*n + 1)"
(simp add: mod_simps) [АТРWиА7(]
Then we again compute the 6CD with the modulo operation.
also have "... ged (7+n 1) ((14tn 3) mod (7+n 1))"
using ged_ ced_red_nat nat by blast [АПРWиАТ
We compute the modulo operation and get: (14n 3)mod(7n + +1) 2.
also have "... ged (7+n 1) (n 2)"
by (simp add: mod_ add_simps) [АТРWаЦР[]
(* Then we again compute the GCD with the modulo operation. *)
also have "... ged using ged_red_mmt red_red_____ by blast [АТРыыТГ
2) ((7*n + 1) mod + 2))
Ne compute
modulo operation and get: (7n + 1)mod(( + 2)
*)
also have "... ged (n 2) 1"
by (eimp add: mod_ sod__iimp) [АТРИм7( (* ged of any number with 1 is 1. +) also have "... 1" by simp [АТРWиN7Q finally show ?thesis [ATPИиAT[] qed
Figure 10: IMO_1959 19_11999_pp
third round.
24

---

Published in Transactions on Machine Learning Research (07/2024)
Statement: Prove that fraction 2ln+d is irreducible every natural mumber n. 14m+3
Informal Proof (GPT-4):
We the Euclidean algorithm prove this. If the greatest common divisor (ged) of the numerator and the denominator 1, then the fraction is irreducible. Let's find the ged of 21n and 14n 3.
We start by subtracting the smaller number from the larger one. We get: (21n + (14n + + = 7n +1 Next, subtract 7n from 14n + 3. We get: (14n
(7n + 1) 7n +
Now, we subtract 7n + from 7n + 2. We get: (7n + 2) (7n 1) = 1.
Since the greatest common divisor of 21n + 4 and
is the fraction 21n44 is irreducible for every 14x+3
natural number
Formal Proof:
theorem imo_1959 1959_p1: fixes nat
shows "ged (21#n 4) (14+n + 3) 1"
proof
(* We can use the Euclidean algorithm to prove this. +)
If greatest common divisor (ged) of the numerator and the denominator 1, then the fraction is irreducible. +)
have hi: "ged (21*n
(14+n + 3) ged (14*n 3) ((21+n mod (14*n 3))
using ged_rey ued_red_nas by blast [АПРWа/Т(]
We start by subtracting the smaller number from the larger one. *)
also have "... ged (14+n + 3) ((21 14) + (4 - 3))" using mod_if by auto [АТРWаТТ]]
(* We get: (21n + 4) (14n + 3) 7 7h-1- 1. *)
also have "... ged (14#n 3) (7mn + 1)*
by simp [АТРИиАТ
(* Next, we subtract 7 + from 14n + 3. *)
also have "... ged (7+n 1) ((14#n 3) mod (7*n 1)) using ged_red red_nat by blast [АТРшатT(]
Ne get: (14n+3) (7n 1)
+2. *)
also have "... ged (7+n 1) ((14 7)*n + (3 1)))
using mod_if by auto [АТРшиГТ
also have "... ged (7+n + 1) (7+n 2)"
by simp [АТРИиТТ
(* Now, subtract 7 1 from 7n + 2. *)
also have "... ged (7+n 2) ((7*n + mod (7+n 2))
using ged_redd_mt rer_red__aa by blast [АПРWаNT
(* Ne get: (7n + 2) (7n = 1)
+)
also have "... ged (7+n 2) (1)"
using mod_if if by auto [АТРииКТC]
Since the greatest common divisor 21m 14n + is +)
also have "... 1"" by simp [АТРИиTT finally show ?thesis [АТРИиИC
ged
Figure 11: IMO_ WM0__959_pP _9:
fourth round.
25

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
E IMO 1981 p6 Case
Statement: The function (a, satisfies
(1) f(0,y) = 1,
(2) [[]++,0)=f(a;
(3) f(z f(++1,y+1) = f(x f(x+11 1,
for all non-negative integers x, y. Determine f(4, 1981)
Informal Proof (Human):
We observe that ((1,0) = f(0, 1) = 2 and that f(1, + 1) (LL,(L..)) ((1,)) 1, so by induction,
f(1, = + Similarly, f(2,0) 0) f(1, 3 and (12 + yieldng f(2, y) = 2y -
We continue with f(3, 0) + - 8; - 2(f(3. y) + 3); ((3 +3 2 and f(4, 0) -
/(4, y) 3 2 (4.0)
It follows that f(4, 1981) 2 when there 1984 2s, Q.E.D.
Formal Proof:
theorem imo_1981 1981_p6: fixes :: "nat \<Rightarrow> nat \Rightarroww> nat"
assumes "\<forall> y. f 0 y
+
and "\<forall> x. f (x 1) =
and "\<forall> y. f (x 1) (y 1)) f (f (x 1) y)"
shows "\sforall> y. f 4 (y 1) 2 3) 3"
(*statement end*)
proof
Ne observe that ((1,0) f(0,1)
= 1(0,1)=2
have "\<forallyy. f 1 y = + 2" proof
fix show "f y y
proof (induction y)
case
then show ?case using asama(2) [rule format, of "O"]
by simp AТРшаNT
next
case (Suc y)
then show ?case using assms (3) [rule format, of "0" "y"]
by simp ATPWаNT[]
ged qed then f1: "\<forallyy. y by simp Similarly, f(2,0) =[(1, = and ʃ(2 + 1) ((2.w) +2, yielding f(2. 2y + *)
have "\<forallyy. f 2*y 3"
Figure 12: IMO [00_1981_pp0
Part 1.
26

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
proof
fix
show "f y 2*y + 3"
proof (induction y)
case
then show ?case using asams (2) [rule_ format, of "1"] using by simp ATP
next
case (Suc y)
then show ?case using assms (3) [rule format, of "1" "y"]
using by simp ATPHINT qed qed
then have f2: "k<forallyy.
2+y 3"
by simp [АTРWАNI
(+ We continue with f(3, 0) =3 f(3, 2(((3,)) +3; 5(3.v) + 2
- *)
have "\<forallyy. f 3 y + 3 2 3)" proof
fix
show "f y y (y 3)"
proof (induction y)
case
then show ?case using assms (2) [rule_ format, of "2"]
using f2 by simp [ATPWaTT[]
next
case (Suc y)
then show Pcase using a3)
of "2" "y"]
using f2 by simp [ATPиıTG]
qed qed then have f3: "\<forallyy.
(y 3)"
by simp [АТРыанT
and f(4, 0) +3 2²²; f(4.)) -3= +3==((44))33 +)
have "\<forallyy. f 4 (y 1) = 2f 4 3) 3" proof
fix
show "f 4 (y 1) 2 (f y 3" proof (induction y) case
then show Pcase using a2sms(2) [rule. format, of "3"]
using f3 by simp [АТРииTT]
next
case (Suc y)
then show ?case using a3ama(3) trule_forman "3" "y"]
using f3 by aimp [ATPииAT(]
ged
ged
then ?thesis simp [АТРИииς
ged
This problem is finished with human informal proof and Tool Correction, but. without Conjecture Correction. Actually, if problem be solved by only Tool Correction, then it can be solved via Tool Correction and Conjecture Correction. The Lyra needs more attempts to solve the IMO_1980 110__981_pppl __61___________ as Conjecture Correction needs more attempts.
Figure 13: IMD MMD_1981_p6 Part 2.
27


Reviews:
[{"review_id": "wgiAyCqMwN", "reviewer": "Reviewer_Gg5E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#Strengths\n\n- The presentation is very nice.  All figures, plots, and listings are legible\n  and well designed.  (This does not apply to section 4.6, which could use\n  better fonts for math and code snippets.)\n- The experiments seem to be designed appropriately, and include both\n  ablation experiments and runtime comparison.\n- Lyra (that is, the combination of the TC and CC algorithms) appears to be\n  effective.\n\n#Weaknesses\n\n- English is acceptable, but I found the text quite hard to follow.  Notions\n  are used without previously being defined (like \"tool\" and \"IMO\" in the\n  abstract) or first introduced by assuming familiarity with very low\n  level technicalities (like the `by (simp add: div_mult_mod_eq)` syntax,\n  which is never really explained in any detail, to the point that by section\n  3 I am still unclear why using it does not work).\n\n  I strongly suggest the authors to introduce all key notions at a sufficiently\n  high level, providing concrete examples for the key notions, and specifically\n  examples that do not rely on the reader being knowledgeable about all\n  low-level details from the get go.\n\n  See below for more pointers.\n\n- Lyra, although effective, is very simple, as made clear by Algorithms 1 and 2\n  (which are a for loop that tries out different \"tactics\" until one works, and\n  a rather straightforward prompting technique, respectively).  This is not a\n  major issue per se: simple modifications can make a world of difference in\n  practice.  Yet I cannot help to notice that these contributions are *very*\n  simple.\n\n- I am not sure why the authors report improved performance on the validation\n  set:  as long as this set is used in the learning procedure, there is no\n  point in doing so.  If the validation split is not used for learning (for any\n  competitor), there's no reason to treat it differently from the test set,\n  that is, the authors should report the success rate on the union of\n  validation and test set.\n\n- The results come with no variance information (and no statistical testing).\n\n- About half chunk of the performance improvement appears to be due to working\n  with GPT-4 rather than other LLMs (the ablation results in Table 2 shows\n  GPT-4 accounts for about 7-8% performance over Codex), while Lyra (TC+CC)\n  accounts for about 4% (48.3% -> 52.8% and 47.9% -> 52.0%).  The question is\n  what is Lyra exactly: is it TC+CC, or is it GPT4+TC+CC?  The introduction\n  seems to follow the former description, the experiments however the latter.\n  The two contributions should be clearly distinguished throughout the text.\n\nAll in all, I am on the fence about this paper.  While the empirical evaluation\nseems to showcase state-of-the-art performance on the miniF2F data sets, I am\nnot familiar enough with the problem domain to know whether this data set is\nstandard or whether achieving SotA performance on it is significant (the\nauthors should probably discuss this in the paper).  At the same time, the\ntechnical contribution is very simple and the text is not particularly easy to\nfollow.\n\n\n#Detailed review\n\n* The abstract should be clarified.\n\n  - The opening sentence is not easy to understand.  It reads: \"Large Language\n    Models (LLMs) present an intriguing avenue for exploration in the field of\n    formal theorem proving\".  In the sense that LLMs can be used for proving?\n    Or that theorem proving tools can be exploited for improving LLMs?  This\n    should be rewritten to provide more guidance to the reader about the\n    indended meaning - currently it is confusing.\n  - \"for guiding the replacement of incorrect tools\": what is a \"tool\" in this\n    context? and how is it related to hallucinations?\n  - why is performance on the *validation* set worth reporting?\n  - what is IMO?\n\n  Please rewrite the abstract to introduce all needed concepts properly (and do\n  not refer to those that cannot be explained in short) so that it is\n  accessible to all readers with background in formal verification and ML (but\n  possibly not super up-to-date on LLM lingo).\n\n* Introduction: equally difficult to follow, as it is not clear if it is the\n  LLM vs. the formal prover that is supposed to carry out reasoning.  Consider\n  for instance the sentence \"In this paper, we seek to build Lyra based on LLM,\n  focusing on formal theorem proving\".  What does it mean that Lyra is \"based\n  on\" LLMs?  This is vague and thus confusing.\n\n  The concrete example provided at the end of page 1 would be useful *if* it\n  didn't refer to overly technical implementation details (what is \"by (simp\n  add: div_mult_mod_eq)\"?).  It also presupposes the reader knows how LLMs have\n  been used to generate proofs, which is not necessarily the case.\n\n  While Figure 1 helps to convey the intuition, I find it unnecessarily\n  complex and - again - overly technical.  A reader unfamiliar with the\n  specific syntax should be able to understand the figure.\n\n* Sec 3.1:\n\n  - What is a \"formal sketch\"?  A formal statement?\n  - \"the informal proof is the informal solution\" - how is a proof a solution?\n    A concrete example would help to ground the notions introduced here.\n  - \"also utilize\" -> utilizes.\n  - \"Scenarios.Given\" - missing space.\n  - \"we should first generation informal proof informal proof\"\n  - \"sketch gaps\" -> gaps in the formal sketch?\n\n* Sec 3.2:\n\n  - \"Tool Correction employs prior knowledge to employ predefined tools (e.g.\n    sledgehammer) to guide incorrect tool replacement\".  This is unclear.\n  - At this stage, I still don't know what `by (simp add: div_mult_mod_eq)` is\n    and why it should fail.\n  - \"The Tool Correction approach entails the validation of a given tactic t\n    using Isabelle. If validation succeeds, we proceed; if not, Tool Correction\n    intervenes to modify the tactic.\"  This very much sounds like a for loop\n    implementing -- essentially -- rejection sampling.\n\n* Sec 3.3:\n\n  - \"mathematical proving\" -> theorem proving.\n\n- p 12: \"Cobjecture\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "First of all, let me apologize for the very late review.\n\nThe authors aim at leveraging LLMs for automated theorem proving.  Focusing on\nthe Draft, Sketch, and Prove (DSP) pipeline, the authors identify issues\nrelated to 1) what tools the LLM decides to deploy in the generated proof,\nwhich sometimes are inappropriate leading to failures; 2) cases in which the\nLLM's output could be easily fixed by integrating a formal prover.  Motivated\nby these observations, they design two strategies -- collectively referred to\nas Lyra -- that 1) loop over alternative tools; and 2) prompt the LLM again\nwith the error message from the Isabelle solver.  Lyra is then tested chiefly\non the miniF2F data set, a collection of 488 high-school math problems\nformalized in three languages, showcasing improved success rate (in proof\nformalization) on the test set.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I strongly suggest the authors to reread and the abstract and sections 1-3 and properly introduce all key notions, using concrete examples to elucidate their meaning, avoiding usage of not-previously-defined syntax and low-level details about the proving procedure.\n\nI'd also appreciate if the authors could clearly distinguish between the improvement due to Lyra (namely, TC+CC) and that due to using GPT-4 as early as the abstract and introduction.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3sK7Bki963", "reviewer": "Reviewer_q1nn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The two proof-repair methods significantly improved the performance of DSP on the MiniF2F dataset. The ablation study section gives convincing arguments about the effects of the two methods. \nWeakness: The ideas behind TC and CC are not novel. In particular, TC fixes the proof step by simply exhausting pre-defined tactics and does not utilize the LLM, which is too limited. Also, on MiniF2F, Lyra does not outperform LEGO-Prover[1] (which has an accuracy of 57.0% and 50.0% on the MiniF2F validation and testing set, respectively). \n[1] Xin, H., Wang, H., Zheng, C., Li, L., Liu, Z., Cao, Q., ... & Liang, X. (2023). Lego-prover: Neural theorem proving with growing libraries. arXiv preprint arXiv:2310.00656.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Lyra, an LLM-based theorem-proving framework in the Isabelle proof assistant. Based on the Draft-Sketch-Proof (DSP) framework, the paper presented two methods to mitigate the hallucination of LLM: 1. The Tool Correction (TC) method finds each failed tactic and fixes it by attempting to replace it with some pre-defined tactics (e.g. \"by auto\", \"by blast\"). 2. The Conjecture Correction (CC) method refines the proof by prompting the LLM with the existing proof and error messages. Compared with DSP, Lyra significantly improved the overall accuracy of the proof from 50.4% to 55.3% on the MiniF2F validation set and from 42.6% to 51.2% on the test set.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would strongly suggest releasing the source code of Lyra to OpenReview. Also, to strengthen the work, it would be better to:\n1. Fix the minor spelling errors (e.g.  \"Falase\", \"Rround\", etc. on page 5)\n2. It is counter-intuitive that Conjecture Correction deteriorates the accuracy when the number of autoformalization attempts is low and the Tool Correction is disabled. Explain more about it.", "broader_impact_concerns": "There is no ethical concern about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tr7X9tLpMC", "reviewer": "Reviewer_u5qc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Arguably, the experimental results are the main strength of the paper, since they indicate a clear improvement over the state-of-the-art DSP method.\n\nThe Lyra framework is designed to improve DSP, which is its main weakness. The Tool Correction feature of Lyra utilizes Sledgehammer and other heuristics when a rule is misused in the sketch proof. If these tools pass the theorem prover successfully, Lyra moves on to the next rule. If not, the proof attempt is deemed unsuccessful. The Conjecture Correction feature of Lyra is based on the well-known restart technique, commonly used in constraint solvers and theorem provers. During each restart, this feature appends the last error message returned by the prover and uses an LLM to improve the proof sketch before reaching the next restart. So, in a nutshell, these correction tools are essentially plugins that help DSP refine proof sketches, but they do not involve any conceptual innovations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on enhancing neural-based theorem provers in formal theorem proving, by using Large Language Models (LLMs). The authors propose a framework called \"Lyra\", which builds upon the recent Draft, Sketch, and Prove method (Jiang et al., ICLR'23). This method begins with informal proofs, which are then mapped into formal sketches and finally, a theorem solver is used to prove the remaining conjectures. Lyra's innovation lies in two components: \"Tool Correction,\" which aims to fix incorrect rules used in proof sketches, and \"Conjecture Correction,\" which improves sketch generation from past sketches and error messages.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "As the Lyra framework is essentially an engineering improvement of DSP, I cannot recommend acceptance of this paper at TMLR. However, I suggest the following corrections if this paper is accepted with revision:\n\n* A more detailed description of DSP would be helpful for the reader. Currently, there is only a short section on Page 4 that provides a broad description of DSP, which is not sufficient to fully understand Lyra's features. \n* Since the Tool Correction feature is just repairing misused tactics in proof sketches using some library of tools, the Conjecture Correction feature should be the main focus of this study. However, the description given on Page 6 is too ambiguous to fully understand the functionality and potential benefits of this feature. Therefore, I suggest providing a more detailed description, replacing the pseudo-code in Page 5 with a figure illustrating the functionality of Conjecture Correction, and explaining correction rounds after each restart.\n* The paper requires more polishing. For the moment, it is littered with typos and spelling mistakes (e.g. \"Falase\", \"Rround\" in Algorithm 2, \"hep\" in Page 6, and so on).", "broader_impact_concerns": "Given that this study is centered on formal theorem proving, there are no ethical issues or broader impact concerns to consider.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While I am recommending acceptance, I think it is important for the authors to incorporate results with respect to different baselines (including LEGO-Prover), even if Lyra does not improve over itself.
Decision: Accept (poster)