SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lean 3 and Lean 4 are two incompatible major versions of Lean, $^{12}$  and both are widely used. Lean 3 was the latest stable version until recently (June 2023). Also, Lean 3 and Lean 4 have separate versions of mathlib. The Lean/mathlib community has recently finished porting theorems and proofs from mathlib 3 to mathlib 4 [100]. Therefore, Lean 3 will gradually become deprecated, and future Lean projects will be using Lean 4. Therefore, it is important for LeanDojo to support Lean 4.
Since Lean 4 is relatively new, we are not aware of any existing work on learning-based theorem proving in Lean 4. Furthermore, no existing tool is available for extracting data from Lean 4. LeanDojo fills in this gap and fully supports Lean 4. Given any repo in Lean 4, LeanDojo can extract
$^{12}$ https://leanprover.github.io/lean4/doc/lean3changes.html
theorem exercise_2_3_2 {G : Type*} [group G] (a b : G) :  
     $\exists g : G, b * a = g * a * b * g^{-1} :=$   
begin  
    exact (b, by simp),  
end  
theorem exercise_11_2_13 (a b : Z) :  
    (of_int a : gaussian_int) | of_int b → a | b :=  
begin  
    contrapose,  
    simp,  
end  
theorem exercise_1_1_17 {G : Type*} [group G] {x : G} {n : N}  
    (hxn: order_of x = n) :  
     $x^{-1} = x^{\wedge}$  (n - 1 : Z) :=  
begin  
    rw zpow_sub_one,  
    simp,  
    rw [- hxn, pow_order_of_eq_one],  
end  
theorem exercise_3_1_22b {G : Type*} [group G] (I : Type*)  
    (H : I → subgroup G) (hH : ∀ i : I, subgroup.normal (H i)) :  
    subgroup.normal (n (i : I), H i) :=  
begin  
    rw infi,  
    rw ←set(image_univ,  
    rw Inf_image,  
    simp [hH],  
    haveI := λ i, (H i).normal,  
    split,  
    intros x hx g,  
    rw subgroup.mem_infi at hx ←,  
    intro i,  
    apply (hH i).conj_mem_(hx i),  
end  
theorem exercise_3_4_5a {G : Type*} [group G]  
    (H : subgroup G) [isSolvable G] : isSolvable H :=  
begin  
    apply_instance,  
end
Figure C: Examples of new proofs discovered by ReProver on ProofNet [29].
data, including file dependencies, ASTs, proof states, tactics, and premise information. In addition, it enables the model to interact with Lean 4 through tactics, in the same way as Lean 3 (Sec. 4).
Similar to constructing the Lean 3 version of LeanDojo Benchmark, we extract data from the commit 3ce43c18f614b76e161f911b75a3e1ef641620ff of mathlib4 released on October 21, 2023. The resulting dataset is named LeanDojo Benchmark 4. It is released under the CC BY 2.0 license and hosted on zenodo.org with DOI "10.5281/zenodo.8040109". LeanDojo Benchmark 4 consists of 102,514 theorems/proofs, 213,067 tactics, and 152,695 premises. We use 2,000 theorems for
theorem exercise_13_6_10 {K : Type*} [field K] [fintype K*] :  
 $\Pi (x:K^{\times})$ $x = -1 :=$   
begin  
exact finite_field.prod_univ.units_id_eq_neg_one,  
end  
theorem exercise_1_17  
(n : N)  
(x y : euclidean_space R (fin n)) -- R^n  
:  $\| x + y\|^2 + \| x - y\|^2 = 2*\| x\|^2 + 2*\| y\|^2 :=$   
begin  
rw [norm_add_sq_real, norm_sub.pow_two_real],  
ring,  
end  
theorem exercise_2_25 {K : Type*} [metric_space K] [compact_space K] :  
 $\exists (B : \text{set} (\text{set} K))$  , set.countable B ∧ is_topological BASIS B :=  
begin  
rcases exists_countable BASIS K with  $\langle B, hBc, hB\rangle$ ,  
exact  $(B, hBc, hB.2)$ ,  
end
Figure D: Three new proofs discovered by ReProver on ProofNet [29] that cannot be found by a baseline without premise retrieval. All of the three proofs rely on premises: "finite_field.prod_univ.units_id_eq_neg_one"
, "norm_add_sq_real", "norm_sub.pow_two_real", and "exists_countable BASIS".
validation, 2,000 theorems for testing, and the rest for training. LeanDojo Benchmark 4 also has two different data splits: random and novel_premises.
We use LeanDojo Benchmark 4 to train and evaluate our method. The model architectures and experimental details are the same as those in Sec. 6. Results on premise selection are in Table B, and results on theorem proving are in Table C.
Table B: Premise selection testing performance on LeanDojo Benchmark 4 (Lean 3 results in Table 1). We train and evaluate two models independently using different data splits (random and novel_premises). R@k is the recall for the top  $k$  retrieved premises, and MRR is the mean reciprocal rank metric.
Method	random	novel_premises
R@1	R@10	MRR	R@1	R@10	MRR
Ours	12.8	34.7	0.29	9.8	32.1	0.24
Table C: Theorem proving Pass@1 (%) on the testing data of LeanDojo Benchmark 4 (Lean 3 results in Table 2).
Method	random	novel_premises
ReProver	48.6	19.9
W/o retrieval	44.5	16.2
E ChatGPT Plugin for Theorem Proving
LeanDojo provides a general tool for interacting with Lean programmatically. As a demo of how it might bridge LLMs and theorem proving, we build a ChatGPT plugin [101] enabling ChatGPT to prove theorems by interacting with Lean through LeanDojo. Plugin developers can wrap any software
as a web service and describe its APIs to ChatGPT. Then, ChatGPT can automatically call the APIs and incorporate the results into the response to the user. Below is a summary of our API description corresponding to the interface in Sec. 4.
Description: Plugin for proving user-specified theorems automatically by interacting with Lean. The user enters information of how to find a theorem (e.g., theorem name and file path). Based on the user's input, ChatGPT first initializes the proof search with the given theorem as the initial state. Then, ChatGPT will first explain the choice for the next tactic step using LaTeX and run that tactic step to the state. If the current state is not promising, ChatGPT can backtrack to previous states by decrementing the "state_id" parameter. If applying tactics to the current state specified by the "state_id" parameter returns an error message, ChatGPT should explain the error, and if repetitive errors occur, ChatGPT should decrement the "state_id" parameter and try a different approach on a previous state. The theorem is successfully proved if there are no unsolved goals in the current state.
Endpoints:
initialize-proof_search: Given the theorem name and file path of a Lean theorem, initialize the proof search. The response includes the initial state and its state ID.   
Args: theorem_name (string): The name of the target theorem to prove. theorem_file_path (string): The file path of the target theorem.   
run_tactic: Run a tactic on a state (specified by its state ID), assuming the proof search has been initialized and some state is available. The response is either the next state and its state ID or an error message, in which ChatGPT should explain the error and consider decrementing the "state_id".   
Args: state_id (string): The ID of the state on which to run the tactic. tactic (string): The tactic to run on a state ( specified by its state ID), assuming the proof search has been initialized.
After exposing the APIs to ChatGPT, we can ask it to prove theorems by specifying the theorem's name and path in any public Lean repo on GitHub. Fig. E-L show an example with the GPT-3.5 version of ChatGPT. And Fig. M-O are the same example with the GPT-4 version. The captions provide detailed step-by-step explanations.
We highlight a few key strengths of ChatGPT observed in multiple examples we evaluated. First, unlike specialized methods for theorem proving (this paper and its prior works), ChatGPT interleaved informal mathematics with formal proof steps. This resembles how humans interact with proof assistants and opens up new avenues for integrating natural language and formal theorem proving. Second, ChatGPT demonstrated impressive capability in explaining error messages from Lean that are quite opaque even to humans. It was able to incorporate the error message to refine its proof strategy. Last, ChatGPT's behavior is more steerable than specialized provers. In Fig. E, we simply gave it the theorem to prove, but we could also provide more detailed instructions. For example, we
could say: "Please describe a high-level proof plan before trying any tactic." This kind of steerability enables future research on prompt engineering for theorem proving, and we have already seen initial benefits in an ongoing work named Sagredo.[13]
However, these strengths by no means imply ChatGPT can already solve theorem proving. In fact, it failed to find a proof for most theorems we tried. Hallucination was common. In Fig. L, ChatGPT falsely asserted the theorem was proved, while we knew it was not, by looking at LeanDojo's response. This demonstrates the value of theorem proving as a rigorous benchmark for addressing LLMs' hallucination problem. Another key limitation of ChatGPT was its inability to search systematically in a large space. We frequently found it stuck to an unpromising path when the correct solution could be found by backtracking and exploring alternative paths. This behavior is consistent with the general observation that LLMs are weak at search and planning. Addressing this weakness is an active area of research [102].
We emphasize a few caveats about our study of theorem proving with ChatGPT. First, data contamination is likely. Many theorems we evaluated have been publicly available on GitHub before ChatGPT's data cutoff date. Therefore, ChatGPT may have seen them in training. Second, our study is exploratory. A more detailed and quantitative study is needed to characterize ChatGPT's capability in theorem proving. Such a study with ChatGPT plugins is challenging, as plugins currently only support interaction through the browser. Also, OpenAI has taken measures to block automated access by bots. Using humans may be an option, but that is beyond the scope of this paper.
Figure E: (ChatGPT-3.5, 1/8) After receiving the theorem to prove, ChatGPT first called "initialize", which returned the initial state. Then it tried to interpret the theorem in natural language. Note that it made a mistake here. The theorem was about natural numbers  $(\mathbb{N})$ , not complex numbers  $(\mathbb{C})$ .
<sup>13</sup>https://www.youtube.com/watch?v=CEwRMTOGpKo
Figure F: (ChatGPT-3.5, 2/8) ChatGPT tried to rewrite the goal using the lemma “ $b + c = c + b$ ”. This was a reasonable but incorrect move. After receiving the error message from Lean, ChatGPT explained the error in natural language. Here the explanation is quite accurate, which is impressive given that the error message looks opaque to anyone not familiar with Lean.
F Limitations and Future Work
Our work is one step toward unlocking the potential of LLMs for generating verifiable formal proofs, and we see abundant space for future exploration. A learning-based prover is a complex system consisting of multiple components: data extraction, interaction with proof assistants, model training, and proof search. While navigating the design space spanned by various components, we err on the side of simplicity and efficiency, instead of pushing performance to the limit. This helps us deliver a reliable, open, and accessible system, laying the foundation for further research. There are many directions in which the system can be improved, and we discuss a few of them here.[14]
Stronger LLMs. Our backbone model, ByT5 [44], was published in 2021 and has 299M parameters, which is not very large by today's standard. Recently, there have been a plethora of open-source LLMs demonstrating strong capabilities in writing code, e.g., CodeGen [103], StarCoder [94], and CodeGeeX [104]. We are excited to see how they might impact theorem proving and, more generally, how far we can go by pushing the limit of the model/data scale.
ByT5's tokenizer-free nature helps us sidestep the difficulty with pretrained tokenizers that may not work well for Lean's Unicode-rich code. However, treating texts as raw bytes makes the sequence length much longer than necessary. Long sequences harm efficiency, as Transformers scale quadratically w.r.t. the sequence length, which may become a bigger problem when we further scale up the model. To solve the issue, it might be helpful to pretrain a customized tokenizer or adopt more advanced tokenizer-free models such as MegaByte [105].
Our ReProver model is based on the pretraining-finetuning paradigm. Recent work on instruction-following LLMs such as GPT-4 [27] has led to successes in many applications by prompting the model without any finetuning. Our preliminary results show that GPT-4 and ChatGPT (Appendix C.2
14Additional limitations: https://leandojo.readthedocs.io/en/latest/limitations.html
Figure G: (ChatGPT-3.5, 3/8) Then it tried to prove the theorem using "r-ing". This was another good move. The ring tactic can prove this theorem, but Lean could not find it since it was not imported into the current file. Again, ChatGPT was able to interpret the error message correctly and concluded that ring was not available. Next, it tried another tactic but failed again.
and E) cannot solve theorem proving out of the box and are currently far behind finetuned models. However, the way we prompt these models is quite naive, and better strategies, such as Tree of Thoughts [102], may lead to further improvements. We consider theorem proving as a promising task for studying LLMs' capabilities in planning and search.
Improving Premise Retrieval. ReProver uses DPR [26] to retrieve premises and fuses them with the current proof state by concatenation. This architecture is simple and effective but does not scale to a large number of retrieved premises. With a length limit of 2,300 tokens, we can fit only 10-15 premises into the input of the tactic generator. To mitigate the problem, we may need an architecture that fuses the retrieved premises in the hidden space, e.g., Fusion-in-Decoder [106].
Figure H: (ChatGPT-3.5, 4/8) ChatGPT made another two failed attempts. Here, the second attempt had the same problem as in Fig. E ("+") is left associative).
In addition, one can also switch from DPR to radically different retrieval architectures. For example, generative retrieval [107-109] is a recent class of models performing retrieval by directly predicting the document IDs, which could be the premise names in our task.
Limitations of Imitating Human-Written Proofs. Human-written proofs extracted by LeanDojo provide valuable data for training the prover. However, we have also observed limitations of using them as the sole training target:
First, they are relatively scarce for today's data-hungry LLMs. LeanDojo Benchmark has 98,734 proofs, covering a large portion of available data in Lean (as of October 2023). The number of proofs
Figure I: (ChatGPT-3.5, 5/8) ChatGPT tried to simplify the goal using the `simp` tactic but failed. Then it tried the "rw add_comm a b" tactic, which successfully rewrote the goal from  $\text{a + b + c} = \text{a + c + b}$  to  $\text{b + a + c = a + c + b}$ .
in other proof assistants has the same order of magnitude (tens or hundreds of thousands). Due to limited data, we cannot constantly improve the performance simply by scaling up the model size. Second, theorem proving in proof assistants is an interactive process, but the proof only captures the final successful trajectory. Without the intermediate history of trial and error, it can be quite opaque how final proofs are derived. Therefore, tactics in human-written proofs can be difficult for the model to learn from. Third, models trained on proofs in one project often struggle to generalize to theorems in new domains [9, 20], e.g., from mathlib to MiniF2F and ProofNet (Appendix C.4).
Figure J: (ChatGPT-3.5, 6/8) Then it successfully performed another rewrite from “b + a + c = a + c + b” to “b + a + c = c + a + b”.
To overcome these limitations, existing work has explored learning from auxiliary data or data collected via online interaction with the proof assistant. For example, Proof Artifact Co-Training (PACT) co-trains the tactic generator on nine auxiliary tasks, such as predicting types and theorem names [16]. MetaGen [38] trains a neural network to generate synthetic theorems/proofs as training data in the Metamath proof assistant [110]. Polu et al. [19] and Lample et al. [17] improve the prover by training it on successful proofs found by itself. Incorporating these techniques into our system may lead to substantial improvements.
Figure K: (ChatGPT-3.5, 7/8) Another rewrite tactic, transforming the goal to “a + b + c = c + a + b”.
Figure L: (ChatGPT-3.5, 8/8) It performed one last rewrite, which brought the goal back to the original goal “a + b + c = a + c + b”. At this point, ChatGPT went back to the starting point without making meaningful progress. However, it asserted that the theorem had been successfully proved, even though the response from LeanDojo clearly indicated “proof_finished: False”.
Figure M: (ChatGPT-4, 1/3) The same theorem using the GPT-4 version of ChatGPT. Despite a minor error when interpreting the theorem (natural numbers, not integers), it generated a sensible tactic, rewriting the goal to “ $a + (b + c) = a + c + b$ ”.
Figure N: (ChatGPT-4, 2/3) A good move, rewriting the goal to “a + (c + b) = a + c + b”.
Figure O: (ChatGPT-4, 3/3) It tried the ref1 tactic but failed. ref1 requires the goal to be an equation whose both sides are identical up to trivial transformations. However, “a + (c + b) = a + c + b” is not trivial since “+” is left associative. ChatGPT was able to interpret this error accurately and finish the proof using the correct premise “add_associ”.