SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation details. LeanStep and LeanDojo use different mechanisms to extract ASTs and proof trees. LeanStep implements an ad-hoc parser in Python for parsing Lean code into ASTs. It also intercepts Lean's tactic system to insert logging code. Then the logs are used to reconstruct proof trees. This implementation is brittle and does not work for the current versions of Lean/mathlib. In contrast, LeanDojo relies on Lean's built-in mechanisms for exporting ASTs and proof states (lean --ast --tsast --tspp), which works robustly for recent Lean/mathlib. This mechanism was developed after LeanStep.
$^{10}$ https://github.com/lean-dojo/LeanDojo/blob/main/tests/interface/test_unexpected Errors.py
11 LeanStep is technically a dataset. We are referring to the lean-proof_recording tool for extracting it.
Regarding interaction with Lean, both lean-gym and LeanDojo rely on Lean's metaprogramming APIs, and LeanDojo partially builds upon lean-gym's code. However, lean-gym has a critical issue in that it misjudges many correct proofs as incorrect (Appendix A.2). The main reason is that lean-gym fails to distinguish two subtly different cases when constructing the proof environment: (1) opening a namespace; (2) being inside a namespace. LeanDojo does not suffer from this issue. Instead of operating as a standalone program in the IO monad, it wraps the interaction code into a special tactic, which is inserted into the correct location in the proof. Therefore, the interaction code is guaranteed to run in the same environment as the original human-written proof.
LeanStep [16]	lean-gym [19]	LeanDojo (ours)
Data extraction	Premise information	×	N/A	✓
Lean 4 support	×	N/A	✓
Recent mathlib	×	N/A	✓
Repos other than mathlib	×	N/A	✓
Interaction	Estimated errors	N/A	21.1%	1.4%
Lean 4 support	N/A	×	✓
Recent mathlib	N/A	×	✓
Repos other than mathlib	N/A	✓	✓
Documentation & unit tests	×	×	✓
Table A: Comparing LeanDojo with existing tools for data extraction and interaction with Lean.
B LeanDojo Benchmark
We describe the data format of LeanDojo Benchmark, which has the following directory structure:
corpus.json...All premises defined in mathlib and Lean's standard library metadata.json...Metadata licenses lean...Attribution to Lean's Apache 2.0 license .mathlib...Attribution to mathlib's Apache 2.0 license README.md..Statement that LeanDojo Benchmark is released under CC BY 2.0 random.Theorems/proofs of the random split train.json...94,734 theorems val.json...2,000 theorems test.json...2,000 theorems novel PREmises Theorems/proofs of the novel PREmises split train.json...94,734 theorems val.json...2,000 theorems test.json...2,000 theorems
Premise Definitions. corpus.json contains the definition of premises. It has 3,280 lines. Each line is in JSON format and corresponds to a Lean file. Below is an example for "init/control/functorLEAN", which directly imports three other files: "init/coreLEAN", "init/functionLEAN", and "init/meta/nameLEAN". It defines two constants that can be used as premises: "functor" and "functor.map_const_rev". For each premise, we have access to its full name, the source code, and its start/end location within the file.
"path": "_target/deps/lean/library-init/control/functorLEAN",
"imports": [
    "_target/deps/lean/library-init/coreLEAN",
    "_target/deps/lean/library-init/functionLEAN",
    "_target/deps/lean/library-init/metamnameLEAN"
],
"premises": [
{ "full_name": "functor", "code": "class functor (f : Type u  $\rightarrow$  Type v) : Type (max (u+1) v) :=\n(map : II {α β : Type u}, ( $\alpha \rightarrow \beta$ )  $\rightarrow$  f α  $\rightarrow$  f β)\\n(map_const : II {α β : Type u},  $\alpha \rightarrow$  f β  $\rightarrow$  f α :=  $\lambda$  α β, map o const β),"   
"start": [11, 1],   
"end": [13, 70],   
"kind": "class"   
},   
{ "full_name": "functor.map_const_rev", "code": "@[reducible] def functor.map_const_rev {f : Type u  $\rightarrow$  Type v} [functor f] {α β : Type u}: f β  $\rightarrow$  α  $\rightarrow$  f α :=\\nλ a b, b <$ a", "start": [18, 1], "end": [19, 14], "kind": "definition" }
Theorems and Tactics. Theorems in LeanDojo Benchmark are split into training/validation/testing using two different strategies (Sec. 4). They are formatted in JSON, and below is an example corresponding to the theorem "real.angle.to_real_pi_div_two". LeanDojo has recorded two tactics: "split" and "linarith [pi_pos]". For each tactic, we have the proof states before/after it. The "linarith [pi_pos]" tactic illustrates how premises are recorded: They are annotated using HTML-like strings such as "linarith [<a>pi_pos</a>]", followed by a "provenance list". Each element in the list corresponds to a premise in the tactic.
"url": "https://github.com/leanprover-community/mathlib",  
"commit": "19c869efa56bbb8b500f2724c0b77261edbfa28c",  
"file_path": "src/analysis/special-functions/trigonometric/ angle lean",  
"full_name": "real.angle.to_real(pi_div_two)",  
"start": [512, 9],  
"end": [513, 56],  
"traced Tactics": [  
{  
    "tactic": "split",  
    "annotated_tactic": ["split", [],  
    "state_before": "-π < π / 2 ∧ π / 2 ≤ π",  
    "state_after": "2 goals\n-π < π / 2\n\n-π / 2 ≤ π"  
},  
{  
    "tactic": "linarith [pi_pos]",  
    "annotated_tactic": [  
        "linarith [<a>pi_pos</a>",  
    [  
        "full_name": "real.pi_pos",  
        "def_path": "src/analysis/special-functions/ trigonometric/basicLEAN",  
        "def_pos": [122, 7],  
    }  
}  
],  
"state_before": "-π < π / 2",  
"state_after": "no goals"
Not all theorems have tactic-style proofs. For those without tactic-style proofs, concatenating the tactics does not lead to a complete proof of the original theorem. However, this is not an issue when using the data for theorem proving evaluation or for training tactic generators.
B.2 Datasheet
We present a datasheet [90] for documentation and responsible usage of LeanDojo Benchmark.
- For what purpose was the dataset created? It was created as a benchmark for learning-based theorem proving in Lean.
- Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? It was created by the authors of this paper.
- Who funded the creation of the dataset? See the acknowledgments in Sec. 7.
- What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? The dataset consists of formal definitions, theorems, and proofs written in Lean [1].
- How many instances are there in total (of each type, if appropriate)? The dataset has 98,734 theorems and their proofs, as well as 130,262 premises defined in 3,384 files.
- Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? The dataset contains all theorems/proofs that LeanDojo can extract from the commit 19c869efa56bbb8b500f2724c0b77261edbfa28c of mathlib released on October 11, 2023.
- What data does each instance consist of? Theorems/proofs in the dataset are Lean code written by programmers and mathematicians.
- Are relationships between individual instances made explicit? Definitions in the dataset are linked to proofs using them as premises.
- Are there recommended data splits? Yes, we recommend two data splits: random and novel_premises. Please see Sec. 4 for details.
- Are there any errors, sources of noise, or redundancies in the dataset? ASTs extracted by LeanDojo contain a small number of errors due to potential flaws in Lean's AST exporting mechanism. However, they do not have a tangible impact on our work.
- Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is self-contained.
- Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? No.
- Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? No.
Collection Process.
- How was the data associated with each instance acquired? The data is directly observable by opening mathlib in VS Code with the Lean plugin. However, we had to instrument Lean to export the data programmatically.
- What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? The data was generated by building a Lean repo using our modified Lean and postprocessing the exported data.
- Who was involved in the data collection process (e.g., students, crowd workers, contractors), and how were they compensated (e.g., how much were crowd workers paid)? No manual effort was involved in the data collection process.
- Over what timeframe was the data collected? The final version of the dataset was generated in October 2023.
- Has the dataset been used for any tasks already? We have used the dataset for training and evaluating machine learning models on the tasks of premise selection and theorem proving.
- Is there a repository that links to any or all papers or systems that use the dataset? Yes, https://leandojo.org.
- Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? Yes, the dataset is publicly available on the Internet.
- How will the dataset be distributed (e.g., tarball on website, API, GitHub)? The dataset can be downloaded as a tarball.
- Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? The dataset is distributed under CC BY 2.0. The data generation code is distributed under the MIT license. The dataset was extracted from mathlib, which depends on lean. Both of them are distributed under the Apache 2.0 license. We include their licenses in the dataset as attribution (Appendix B.1).
- Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
- Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.
- Who will be supporting/hosting/maintaining the dataset? The authors of this paper.
- How can the owner/curator/manager of the dataset be contacted (e.g., email address)? Please contact Kaiyu Yang at kaiyuy@caltech.edu.
- Is there an erratum? No.
- Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? Please check https://leandojo.org for any update.
- If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Yes, they can use our data generation code, which is publicly available.
B.3 Data Hosting, Licensing, and Maintenance
LeanDojo Benchmark is distributed under the CC BY 2.0 license. The data is hosted on zenodo.org (a long-term data repository operated by CERN). The LeanDojo tool for data extraction and interaction with Lean is released at https://github.com/lean-dojo/LeanDojo under the MIT license. Our model checkpoints are hosted on Hugging Face Hub. LeanDojo's documentation is hosted on Read the Docs at https://leandrojo.readthedocs.io. LeanDojo's website (https://leandrojo.org) is the entry point for everything related to it, including any future updates or maintenance.
C.1 Details and Hyperparameters
The premise retriever and tactic generator in ReProver are initialized by the google/byt5-small1 checkpoint on Hugging Face. It is a T5-like [97] encoder-decoder Transformer that operates directly
on UTF-8 bytes without tokenization. We choose ByT5 [44] instead of T5 because Lean code makes extensive use of Unicode math symbols, which may cause problems to T5's pretrained tokenizer. The retriever uses the encoder only, whereas the generator uses both the encoder and the decoder.
In training, we use one NVIDIA A100 GPU with 80GB of memory. The code is implemented in PyTorch and PyTorch Lightning, with bfloat16 mixed precision and DeepSpeed ZeRO Stage 2 [98]. Both the retriever and the generator are optimized using AdamW [99] with a batch size of 8. In the first 2,000 steps, the learning rate warms up linearly from 0 to the maximum value. Then it decays to 0 following a cosine schedule. The maximum learning rate is  $10^{-4}$  for the retriever and  $5 \times 10^{-4}$  for the generator. When training the retriever, we sample 3 negative premises for each example, including 1 in-file negative premise. When training the generator, we apply dropout to retrieved premises with a dropout rate of 0.5. Then, we truncate the generator's input to 2,300 tokens.
During evaluation, the tactic generator is combined with best-first search to find proofs. At each search step, it produces 64 tactic candidates using beam search. Each tactic is associated with a log-likelihood score. In best-first search, we prioritize the states by the sum of log-likelihoods of tactics leading to that state.
C.2 The GPT-4 Baseline
Now we describe the GPT-4 [27] baseline in Sec. 6. Similar to ReProver, it is a tactic generator combined with best-first search. However, the tactic generator is based on GPT-4's capability to follow instructions in zero-shot. Specifically, given a proof state, we use the following prompt to instruct GPT-4 to produce a list of tactics, each paired with a confidence score:
Prompt Template:
You are an expert in Lean3 theorem proofs. We are trying to solve the Lean3 theorem 'THEOREM_FULL_NAME' from the mathlib file 'FILE_PATH'. The current tactic state is: 'TACTIC_STATE'. Suggest exactly 35 unique tactics to progress in solving 'THEOREM_FULL_NAME', along with their confidence levels as a float between 0 and 1. Rank them in order of effectiveness. Present the tactics and their confidence levels as comma-separated tuples in this format: #(tactic_{1}, confidence_{1})#, #(tactic_{2}, confidence_{2})#, ..., #(tactic_{35}, confidence_{35})#.
We adapted the prompt to a particular theorem and state by substituting the variables with the appropriate values. Given the inherent variability in GPT-4's outputs, we requested 35 and filtered out invalid ones. We used a token length limit of 1,024 and kept all other API parameters at their default values. Below are a few example prompts and GPT-4's responses:
Example Prompt 1:
You are an expert in Lean3 theorem proofs. We are trying to solve the Lean3 theorem 'polynomial.chebyshev.aeval_U' from the mathlib file 'mathlib/src/analysis/special-functions/trigonometric/chebyshevLean'. The current tactic state is: 'R:Type u_1,A:Type u_2,_inst_1 : comm_ring R,_inst_2: comm_ring A,_inst_3: algebra R A,x : A,n:N $\vdash$  (aeval x) (chebyshev.U R n) = eval x (chebyshev.U A n). Suggest exactly 35 unique tactics to progress in solving 'polynomial.chebyshev.aeval_U', along with their confidence levels as a float between 0 and 1. Rank them in order of effectiveness. Present the tactics and their confidence levels as comma-separated tuples in this format: #(tactic_{1},confidence_{1})#, #(tactic_{2},confidence_{2})#, ..., #(tactic_{35},confidence_{35})#.
Example Response 1:
#(rw chebyshev.U, 0.9)#, #(simp only [chebyshev.U], 0.9)#, #(simp_rw chebyshev.U, 0.8)#, #(apply aeval_add, 0.75)#, #(apply aeval.mul, 0.75)#, #(apply aeval_sub, 0.75)#, #(apply aeval_X, 0.7)#, #(apply aeval_C, 0.7)#, #(apply aeval.pow, 0.7)#, #(rw \u2190aeval.pow, 0.65)#, #(rw 2190aeval.mul, 0.65)#, #(rw \u2190aeval_add, 0.65)#, #(rw \u2190aeval_sub, 0.65)#, #(apply algebra_map_eq, 0.6)#, #(intro aeval_algebra_map_eq, 0.6)#, #(apply eval_map, 0.55)#, #(rw eval_map, 0.55)#, #(apply aeval_eq.eval, 0.55)#, #(rw_for_target aeval_eq.eval, 0.5)#, #(apply aeval_eq.eval', 0.5)#, #(rw_for_target aeval_eq.eval', 0.5)#, #(apply aeval_commutes, 0.45)#, #(rw aeval_commutes, 0.45)#, #(apply aeval_eq_of_isScalar_tower, 0.4)#, #(rw isScalar_tower.aeval_apply, 0.4)#, #((try_for_100 exact_modCAST chebyshev.U_eq_refl, 0.35)#, #(simp * at *, 0.3)#, #(simp int.coe_nat ringinghom, 0.25)#, #(apply chebyshev.U_is_polynomial, 0.2)#, #(rw_algebra_map_eq_of ringinghom, 0.15)#, #(unfold chebyshev.U, 0.1)#, #(congr, 0.05)#, #(change (chebyshev.U R n).eval x = chebyshev.U A n, 0.05)#, #(change (\u21d1(aeval x) (chebyshev.U R n) = chebyshev.U A n), 0.05)#, #(apply_fun aeval x, 0.05)#, #(apply_fun eval x, 0.05)#, #(simp context_md := reducible [], 0.01)#
Example Prompt 2:
You are an expert in Lean3 theorem proofs. We are trying to solve the Lean3 theorem 'euler_sine.integral_cos_pow_pos' from the math-lib file 'mathlib/src/analysis/special-functions/trigonometric/euler_sine_PROD lean'. The current tactic state is: 'n:  $\mathbb{N} \vdash 0 < \int (\mathbf{x}: \mathbb{R})$  in  $0..\pi / 2$ , cos x ^ n'. Suggest exactly 35 unique tactics to progress in solving 'euler_sine.integral_cos_pow_pos', along with their confidence levels as a float between 0 and 1. Rank them in order of effectiveness. Present the tactics and their confidence levels as comma-separated tuples in this format: #(tactic_1, confidence_1)#, #(tactic_2, confidence_2)#, ..., #(tactic_35, confidence_35)#.
Example Response 2:
(#(exact integral_pos_of.antimono_increasing_on cos.pow.antimono_increasing_on.cos_even_Nreal, 0.99)#, #(apply integral_pos_of.antimono_increasing_on, 0.95)#, #(interval_integrableIntegral_pos, 0.90)#, #(exact integral_pos_pre, 0.85)#, #(interval_integ_valuesout, 0.80)#, #(interval_integrable_pos, 0.75)#, #(exact integral_pos_of_nonneg_dint_val, 0.70)#, #(exact interval_integrable_pos_left, 0.65)#, #(exact interval_integrable_pos_right, 0.60)#, #(exact integral_pos_of_nonneg_f, 0.55)#, #(rw integral_on_interval, 0.50)#, #(rw intervalIntegral_pos_of_supporting, 0.45)#, #(rw integral_pos_of_interval_integrable_pos, 0.40)#, #(apply integral_pos_of.antimono_increasing_on_cos_even_Nreal, 0.35)#, #(norm_cast apply_pos, 0.30)#, #(apply_pos_of_nonneg_pos, 0.25)#, #(rw intervalIntegral_pos_of.antimono, 0.20)#, #(use cos Slowmono_on_pos, 0.15)#, #(apply cos SlowMono_on_pos, 0.10)#, #(exact integral_pos_preCos_pow_even, 0.09)#, #(norm_cast apply_nonneg_pos, 0.08)#, #(rw integral_pos_of_monotone_integrable, 0.07)#, #(use cos_pos_constant_sequence, 0.06)#, #(apply cos_pos_constant_sequence, 0.05)#, #(use integral_pos_ofascending_sequence, 0.04)#, #(apply integral_pos_ofAscending_sequence, 0.03)#, #(intervalIntegral_pos_of.antimono_increasing_on_pos, 0.02)#, #(norm_cast apply_pos_of_nonzero_pos, 0.01)#, #(apply_pos_of_nonzero_pos, 0.005)#, (#rw integral_pos_of_nonneg_f_monotone, 0.001)#
Data contamination is possible. Our GPT-4 experiments were performed in 2023, but many theorems and proofs in the dataset have been publicly available on GitHub before GPT-4's data cutoff date (September 2021).
C.3 Justifications for Not Comparing with Existing LLM-Based Provers
In Table 2, we do not empirically compare ReProver with any existing LLM-based prover. Unfortunately, such a comparison is infeasible. Provers targeting different proof assistants are generally not comparable, so we focus the discussion on the three existing provers in Lean [16, 17, 19]. Most importantly, they are impossible to reproduce with reasonable effort, due to private code and pretraining data. Therefore, the only potential comparison is to evaluate ReProver under their experimental settings and compare with the numbers reported in their papers. However, that is also impractical for numerous reasons:
- The data is different. All existing methods used an outdated version of mathlib more than two years ago. We cannot use LeanDojo to extract data from this version. As mentioned in Sec. 4, LeanDojo only supports repos released after March 24, 2022. Also, we cannot use their dataset directly, since it does not contain premise information required by ReProver.
- Lample et al. [17] trained on a synthetic dataset named Equations, which is not publicly available.
- All existing methods co-train the tactic generator on auxiliary tasks from the PACT dataset [16]. Co-training increases the data/compute requirements by an order of magnitude, which cannot be afforded by us (or probably most academic labs). All existing methods were developed by researchers in the industry.
- Polu et al. [19] and Lample et al. [17] further finetuned their models on new proofs collected through online interaction with Lean, whereas our method is only trained on human-written proofs.
- The tool for interacting with Lean may impact the performance. Han et al. [16] and Polu et al. [19] used lean-gym, which has severe limitations (Appendix A.2). Lample et al. [17] developed their own private tool, which is not publicly available.
Most of these difficulties are due to the private nature of existing methods. By releasing our code and models, we take a major step in establishing accessible baselines for future work to build upon.
C.4 Evaluation on MiniF2F and ProofNet
We evaluate our ReProver model on MiniF2F [28] and ProofNet [29] (Sec. 6) to test its capability in proving theorems outside its training data distribution. We use the same hyperparameters and evaluation setup as the previous experiments (Appendix C.1).
MiniF2F. We use the commit 5271ddec788677c815cf818a06f368ef6498a106 of Meta's version of MiniF2F [17]. ReProver achieves a Pass@1 of  $26.5\%$  on the test set, which is competitive with state-of-the-art methods without reinforcement learning (25.9% in Polu et al. [19]). Moreover, ReProver can prove 33 theorems that currently do not have Lean proofs (examples in Fig. B). For the complete list of 33 new proofs, please see our pull request to MiniF2F.
There are caveats about quantitatively comparing ReProver with existing methods on MiniF2F. Many difficulties in Appendix C.3 still apply, e.g., different tools for interacting with Lean may impact the performance. Also, MiniF2F is a test-only dataset without training theorems, and existing methods focus on reinforcement learning (RL) to learn from proofs collected via online interaction with the proof assistant [17, 19]. In contrast, ReProver is trained via supervised learning on a static dataset, so we only compare with the non-RL baseline in existing methods (Polu et al. [19] achieves a Pass@1 of  $25.9\%$  without RL and  $29.6\%$  with RL). Furthermore, we do not compare with Lample et al. [17] due to differences in the evaluation metric. They use Pass@64, which requires running the prover on each theorem 64 times. We use Pass@1, and it already takes one day for a single evaluation on MiniF2F's test set. Therefore, evaluating Pass@64 would be too computationally expensive for the resources we have access to. Finally, MiniF2F is available in multiple proof assistants [18, 69, 70]. Results across different proof assistants are not comparable, so we only compare with existing work in Lean.
ProofNet. We use the commit e8645aa830ce17c33a8b8482a8195f0f97d6a74a of ProofNet. ReProver can prove 48 out of 349 theorems, achieving a Pass@1 of  $13.8\%$ , which is the first reported
theorem mathd_numbertheory_237 :  
(∑ k in (finset(range 101), k) % 6 = 4 :=  
begin  
rw [finset-sum_range_succ'],  
norm_num [finset-sum_range_succ],  
end  
theorem mathd_numbertheory_175 :  
(2^2010) % 10 = 4 :=  
begin  
norm_num [pow_succ],  
end  
theorem mathd_numbertheory_293  
(n : N)  
(h₀ : n ≤ 9)  
(h₁ : 11|20 * 100 + 10 * n + 7) :  
n = 5 :=  
begin  
contrapose! h₁,  
norm_num [h₁],  
dec_trivial!,  
end  
theorem mathd_algebra_616  
(f g : R → R)  
(h₀ : ∀ x, f x = x^3 + 2 * x + 1)  
(h₁ : ∀ x, g x = x - 1) :  
f (g 1) = 1 :=  
begin  
simp only [h₀, h₁, pow_one],  
ring,  
end
Figure B: Examples of new proofs discovered by ReProver on MiniF2F [28].
theorem proving result on ProofNet. Moreover, 39 out of the 48 proved theorems do not have existing Lean proofs (examples in Fig. C), and 3 of them can only be proved with the help of premise retrieval (Fig. D). We have contributed the 39 new proofs to ProofNet, which helped them reveal and fix problems in the formalization of 7 theorems (details in our pull request).
D LeanDojo for Lean 4