A Survey on Deep Learning for Theorem Proving
Authors: Zhaoyu Li
Jialiang Sun
Logan Murphy
Qidong Su
Zenan Li
Xian Zhang
Kaiyu Yang
Xujie Si
Venue: COLM (2024)
Keywords: Deep Learning, Theorem Proving, Automated Reasoning
TL;DR: A comprehensive survey of deep learning for theorem proving.

Abstract: Theorem proving is a fundamental aspect of mathematics, spanning from informal reasoning in natural language to rigorous derivations in formal systems. In recent years, the advancement of deep learning, especially the emergence of large language models, has sparked a notable surge of research exploring these techniques to enhance the process of theorem proving. This paper presents a comprehensive survey of deep learning for theorem proving by offering (i) a thorough review of existing approaches across various tasks such as autoformalization, premise selection, proofstep generation, and proof search; (ii) an extensive summary of curated datasets and strategies for synthetic data generation; (iii) a detailed analysis of evaluation metrics and the performance of state-of-the-art methods; and (iv) a critical discussion on the persistent challenges and the promising avenues for future exploration. Our survey aims to serve as a foundational reference for deep learning approaches in theorem proving, inspiring and catalyzing further research endeavors in this rapidly growing field. A curated list of papers is available at https://github.com/zhaoyu-li/DL4TP.

Published as a conference paper at COLM 2024
A Survey on Deep Learning for Theorem Proving
Zhaoyu Li', Jialiang Sun', Logan Murphy', Qidong Su', Zenan Li?, Xian Zhang'
Kaiyu Yang", Xujie Si1,5
University of Toronto, NNajjing University, Microsoft Research Asia, "Meta FAIR, SCIFAR AI Chair {zhaoyu, six}ecs. coronto.edu
Abstract
Theorem proving is a fundamental aspect of mathematics, aaahematics,spanninng from informal reasoning in natural language to rigorous derivations in formal systems. In recent years, the advancement of deep learning, especially the emergence of large language models, has sparked notable surge of research exploring these techniques to enhance the process of theorem proving. This paper presents comprehensive survey of deep learning for theorem proving by offering (i) thorough review of existing approaches across various tasks such as autoformalization, premise selection, proofstep generation, and proof search; (ii) an extensive summary of curated datasets and strategies for synthetic data generation; (iii) a detailed analysis of evaluation metrics and the performance of faate-of-the-art methods; and (iv) a critical discussion on the persistent challenges and the promising avenues for future exploration. Our survey aims to serve as foundational reference for deep learning approaches in theorem proving, inspiring and catalyzing further research endeavors in this rapidly growing field. A curated list of papers is available at //tttt:ss/gthhhhi http:///thhu.ccm/hhooooooo/l/a//
com/ 2hayyyyoooou1
1 Introduction
Proving theorems is cornerstone of mathematics, Since the era of Euclid around 300 B.C., people have crafted theorems and proofs using a blend of natural language and mathemati- cal symbols, meticulously evaluating their correctness through manual inspection. In the 1950s, paradigm shift occurred with the exploration of computer-assisted proofs (Davis) 1957: Davis & Putnam 1960), wherein a machine automatically applies deduction rules to prove assertions. These innovations laid the groundwork for the subsequent development of interactive theorem provers (Bruijn, de 1970 Milner 1972), enabling people to construct more intricate theorems and proots by interacting with these systems. Building upon these advancements, later research extended the scope of theorem proving beyond mathematics,
applying it to various practical applications such as software verification (Schumann) 2001 and hardware design (Kern & Greenstreet 1999).
Exploring baaming-based approaches for theorem prov- ing has been long-standing research focus, dating back to the 1990s (Suttner & Ertel 1990 Denzinger et al. 1999). The recent development of deep learning, especially with the evolution of large language models (LLMs), has ignited a wave of research interest in this area again. As shown in Figure [ the volume of papers on deep learning for theorem proving has grown ap- proximately from 2 in 2016 to 45 in 2023, continuing to rise 2024. However, despite such remarkable growth, this domain is characterized by wide range of tasks, methods, datasets, and evaluations, which lack a cohesive framework to comprehend the true extent of progress and identify the underlying challenges and potential future work.
Papers
12
2035 2017 2018
2019
2010
2024 Year
2003
2002 2003
0an.
Figure 1: Papers on deep learning for theorem proving over the years (data for 2024 is up to July).
* Research conducted while Kaiyu Yang was Caltech.
1

---

Published as a conference paper at COLM 2024
Theorem
Proof
bo leogrn laege supordd the cenigeppptleeinee fnreele Harlacksss
Tere
Premismn
Remustannttess Lsen no, luppted Tecgnss (i 4
fartors
P.add
theeren
pr Factors (s No 1p
gool and frastars
facters it Jrat,
Ihl
sipp. l prise
sing Balle
Fiw 1ist (facters
+ Peee I, Priee
factors Prite P
Thoorem
reecres untton C 19 c Lite facttrs by
relune fattt
J.yy
joust bb
Duci far
Proof
reftine perw oftiprod 103 reture"
Sprodd factors Jyseer
prod
Polsc
laeod cor LIFT thes. trancurncarre
his B
<iippuu I- grraetrrt
Figure 2: Top: The informal statement and proof of the Fundamental Theorem of Arithmetic in ProofWiki Bottom Left: The formal statement and proof of the same theorem in the mathlib library (mathlib Community 2020) of Lean 4. Bottom Right: The corresponding proof tree illustrating the tormal proof process in Lean 4. Only changes in the local context of each node are marked for clarity. The references and premises used in the informal and formal proof are highlighted by underlines and colors respectively.
In this paper, we provide a comprehensive survey of more than 180 research papers in deep learning for theorem proving, aiming to map out the current research landscape and highlight key advancements systematically. We begin with the background for informal and formal settings of theorem proving ($2). Subsequently,
delve into the details of the tasks and methods within this domain ($3 which include autoformalization, premise selection, proofstep generation, proof search, and other tasks. We also review the datasets for theorem proving ($4), including manually curated and synthetically generated ones. Moreover, we examine the evaluation metrics and assess state-of-the performance ($5). Following this, we discuss the prevailing challenges and conclude with future directions ($6).
2 Background
In this section, we recall some fundamental concepts of theorem proving, including both informal and formal settings. Figure 2 shows an illustrative example of these two settings.
2.1 Informal Theorem Proving
Informal theorem proving involves establishing the truth of mathematical statements build- ing existing knowledge via intuitive reasoning and natural language explanations. This mirrors how people learn and prove theorems in everyday mathematics. For instance, to prove the Fundamental Theorem of Arithmetic in Figure| 2 (Top), one needs to comprehend basic concepts like primes and might apply established results to draw a conclusion. Despite the ubiquity of informal theorem proving, mathematics evolves, the theories and proofs tend to be more intricate, making verifying their correctness increasingly difficult.
2.2 Formal Theorem Proving
Formal theorem proving represents theorems and proofs in a machine-verifiable format, en- suring their correctness using rigorous logical rules. This field can be broadly classified into two paradigms: automated theorem proving (ATP) and interactive theorem proving (TTP).
ATP aims to verify formal statements fully automatically. Saturation-based theorem provers, including E (Schulz) 2002) and Vampire (Kovács& & Voronkov 2013), mainly operate on first- order logic (FOL) to autonomously generate logical consequences from set of axioms until proof or refutation is derived, or computational limits are reached. Similarly, geometric ATP systems such GEX (Chou et al. 2000) prove geometry problems by iteratively applying
2

---

Published as a conference paper at COLM 2024
deduction rules. Other approaches, such as tableau-based methods like leanCoP (Otten & Bibel 2003) and instantiation- methods like iProver (Korovin 2008), use other forms of proof calculi for proof construction. In addition to these, Boolean satisfiability (SAT) solvers (e.g,, MiniSat (Fén & Sörensson 2003), CaDiCal. (Biere. 2019)) and satisfiability modulo theories (SMT) solvers (e.g., Z3 (De Moura & Bjørner 2008), CVC5 (Barbosa et al.) 2 play a crucial role in ATP by efficiently handling propositional logic and theories such as arithmetic, bit-vectors, and arrays. Despite the sophisticated designs of these ATP systems, the inherent vast search space often limits their practicality in more complex problems.
In ITP, humans collaboratively prove theorems by interacting with proof assistants, such as Isabelle (Paulson 1994), HOL Light (Harrison, 1996), Cog (Bartas et al. 1999), Meta- math (Megill & Wheeler 2019), and Lean (Moura Ullrich, Theie-These proot assistants typically enable users to formalize theorems in higher-order logic and provide language (Bottom Left), to prove theorem (initial
to build verifiable proofs. As shown in Figure 2 goal) in Lean, one can use factics like refine' and rw as the proof steps. Applying tactic either finishes the goal or decomposes it into simpler sub-goals, and the proof is complete when no further goals remain. When proving the current goal, one can apply assumptions in the local context and previously proven premises in the environment as tactic arguments. For example, the premise perm of prod eq prod is used as the argument of refine'. The proving process can be modeled as proof tree, where each node is a proof state with a goal and its local context, and each edge is tactic, as shown in Figure 2 proof assistants, researchers have successfully formalized and proved landmark theorems like the Four Color Theorem (Gonthier 2008) and the Kepler Conjecture (Hales al. 2017) and verified the correctness of critical software such as the seL4 microkerne! (Klein et al. 2009) and the CompCert C compiler (Leroy et al. 2016). However, it is worth noting that these projects took several Ph.D. years to complete, requiring substantial labor and expertise.
(Bottom Right). Using
3 Tasks and Methods
The emergence of deep learning has opened new avenues for the landscape of theorem proving, either enhancing or substituting traditional components involved in the process. This section categorizes and summarizes existing deep learning approaches into 5 tasks: autoformalization, premise selection, proofstep generation, proof search, and others.
3.1 Autoformalization
Autoformalization aims to convert informal theorems and proofs into machine-verifiable formats automatically. This task is notoriously challenging, requiring profound under- standing of semantics across informal and formal mathematics (Kaliszyk et al. 2014 2015).
Nonetheless, the success of autoformalization promises to facilitate the verification of math- ematical papers and pave the way for general-purpose reasoning engines (Szegedy 2020).
Wang et al. 20118 2020) first explore using deep learning models in autoformalization.
models in neural machine translation (Sutskever et al. 2014: Cho al. 2014), they experiment various encoder-decoder frameworks (Luong al.
Inspired by the sequenc--o-ssqqeene
2017
Lample et al. 2018: Lample& & Conneaul 2019) to convert L'IleX-witten mathematical
problem texts to the Mizar language (Rudnicki 1992.. Subsequent studies (Bansal Szegedy
2020
Cunningham et al. 2022) utilize similar neural architectures for HOL Light and Coq.
The recent development of LLMs and their in-context learning capabilities (Brown et al.,
2020) have provided new opportunities for autoformalization. Some researchers study
the potential of advanced L.MSs using few-shot prompting techniques: [Wu al. (2022);
Agrawal et al. 2002); Gadgil et al. (2022) leverage PaL.M (Chowdhery et al., (023)) and
Codex (Chen et al. 2021) to translate high school and undergradduate-leve mathematical
problems into Isabelle and Lean. LeanEuclid (Murphy et al. 2024) uses GPT-4 (Achiam et al. 2023) and GPT-4V to formalize both theorems and proots in Euclidean geometry. Other research efforts (Jiang et al.) 2023b Patel et al. 2023 Zhao et al., 2024 Lu et al. 2024 Ying let al. 2024aa Poiroux et al. 2024) propose more structured approaches to autotormalization
For example, DSP (Jiang et al." 2023b) utilizes Minerva (Lewkowyez et al. 2022) to draft
3

---

Published as a conference paper at COLM 2024
informal proofs and map them into formal sketches, with ATP systems employed to fill in the missing details in the proof sketch. Zhao et al. (2024) improves informal proofs and formal sketches in DSP with sub-goal proots and prompt selection respectively. Additionally,
a line of research (Azerbayev et al. 2023: Jiang et al. 2023a Azerbayev et al. 2024| Shao et al.
2024; Ying et al.] (0024b) locuses on training LLMS on large-scale datasets containing both informal and formal mathematical data to evaluate their performance on autoformalization
Recent studies (Liu et al. 2023 Pan et al. 2023 Olausson et al. 2023 Ye et al. 2023 Zhou
et at. 2024a Huang et al. 2024 Xin et al. 2024b Jiang et al. 2024 Quan et al. 2024 Xin et al.
2024a) also apply autoformalization as key step in various downstream tasks. For instance,
SAT-LM (Ye et 2023) uses LLMs to formalize natural language problems using declarative
prompting and solve them using Z3 on several reasoning tasks. DTV (Zhou et al. 2024a) leverages autoformalization to ground LLM reasoning, formalizing L. answers and verifying them with ATP tools. Besides these efforts, WW et al. 2022); Azerbayev et al.
(2033) ; [iang et al. (202a)); Lu et al. (2024) explore advanced LLMs like Codex and GPT-4 for intormalization, l.e., the translation of formal statements into natural language.
3.2 Premise Selection
Given a large collection of previously proven lemmas, premise selection is to retrieve the helpful lemmas that can contribute to successful proof. an enduring challenge in both mathematical research and ATP /TP/ITP systems (Kühlwein et al. 2012 Alama et al. 2014).
The seminal works (Irving et 2016 Kaliszyk al. 2017 model premise selection as a binary classification task, embedding theorems and premises with a variety of neural networks including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and hybrid models. These embeddings then combined to feed into logit layer to predict their relevance. Follow-up works (Kucik & Korovin, 2018 Bansal et al., 2019]
Piotrowski & Urban 2020a Proroković et al., 2021; Szegedy et al. 2022 extend previous frameworks by using a better representation of features or more sophisticated architectures like Wavenet (Van Den Oord et al. 2016) and Transformer (Vaswani et al. 2017).
Given the inherent structured nature of mathematical formulas, a stream of research (Wang et al. 2017 Peng & Ma 2017 Olšák et al. | Goertzel
| & Urban 2019 Crouse et al. 2019
Paliwal et al., 2020; Rawson & Reger] (2020] Liu et 2022a Goertzel et al. 2022 Holden &
Korovin 2023 Jakubuy al. 2023) parses formal statements into trees or graphs and lever-
ages tree-structured neural networks (Tai et al. 2015) or graph neural networks (GNNs) (Du- venaud et al. 2015; Veličković et al. 2018 Xu et al., 2019 for encoding. For example, Formula (Wang et al. 2017) proposes graph embedding method that preserves the information of edge ordering. Olšák et al. (2019) introduces a GNN framework that captures several logical invariances in FOL formulas. Paliwal et al. (2020) conducts comprehensive experiments to evaluate various designs for the graph representations of formulas in HOL 2021b Lin et al. 2021) explore graph contrastive learn- Light. Subsequent works (Li et al.
ing (Oord et al.) 2018 Chen et al. 2020) to train GNNs for premise selection. Moreover, Ferreira & Freitas (2020b); Bauer et al. (202)) construct dependency graph over large
corpus by representing theorems and premises as nodes and their dependencies as edges, and leverage GNNs to predict the link between nodes for premise selection.
With the advancement of pre-trained language models, some efforts (Ferreira & Freitas 2020a: Welleck et al., 2021) fine-tune BERT (Devlin et al. 2019)-like models to embed natural
language statements into vectors and select premises using a linear classifier layer. Later
works (Ferreira & Freitas 2021 Tran et al. 2022 Trust al. 2022: Dastgheib & Asgari 2022
Yeh et al.) 2023 Yang et al.) 2023) leverage different pre-trained models (Liu et al.) 2019
Song et al. 2020 Xue et al. (022) for encoding and retrieve informal/formal premises based
on several similarity metrics. Specifically, ReProver (Yang et al.) 2023) trains its retriever
based on dense passage retrieval (DPR) (Karpukhin et al. 2020) to select premises in Lean.
Han et al. (2021) also explores fine-tuning over large informal mathematical corpora using the contrastive objective (Oord et al. 2018), while PACT (Han et al., 2022) uses the auto- regressive objective for formal premise selection. Additionally, several research (Kovriguina let al., 2022: Tworkowski et al. 2022 Mikula et al. (024) design a second phase to re-rank the selected subset of premises, enabling more accurate selection.
4

---

Published as a conference paper at COLM 2024
3.3 Proofstep Generation
Proofstep generation is the core problem for theorem proving, which aims to predict one or more steps to build the proof of a theorem. This task also refers to tactic prediction in ITP, which has been widely studied in tactic-based ATP systems (hammers) (Böhme & Nipkow
2010 Blanchette et al. 2016 Czajka & Kaliszyk 2018
A stream of research (Whalen 2016 Huang et al. 2019 Bansal al. 2019 Paliwal et al.
Sanchez-Senn et al. 2020 Wu et al. 2021b Rute et al. 2024) treats tactic prediction
as a classification problem and uses separate neural networks predict the tactic and its
arguments. For example, Gamepad (Huang et al. 2019) employs TreeLSTM (Tai et al. 2015
to encode the proof states and two distinct linear layers for tactic and argument prediction.
Proverbot9001 (Sanchez-Stern et al. (0200) uses feed-forward neural network and an RNN to predict the tactic and its arguments respectively. Besides these works, ASTactic (Yang)
& Deng 2019) proposes a decoder that generates the tactic as program, using an RNN to control the generation based on a predefined context-free grammar. Later studies (First al.
2020 First & Brun 2022 Sanchez-Stern et al. 2023 improve ASTactic by incorporating
prior proof scripts, combining varied models, and modeling identifiers of theorems.
Subsequent advancements (Polu & Sutskever, 2020:
Polu et al. 2023 Han et al., 2022 Jiang
et al. 2021 Zhang et al.] 2023a] Yeh et al. 2023:
Xiong et al. [0033 Wang et al. 2023a
Vishwakarma & Mishra (223)) Gloeckle et al.
2023
First et al. 2023) Xin et al. al.] 2024a Wang
et al. 2024 Lin et al. 20244a] Wu et al. 2024) formulate tactic prediction as language modeling,
Specifically, GPT-f Polu & Sutskever (020) first apply conditional language modeling
objective to train decoder-only Transformers to generate a proof step in the format of GOAL <GOAL> PROOF STEP EPRCOFSTEP><EOT: Baldur (First et al.) 2023) applies a similar objective
to generate or repair the whole proof at once. POETRY (Wang et al. 2024) introduces level-by-level approach, recursively generating a formal sketch of the proof at each level and solving the current level's s theorem or conjecture until the proof is complete. Several
studies (Szegedy et al. 2021; Tworkowski et al. 2022 Welleck et al. 2022 Jiang et al., 2022 Yang et al.) 2023) also jointly train tactic prediction with premise selection. For instance, Naturall'rover (Welleck et al., 2022) trains GPT-3 (Brown et al., 2020) with constrained decoding to encourage using retrieved references in the proof steps. Thor (Jiang et al. 2022) adds a <hammer> token to learn when to invoke a hammer system (Böhme & Nipkow
2010) for premise selection to simplify the proof. In the geometry domain, CCen et al.
(2022) Liang et al. (2023); Trinh et al. (2024) follow the same paradigm, auto-regressively
generating the proof sequence at each step. Notably, AlphaGeometry (Irinh et al. 2024 trains decoder-only Transformer to predict the auxiliary constructions in the proots of International Mathematical Olympiad (IMO) geometry problems. Besides training on proof data, Azerbayev et al. (2024); Shao et al. Ying) et al. (2024b) train LLMs on extensive general mathematical corpora and evaluate their abilities for generating formal proofs.
With the development of LLMs, some researchers (Zhang et al. 2023b Yousefzadeh & Cao
2023 Scheidt 2023 Frieder et al. 2/23a/bcc Zhang et al. [2024] Poulsen et al. (0224) also
explore prompting state-ot-the-art LLMs without any additional training to generate proofs across various domains. For example, Frieder et al. (2023b) evaluates the performance of ChatGPT and GPTT-4 on completing intormal mathematical proofs, and Selene (Zhang) eet al. 2024) focuses on project-level automated proof in software verification based on the seL.4 project. Recent explorations further propose more structured pipelines for formal
proof generation (Jiang et al.) 2023b Zhao et al. 2024 Zheng et al., 2024 Xin et al., 2024b Huang et al. 2024: Thakur al. 2024): For instance, Lyra (Zheng et al. 2024) leverages two correction strategies, namely tool correction and conjecture correction, as post-processing and error feedback mechanisms to refine incorrect proofs generated by GPT-4.
3.4 Proof Search
Proof search seeks to systematically traverse the vast landscape of potential proof paths to construct a valid proof tree for given theorem in formal systems. It is not only long- standing research focus in ATP (Urban et al. 2011 Kaliszyk & Urban 2015a Jakubüy & Urban 2017) but also vital process for factic-based models to complete the proof in ITP.
5

---

Published as a conference paper at COLM 2024
A thread of research treats branching as classification task and trains deep learning models on successful proof paths in a supervised fashion to guide the search in various ATP systems.
Specifically, large body of works (Loos et al. 2017] Chvalovský et al. 2019 Jakubüy &
Urban 2019 Aygün et al. 2020 Jakubuy et al. 2020 Suda 2021b Chvalovsky et al. 2021
Goertzel et 202T Suda 202Ta; Firoiu et al. 2
Aygün et al. [0022] Goerizel et al. 2022
Jakubuy et al. 2023: Bartek & Suda 2023) exploit various RNNs, GNNs, or hybrid models
for the clause selection in saturation-based provers. Similarly, l'iepenbrock et al. (022b);
Chvalovský et al. (2023) and Piotrowski & Urban (2020b) focus on guiding instantiation
and connection tableau respectively. Some works (Rawson & Reger 2019 Olšák et al.
2019: Rawson & Reger 2021; Zombori et al., 2021b Wei et 2024) further combine the supervised trained models as the policy or value networks to guide the Monte Carlo Tree Search (MCTS) or A search across various ATP systems. Additionally, another direction
of research (Kusumoto et al. 2018 et al. 2019 Abdelaziz et al., 2020 Zombori et al.
2021a Crouse et al. 202 1 Piepenbrock et al. 2021; 2022a Liu et al.) 2022b; Abdelaziz et al.
2022 Fokoue et al. 2023; McKeown & Sutclitte 2003 /Shminke] (023) models proot search
as a Markov decision process and applies reinforcement learning (RL) to train and guide the proof search. For instance, TRAIL (Crouse et al. 2021) uses the policy gradient (Sutton 1999) to train an attention-based action policy in saturation-based provers, and Fawzi let al.
et al. 2019) applies deep Q-learning (Mnih et al. 2013) to guide the choice of inference rules in a semi-algebraic proof system (Lovász & Schrijver 1991) for polynomial inequalities.
Most tactic-based models in ITPPs use beam search to sample multiple tactic predictions per step with breadth-first (Bansal et al., 2019), depth-first (Yang & Dengg 2019), or best-first heuristics (Polu & Sutskever (2020) to traverse the search space. In particular, GPT-f (Polu
& Sutskever 2020) and FMSCL (Polu et al. 2023 train language models with outcome and proof size objectives as value functions to perform the best-first search. Besides these
meaalds,WWalen
Whalen (2016); Mo et al. (2020); Gauthier 2020 Wu et al. (2021a); Gauthier 22021} ;
Lample et al. (2022); Wang et al. (2023a) ; Brandfonbrener et al. (0024) combine MCTS or
use RL to train and guide the search procedure. For example, TacticZero (Wu et al. 2021a
employs the policy gradient to jointly learn tactic prediction and proof search, HTPS (Lample
et 2022) adopts an AlphaZero (Silver et al. 2018 -like approach with online training, and
DT-Solver WWagg et al. 2023a) improves MCTS with dynamic tree sampling and proof-level
value function. Additionally, COPRA (Thakur et al. 2024) implements a language-agen method that uses GPT-4 to perform a backtracking search based on the proof history, and
TrialMaster An et al. 2024 finetunes LLMs with trial-and-error data to do backtracking.
3.5
Other Tasks
In addition to the primary tasks outlined previously, we briefly list several other prediction
tasks that are related or helpful to theorem proving, One prominent line of research (Clark
let al. 2020 Saha et al., 2020 Dalvi et al. 2021 Tafjord et al. 2021 Sanyal et al. 2022 Bostrom
iet al. 2022 Hong et al. 2022; Mishra et al. 2022 Yang et al. 2022 Tatjord et al. 2022
Morishita et al. 2023) Saparov & He 2023) tocuses on generating step-by-step rationale of
hypothesis from set of known facts, which answers open-domain question. These works primarily perform FOL rule deduction over natural language, in the form of an entailment tree, an analogy to the proof tree in formal theorem proving. Another area of interest (Urban & Jakubüv 2020 Piotrowski & Urban| 2020b; Rabe et al., 2021 J & Smallbone [023 Bengio & Malkin 2024| Poesia et al. 2024) is automated conjecturing, which aims to discover new and interesting theorems beyond existing data. Additionally,
research efforts such as Huang et al. (211)); Glorot et al. P Polu et al. (2023) predict
the proof length remained for a goal. Lee et al. (2020) Wu & Wu (201)) investigate proving theorems in the latent space. IsarStep (Li et al. 2021a) predicts the intermediate proposition given surrounding proofs. Skip-tree (Rabe et al. (021)) and PACT (Han et al. (202)) propose several self-supervised tasks by masking various proof terms for training language models. LIME (Wu et al. 2021c) creates three synthetic pre-training tasks inspired by three reasoning primitives of deduction, induction, and abduction, to improve the performance of formal theorem proving. Recently, Li et al. (2023) proposes to match the informal proofs with theorem statements from a large database, REFÁCTOR (Zhou et al. 2024b) and ATG (Lin et al. 2024c) focus on extracting or generating new useful formal theorems from proofs.
6

---

Published as a conference paper at COLM 2024
Language
Nited
Cr.. ML-5
Jandrr & Frodee brohs
Naturiiroee
Millcke dl plat NatanPreee-Ca (Riled et pin
Er. Ganddal Puencct
Neng
Dancho-chmm
ptar
PRISK
Roddedid al pta
Ec. buckag Lintal baptel (BA
bangcca Pont Impaltt (2ing dial Jazz
MAPL Oticala st
pad Solane
Chang et 2a04
Pille (A A Norii prlnnd Tookkhore No14
Oaty Cilletsve
Leas
PhostNal (Aaderr
gg Levinn Fineet al bry
mixin (heng ot al pans EMM lis al pind TEDO No Feo Lon Page
LEAN-GIHFe petr
Kong mtll ben
(KL Light
Er. Netisp Salacyy st
pin HoUE (enaa et
pror minitt
Glarg ci
Mioer
Sg. MPTP3IS More
Miovii
Kollayk
Uifon
Rolloyke
Couticra
(arplestal
Linguaga
(00-4G-0
Prensworn
Satker (mmppprar
LIME
Geseoves
Augnooration
Sr. DappHOL (annel at
(olz &e Setaaver (101)
PMBCL Phlc al aane
HEX Appton pap
Lerno
Diooayy
Sr. 18225Paner Nir pocn REFACTOR Chousttl
AlG Luntell baok
Ausdaa
Te., 3086 [[lngg al Jaeu Figet la
MESTARD
Stunng
DeggloddPooeer
pinte
LLen Weeldon Vrnger piccal
Figure 3: The taxonomy of datasets in theorem proving.
4
Datasets
This section classifies and summarizes datasets for theorem proving into 2 categories: i) datasets extracted from existing corpora or manually curated and ii) those using synthetic generation or augmentation methods. The taxonomy of these datasets is shown in Figure 3 and Table i in the Appendix provides more detailed information.
4.1 Data Collection
We begin with the review of informal datasets. NL-PS (Ferreira & Freitas 2020a) first builds natural language premise selection dataset source from ProotWiki Similarly, NaturalProofs (Welleck et al. (021) further incorporates theorems from Stacks, and textbooks for premise selection. Adapted from NaturalProofs, utilizes a subset of theorems and proofs for informal proof generation.
SoturallProof-Gerr (Welleck al. 2022
For formal datasets, a line of efforts has been made to extract and clean theorems and proofs from established formal libraries and verification projects. Notable datasets for Coq include GamePad (Huang et al. 22019), CoqGym (Yang & Deng 2019), Proverbot9001 (Sanchez-Stern et al. 2020), PRISM (Reichel et al. 2023), and Graph2lac (Rute et al. (024)) which are constructed based on mathematical or software verification projects. For Isabelle, datasets like IsarStep (Li et al. 2021a PISA (Jiang et al. 2021 , and MAPL (Mikula et al. 2024) are built on the Archive of Formal Proofslane, Isabelle Standard Library while Selene (Zhang) et al.
2024) and FVELer (Lin et (0024) are constructed based on the seL4 project (Klein
let al.
2009). LeanStep (Han et 2022), LeanDojo (Yang et al. 2023), MLFMF (Bauer et al.
2023), and LEANN-itHuu (Wu et al. 2024) utilize Lean's open-source libraries (e-g., the
mathlib library (mathlib Community 2020)). Datasets for other proof assistants include Hol- Step (Kaliszyket et 2017) HOList (Bansal et al. 2019) for HOL Light, MPTP2078 (Alama let al. 2014), Mizar40 (Kaliszyk & Urban 2015b), and M2K (Kaliszyk et al. 2018) for Mizar, etc. Besides extracting data from existing projects, several works manually formalize or annotate the problems into formal languages: miniF2F (Zheng et al. 20022,, FIMO (Liu et al.
2023), ProofNet (Azerbayev et al. 2023), and PutnamBench (Tsoukalas et al. 2024 manually formalize high school and college-leve level problems in mathematical competitions or textbooks in Lean or Isabelle, and miniCodeProps pros (Lohn& & Welleck 2024) translates Haskell programs
into Lean. For other domains, TRIGO (Xiong et al. [2023) formalizes the trigonometric reduc- tion problem in Lean, while UniGeo (Chen et al. 2022] and IMO-AG-30 (Trinh et al., 2024 annotate proof steps for geometry proving problems in their designed formal languages.
On the other hand, a large body of recent studies leverages large-scale online corpora with billions of tokens from informal and formal mathematical data to build the pre-training datasets that could aid in theorem proving. These datasets include WebMath (Polu & Sutskever 2020), Proof-Pile (Azerbayev et al. 2023), DeepSeekMath (Shao et al. 2024), etc.
7

---

Published as a conference paper at COLM 2024
4.2 Data Generation
Beyond utilizing existinggprooects, researchers also study the generation of new theorems and proofs. A line of work (Wu et al. c Firoiu et al., 2021 Trinh et al. [202 An et al. 2 Wei et al. 2024) develops rule-based generators to produce new data by iteratively sampling interence rules from a pre-defined set. This process enables manual control over both the quantity and difficulty of the generated theorems. For example, INT (Wu al., 2021b produces 1.5 million inequality training problems with different proof lengths and axiom distributions than the testing ones, while AlphaGeometry (Trinh et al. 2024 synthesizes over 100 million training data with proof lengths ranging from to 247. MetaGen (Wang &
Deng 2020 also trains a neural generator to produce theorems similar to human-write ones.
Alternative approaches turn to iteratively augment the training dataset with fixed theorems but newly generated proofs. A line of work (Bansal et al. 2019 Polu & Sutskever 2020 Polu et al., 2023) adopts the idea of expert iteration (Silver et al. 2018), which repeatedly
applies the trained prover on existing theorems and adds the successtul proof paths as new data points train the prover further. Aygün et al. (2022) also proposes to adapt hindsight experience replay (Andrychowicz et al. 2017) to FOL provers, which leverages previously unsuccessful proof trajectories by viewing their final states as the desired ones. Meanwhile, a line of RL-based methods can also be viewed in this category.
Additionally, some studies aim generate intermediate helpful lemmas. REFACTOR (Zhou et al. 2024b) trains GNN to extract lemmas from proofs, which can be used to streamline the proots of other theorems. Similarly, ATG (Lin et al., 2024c) generates new theorems based on MCTS and self-play learning to shorten proots. LEGO-Prover (Xin et al. 2024b prompts GPT-4 to generate sub-goal lemmas for the proof of theorem and finalize it by proving or retrieving these lemmas. Newly proven lemmas are added to a library for future use. Works on conjecturing are also relevant to this field, although the generated conjectures could be more challenging than existing theorems and may be incorrect difficult to prove.
Moreover, several recent approaches leverage furooii)formlliiztioon for data generation. Using GPT-4, MMA (Jiang al. 2023a) informalizes all theorem statements in Archive of Formal Proofs and mathlib, while F4rmL4 (Lu et al., 2024) generates informal descriptions of both theorems and proofs extracted from mathlib. MUSTARD (Huang et al. 2024) syn- thesizes problems from a few sampled seed concepts, crafts informal proots, and translates them into Lean to verify their correctness. DeepSeek-Proven (Xin et al.) 2024a) autoformal- izes mathematical competition problems and filters high-quality statements through model scoring and hypothesis rejection methods. Similarly, Lean Workbook (Ying et al. (0244)) proposes an active learning pipeline that autoformalizes mathematical questions and filters them through Lean's compilation, LLMs' evaluation, and human diagnostics at each round.
5 Evaluations
In this section, we focus on the evaluations of deep learning approaches in theorem proving, analyzing the key metrics and state-of-the-art performance for each task. Detailed accuracies of state-of-the-art methods on several key datasets are provided in Table in the Appendix.
The assessment of autoformalization mainly relies on manually check- ing the equivalence between informal and formalized statements. Recent studies (Wu et al.) 2022 Azerbayev et al., 2023) indicate that slate-of-the-arr LLMs with few-shot prompting can only correctly formalize 25% and 13% high-school and undergraduate- highlighting the challenge in autoformalization. In addition, both studies reveal that L.LMs exhibit considerably higher efficacy in informalization, achieving accuracies of around 76% and 62%, respectively. Despite the modest success in autoformalizing statements, subse- quent research (Jiang et al.) 2023b Zhao et al. 2024 Xin et al. 2024b) reveals the benefit of autoformalizing intermediate sub-goals to generate formal proots in modularized pipelines.
Autoformalization.
problems,
Premise Selection. Retrieval metrics are widely used as the evaluation metric for premise selection. For example, recall at k (R@k) measures the ratio of correctly used premises in ground-truth proof within the top-k selections, and the mean reciprocal rank (MRR)
8

---

Published as a conference paper at COLM 2024
computes the average reciprocal rank of the first correctly selected premise. Among existing methods, DPR with a Transformer encoder has significantly improved upon traditional methods, demonstrating a remarkable generalization ability to unseen data for premise selection. For example, ReProver (Yang et al. 2023) achieves 27.6% for R@10 and 0.24 for MRR on the LoanDojo benchmark for retrieving unseen premises in training, while the baseline method BM25 (Robertson et al. 2009) achieves 15.5% for R@10 and 0.14 for MRR. Furthermore, DPR-based retriever, Magnushammer (Mikula et al. 2024), outperforms the classic hammer system, Sledgehammer (Böhme & Nipkow 2010), when used to select premises for the Thor prover (Jiang al., 2022), improving the theorem proving success rate from 57% to 71% on the PISA dataset and from 28.3% to 36.9% on the miniF2F-valid dataset.
Theorem Proving. The effectiveness of proofstep generation, proof search, and support from autoformalization and premise selection can be collectively evaluated by their success rate in proving theorems within a test set. Recent research (Zheng et al. 2024 Xin et al. 2024b) shows impressive performance increases using state-of-the-art LLMs like GPT-4 in structured frameworks than fine-tuned tactic-based language models with search heuristics. For instance, LEGO-Prover (Xin et al., 2024b) achieves 57.0% accuracy and 50.0% accuracy on the valid and test set of miniF2F, while the previous best prover Thor (fiang et al., 2022) with expert iteration training (Wu et al. 2022) achieves 37.3% and 35.2%. Moreover, advanced learning-basse systems. HTPS (Lample et al., 2022) achieves an accumulative successful rate of 58.6% on Finil2FF-valid through online training and 41.0% success rate with 64 search attempts on miniF2FFtestt. Besides, the RL-based ATP system NIAGRA (Fokoue et al. 2023) outperforms both E (Schulz, (002) and Vampire (Kovács & Voronkov 2013) on the MPTP2078 dataset.
proof searches could further improve the performance of ITP/ATI
Caveats. Tasks related to theorem proving can be tricky to evaluate. For autoformalization, manual evaluation is costly, whereas most automated metrics are inaccurate. For example, the compilation rate evaluates only syntactic correctness, and the BLEU score (Papineni et al. 2002) struggles with formalizations semantically similar but not logically equivalent to the ground truth. Although LeanBuclid (Murphy et al. 2024 provides automatic semantic evaluation between the autoformalized theorem statement and the ground truth, its domain is limited to Euclidean geometry, making it difficult to generalize to other areas. For premise selection, metrics rely the premises that are used in ground-truth proofs, they may neglect other valid premises for alternative correct proofs different from the ground-truth ones, causing false negatives. Additionally, the evaluation of theorem proving is further complicated by the variety of experimental setups, as detailed in $6.1
Discussions
6.1 Challenges
Despite significant progress, deep learning for theorem proving still faces many challenges, including data scarcity, disunified evaluation protocols, and human-Al interaction.
Data Scarcity. The amount of formal proof data is growing but is still far behind other domains where LLMs are successful, e.g., code generation. The largest corpora of Isabelle proofs, Archive of Formal Proofs, currently contains 250K proofs. Lean's 's mathlib contains 140K 140K proofs. This amount of data decent for small models (e.g., billions of parameters) but insufficient for models with hundreds of billions of parameters. Although the use of rule- based generators could offer some assistance, the complexity and quality of the generated data often do not match that of human-written ones. Furthermore, autoformalization is even more data-scarce, rce, due to the difficulty in obtaining aligned informal-formal pairs.
Compared to traditional deep learning tasks such as classification, it is consider- ably more complex to evaluate the performance theorem provers comprehensively, Firstly, results across different proof assistants are not directly comparable. Even though miniF2F is available across multiple proof assistants, the impact of proof automation tools (e.g,, Isabelle has Sledgehammer (Böhme& & Nipkow] (010) whereas Lean does not) often outweighs the differences attributable to deep learning models. Secondly, resource constraints during evaluation, such as time and the number of attempts, can significantly affect performance
Evaluation.

---

Published as a conference paper at COLM 2024
and the relative ranking of different methods. While more attempts may favor larger models like LLMs, tight time constraints, especially in real-time applications, may favor simpler, faster models. Without specific application as the context, it is unclear what evaluation setting makes the most sense. Thirdly, the use of pre-trained LLMs like GPT-4 may introduce data contamination issues (Rute et al. 2024), as these models might have been pre-trained on existing informal or formal testing problems, leading to unfair comparisons. We as community still lack a systematic evaluation framework across different types of neural theorem provers, despite nascent efforts (Lamont et al. 2024 Rute et al. 2024).
Human-AI Interaction. One motivation for theorem proving to assist human mathe- maticians [Castelvecchi] 2021; Sorensen et al. (021). However, existing research has led to surprisingly lew tools useful for them. Current methods are evaluated following standard deep learning protocols: running neural networks or querying LLMs in a Python program, testing the prover on a dataset, and calculating the percentage of successfully proved ones. This practice is misaligned with the needs of mathematicians. First, mathematicians need tool that can be called easily in a proof assistant. Second, the tool must run on consumer CPUs with low latency. Third, instead of performance measure, mathematicians care more about whether the prover can help with the specific theorems they are working on. These theorems are often out of the training distribution and pose a challenge for the prover to gen- eralize to other domains. Although initial efforts have been made to develop user-oriented tools (Welleck & Saha 2023 Song et al.) 2024 Rute et al. 2024), there is abundant room to improve the user experience and explore other forms of interaction, which requires close collaboration between deep learning researchers and mathematicians (Collins et al., 2023
6.2 Future Directions
Combining deep learning, especially LLMs, with theorem proving provides a promising avenue for enhancing the mathematical capabilities of AT and may significantly impact various disciplines. We conclude our survey paper by listing a few future directions we are particularly excited about, envisioning significant strides in these burgeoning domains:
Conjecturing. Beyond merely proving theorems, mathematicians would always explore theories in domain, identify underlying problem structures, and formulate new conjec- tures. These explorative activities around conjecturing are indispensable for mathematicians but are relatively limited in current deep learning approaches (Urban & Jakubüv 2020 Johansson & Smallbone) 2023 Bengio & Malkin 2024). By enabling Al to generate usetul conjectures, it can explore the space of mathema tics autonomously. When combined with theorem proving, such exploration can also be used to discover new mathematical knowl- edge. Furthermore, a direct application of conjecturing is to generate more theorem (and proof) data, which could mitigate the data scarcity inherent in theorem proving.
Verified Code Generation. As AI coding assistants such GitHub Copilot become preva- lent, it is increasingly important to be able to verify LLM-generated code. Proof assistants, especially Coq, have been widely used for software verification (Leroy et al. 2016 Gu et al., 2016). Therefore, methods for theorem proving surveyed in this paper could potentially play role in generating verified code. There is a plethora of problems explore in this space. For example, one can train or prompt LLMs to synthesize program invariants (Pei let al. 2023 Kamath et al. 2023) or generate programs in verification-friende languages such as Datny (Sun et al. 2023), Verus (Yao et al. 2023), and F* (Chakraborty et al.. 2024).
Math Education. A roadblock democratizing math education is the lack of qualified tutors to provide feedback to students (Kumar et al. 2023). Formal mathematics can poten- tially mitigate this issue by providing an environment for students to explore and receive automatic and reliable feedback. AT has demonstrated promise in guiding students in this process. For example, Buzzard (2024) reported that Lean Copilot (Song et al. 2024 effectively assisted in proving a bunch of problems in his undergraduate course and solved questions on the Lean Zulip To further integrate Al-driven formal tutoring into mainstream education, informalization essential to make formal proofs accessible to students unfamil- iar with formal languages. Looking ahead, we believe that theorem proving with LLMs will pave the way for intelligent tutors, enhancing math education for a broader audience.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank the anonymous reviewers for their insightful comments. This work was supported, in part, by Individual Discovery Grants from the Natural Sciences and Engineering Research Council of Canada, and the Canada CIFAR Al Chair Program.
References
Ibrahim Abdelaziz, Veronika Thost, Maxwell Crouse, and Achille Fokoue. An Experimental Study of Formula Embeddings for Automated Theorem Proving in First-Order Logic. arXiv preprint arXi::2002.00423, 2020.
Ibrahim Abdelaziz, Maxwell Crouse, Bassem Makni, Vernon Austel, Cristina Cornelio, Sha- jith Ikbal, Pavan Kapanipathi, Ndivhuwo Makondo, Kavitha Srinivas, Michael Witbrock, et al. Learning to Guide iiuurattio-BBsee Theorem Prover. IEEE Transactions Patlern Analysis and Machine Intelligence, 2022.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report. ar preprint ar Xir:2233.00877,, 2023.
Ayush Agrawal, Siddhartha Gadgil, Navin Goyal, Ashvni Narayanan, and Anand Tadipatri. Towards a Mathematics Formalisation Assistant using Large Language Models. Xiv preprint ar Xri::2221.00722 2022.
Jesse Alama, Tom Heskes, Daniel Kühlwein, Evgeni Tsivtsivadze, and Josef Urban. Premise Selection for Mathematics by Corpus Analysis and Kernel Methods. Journal of Automated Reasoning, 2014.
Chenyang An, Zhibo Chen, Qihao Ye, Emily First, Letian Peng, Jiayun Zhang, Zihan Wang, Sorin Lerner, and Jingbo Shang. Learn from Failure: Fine- Fine-Tuning LLMs with Trial-and- Error Data for Intuitionistic Propositional Logic Proving. arXiv preprint ar 201:22404.07388 2024.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay. In Proceedings of the International Conference on Neural Information Processing Systems, 2017.
Eser Aygün, Zafarali Ahmed, Ankit Anand, Vlad Firoiu, Xavier Glorot, Laurent Orseau, Doina Precup, and Shibl Mourad. Learning to Prove from Synthetic Theorems. arXiv preprint ar XrXi:2206..1155) 2020.
Eser Aygün, Ankit Anand, Laurent Orseau, Xavier Glorot, Stephen M Mcaleer, Vlad Firoiu, Lei M Zhang, Doina Precup, and Shibl Mourad. Proving Theorems using Incremental Learning and Hindsight Experience Replay. In Proceedings of the International Conference on Machine Learning, 2022.
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. ProofNet: Autoformalizing and Formally Proving Undergraduutt--evel Mathematics. arXiv preprint ar arXio:2302.12433 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An Open Lan- guage Model for Mathematics. In Proceedings of the International Conference on Learning Representations, 2024.
Kshitij Bansal and Christian Szegedy. Learning Alignment between Formal & Informal Mathematics. Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2020.
11

---

Published as a conference paper at COLM 2024
Kshitij Bansal, Sarah M. Loos, Markus Norman Rabe, Christian Szegedy, and Stewart Wilcox. HOList: An Environment for Machine Learning of Higher Order Logic Theorem Proving. In Proceedings of the International Conference on Machine Learning, 2019.
Haniel Barbosa, Clark Barrett, Martin Brain, Gereon Kremer, Hanna Lachnitt, Makai Mann, Abdalrhman Mohamed, Mudathir Mohamed, Aina Niemetz, Andres Nötzli, et al. CVC5: A Versatile and Aeddustrill-Strnngeee SMT Solver. In Proceedings of the International Conference on Tools and Algoritims for the Construction and Analysis of Systems, 2022.
Bruno Barras, Samuel Boutin, Cristina Cornes, Judicaël Courant, Yann Coscoy, David Chhrstopte Filliâtre, Eduardo Giménez, Hugo
Delahaye, Daniel de Rauglaudre, Herbelin, et al. The Coq Proof Assistant Reference Manual. INRIA, 1999.
Filip Bártek and Martin Suda. How Much Should This Symbol Weigh? A GNN-Advised Clause Selection. In Proceedings of International Conference on Logic for Programming, Artificial Intelligence and Reasoning, 2023.
Andrej Bauer, Matej Petković, and Ljupco Todorovski. MLFMF: Data Sets for Machine Learning for Mathematical Formalization. In Proceedings of the International Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Yoshua Bengio and Nikolay Malkin. Machine Learning and Information Theory Concepts Towards an AT Mathematician. Bulletin of the American Mathenatical Society, 2024.
Armin Biere. CaDiCaL at the SAT Race 2019. In Proczedings of SAT Race 2019 Solver and Benchmark Descriptions, 2019.
Jasmin Christian Blanchette, Cezary Kaliszyk, Lawrence C Paulson, and Josef Urban. Ham- mering Towards QED. Journal of Formalized Reasoning, 2016.
Sascha Böhme and Tobias Nipkow. Sledgehammer: Judgement Day. In Proceedings of the International Joint Conference on Automated Reasoning, 2010.
Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. Natural Language Deduction through Search over Statement Compositions. In Findings of the Association for Computational Linguistics: EMNLP, 2022.
David Brandfonbrener, Sibi Raja, Tarun Prasad, Chloe Loughridge, Jianang Yang, Simon Henniger, William E Byrd, Robert Zinkov, and Nada Amin. Verified Multi-Step Syn- thesis using Large Language Models and Monte Carlo Tree Search. arXiv preprint axXiv:2402.08147, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. In Proceedings of the International Conference Neural Information Processing Systems, 2020.
N.G. Bruijn, de. The Mathematical Language Automath, Its Usage and Some of Its Exten- sions. In Proceedings of the Symposium on Automatic Demonstration, 1970.
Kevin Buzzard. Lean in 2024.
http::///eenpprjeettworrrrrrssss wttp:/ss....mm/22/// com.2024////2///
lean-ing ier-ii-222200 -n22444/ 2024.
Davide Castelvecchi. Mathematicians Welcome Computer-Assisted Proof in 'Grand Unifica- tion' Theory. Nature, 2021.
Saikat Chakraborty, Gabriel Ebner, Siddharth Bhat, Sarah Fakhoury, Sakina Fatima, Shu- vendu Lahiri, and Nikhil Swamy. Towards Neural Synthesis for SMT-Assisted Proof- Oriented Programming. arXiv preprint ar XrXi::2405.01777, 2024.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expres- sion. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022.
12

---

Published as a conference paper at COLM 2024
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating Large Language Models Trained on Code. arXio preprint XxXi:22177003374, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Frame- work for Contrastive Learning of Visual Representations. In Proceedings of the International Conference on Machine Learning, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014.
Shang-Ching Chou, Xiao-Shan Gao, and Jing-Zhong Zhang, A Deductive Database Ap- proach to Automated Geometry Theorem Proving and Discovering, Journal of Automated Reasoning, 2000.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research, 2023.
Karel Chvalovský, Jan Jakubüv, Martin Suda, and Josef Urban. ENIGMA-NG: Efficient Neural and Cradient-Booster Inference Guidance for E. In Proceedings of the International Conference on Automated Deduction, 2019.
Karel Chvalovský, Jan Jakubūv, Miroslav Olšák, and Josef Urban. Learning Theorem Proving Components. In Proceedings of the International Conference on Automated Reasoning with Analytic Tableaux and Related Methods, 2021.
Karel Chvalovský, Konstantin Korovin, Jelle Piepenbrock, and Josef Urban. Guiding Instantiation Prover with Graph Neural Networks. In Proceedings of International Conference on Logic for Programming, Artificial Intelligence and Reasoning, 2023.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as Soft Reasoners over Language. In Proceedings of the International Joint Conference on Artificial Intelligence, 2020.
Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua Tenenbaum, William Hart, et al. Evaluating Language Models for Mathematics through Interactions. arv preprint arXiv:2306.01694, 2023.
Maxwell Crouse, Ibrahim Abdelaziz, Cristina Cornelio, Veronika Thost, Lingfei Wu, Kenneth Forbus, and Achille Fokoue. Improving Graph Neural Network Representations of Logical Formulae with Subgraph Pooling, aro preprint ar 1rXie:1911.06904, 2019.
Maxwell Crouse, Ibrahim Abdelaziz, Bassem Makni, Spencer Whitehead, Cristina Cornelio, Pavan Kapanipathi, Kavitha Srinivas, Veronika Thost, Michael Witbrock, and Achille Fokoue. A Deep Reinforcement Learning Approach to First-Order Logic Theorem Proving. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Garett Cunningham, Razvan C Bunescu, and David Juedes. Towards Autoformalization of Mathematics and Code Correctness: Experiments with Elementary Proofs. In Proceedings of the Workshop on Mathematical Natural Language Processing, 2022.
Lukasz Czajka and Cezary Kaliszyk. Hammer for Coq: Automation for Dependent Type Theory. Journal of automated reasoning, 2018
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengman Xie, Hannah Smith, Leighanna Pi- patanangkura, and Peter Clark. Explaining Answers with Entailment Trees. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021.
13

---

Published as a conference paper at COLM 2024
Doratossadat Dastgheib and Ehsaneddin Asgari. Keyword-Based Natural Language Premise Selection for an Automatic Mathematical Statement Proving. In Proceedings of TeettGrpps-16: Graph-Busee Based Methods for Natural Language Processing, 2022.
Martin Davis. A Computer Program for Presburger's Algorithm. Symbolic Computation Automation of Reasoning 1, 1957.
Martin Davis and Hilary Putnam. A Computing Procedure for Quantification Theory.
Journal of the ACM, 1960.
Leonardo De Moura and Nikolaj Bjørner. Z3: An efficient SMT solver. In Proceedings of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems, 2008.
Jörg Denzinger, Matthias Fuchs, Christoph Goller, and Stephan Schulz. Learning from Previous Proof Experience: A Survey. Technical Report, 1999.
Jacob Devlin, Ming- WWi Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, 2019.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In Proceedings of the International Conference on Neural Information Processing Systems, 2015.
Niklas Eén and Niklas Sörensson. An Extensible SAT-Solver. In Proceedings of the International Conference on Theory and Applications of Satisfiability Testing, 2003.
Alhussein Fawzi, Mateusz Malinowski, Hamza Fawzi, and Omar Fawzi. Learning Dynamic Polynomial Proofs. In Proceedings of the International Conference on Neural Information Processing Systems, 2019.
Deborah Ferreira and André Freitas. Natural Language Premise Selection: Finding Sup- porting Statements for Mathematical Text. In Proceedings of the Language Resources and Evaluation Conference, 2020a.
Deborah Ferreira and André Freitas. Premise Selection in Natural Language Mathematical Texts. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics, 2020b.
Deborah Ferreira and André Freitas. STAR: Cross-Modal [STA]tement [Rlepresentation for Selecting Relevant Mathematical Premises. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistic, 2021.
Vlad Firoiu, Eser Aygun, Ankit Anand, Zafarali Ahmed, Xavier Glorot, Laurent Orseau, Lei Zhang, Doina Precup, and Shibl Mourad. Training First-Order Theorem Prover from Synthetic Data. In International Conference on Learning Representations Workshop on Mathematical Reasoning in General Artificial Intelligence, 2021.
Emily First and Yuriy Brun. Diversity-Driven Automated Formal Verification. In Proceedings of the International Conference on Software Engineering, 2022.
Emily First, Yuriy Brun, and Arjun Guha. TacTok: Semantics-Aware Proof Synthesis. Proceedings of the ACM Programming Languages, 2020.
Emily First, Markus Rabe, Talia Ringer, and Yuriy Brun. Baldur: Whole-Proof Generation and Repair with Large Language Models. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023.
14

---

Published as a conference paper at COLM 2024
Achille Fokoue, Ibrahim Abdelaziz, Maxwell Crouse, Shajith Ikbal, Akihiro Kishimoto, Guilherme Lima, Ndivhuwo Makondo, and Radu Marinescu. An Ensemble Approach for Automated Theorem Proving Based on Efficient Name Invariant Graph Neural Rep- resentations. In Proceedings of the International Joint Conference on Artificial Intelligence, 2023.
Simon Frieder, Julius Berner, Philipp Petersen, and Thomas Lukasiewicz. Large Language Models for Mathematicians. Xtv preprint ar arXiv:2312.mmm 2.44556, 2023a.
Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, and Julius Berner. Mathematical Ca- pabilities of ChatGPT. In Proceedings of the International Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b.
Simon Frieder, Martin Trimmel, Rashid Alawadhi, and Klaus Gy. LLM vs ITP. In International Conference on Neural Information Processing Systems Workshop on MATH-AI, 2023c.
Siddhartha Gadgil, Anand Rao Tadipatri, Ayush Agrawal, Ashvni Narayanan, and Navin Goyal. Towards Automating Formalisation of Theorem Statements using Large Language Models. In International Conference on Neural Information Processing Systems Workshop on MATH-AI, 2022.
Thibault Gauthier. Deep Reinforcement Learning for Synthesizing Functions in Higher- Order Logic. In Proceedings of the International Conference on Logic for Programming, Artificial Intelligence and Reasoning, 2020.
Thibault Gauthier. Learned Provability Likelihood for Tactical Search. In Proceedings of the International Symposium Symbolic Computation in Software Science, 2021.
Thibault Gauthier, Cezary Kaliszyk, and Josef Urban. TacticToe: Learning to Prove with Tactcc. Journal of Automated Reasoning, 2020.
Fabian Gloeckle, Baptiste Roziere, Amaury Hayat, and Gabriel Synnaeve. Temperature- Scaled Large Language Models for Lean Proofstep Prediction. In International Conference on Neural Information Processing Systems Workshop on MATH-AI, 2023.
Xavier Glorot, Ankit Anand, Eser Aygun, Shibl Mourad, Pushmeet Kohli, and Doina Precup. Learning Representations of Logical Formulae using Graph Neural Networks. In International Conference on Neural Information Processing Systems Workshop on Graph Representation Learning, 2019.
Zarathustra A Goertzel and Josef Urban. Usefulness of Lemmas via Graph Neural Networks. In Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2019.
Zarathustra A Goertzel, Karel Chvalovský, Jan Jakubüv, Miroslav Olšák, and Josef Urban. Fast and Slow ENIGMAs and Parental Guidance. In Proceedings of the International Symposium Frontiers of Combining Systems, 2021.
Zarathustra Goertzel, Jan Jakubūv, Cezary Kaliszyk, Miroslav Olšák, Jelle Piepenbrock, and Josef Urban. The Isabelle ENIGMA. In Proceedings of the International Conference on Interactive Theorem Proving, 2022.
Georges Gonthier. The Four Colour Theorem: Engineering of Formal Proof. In Proceedings of the Asian Symposium on Computer Mathematics, 2008
Ronghui Gu, Zhong Shao, Hao Chen, Xiongnan Newman Wu, Jieung Kim, Vilhelm Sjöberg, and David Costanzo. CertiKOS: An Extensible Architecture for Building Certified Con- current OS Kernels. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation, 2016.
Thomas Hales, Mark Adams, Gertrud Bauer, Tat Dat Dang, John Harrison, Hoang Le Truong, Cezary Kaliszyk, Victor Magron, Sean McLaughlin, Tat Thang Nguyen, et al. A Formal Proof of the Kepler Conjecture. In Forum of Mathematics, Pi, 2017.
15

---

Published as a conference paper at COLM 2024
Jesse Michael Han, Tao Xu, Stanislas Polu, Arvind Neelakantan, and Alec Radford. Con- trastive Finetuning of Generative Language models for Informal Premise Selection. In Proceedings of the Conference Artificial Intelligence and Theorem Proving, 2021.
Jesse Michael Han, Jason Rute, Yuhuai Wu, Edward W Ayers, and Stanislas Polu. Proof Artifact Co-Training for Theorem Proving with Language Models. In Proceedings of the International Conference on Learning Representations, 2022.
John Harrison. HOL Light: A Tutorial Introduction. In Proceedings of the International Conference on Formal Methods in Computer-Aided Design, 1996.
Edvard K Holden and Konstantin Korovin. Graph Sequence Learning for Premise Selection. In Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2023.
Ruixin Hong, Hongming Zhang, Xintong Yu, and Changshui Zhang. MetGen: A Module- Based Entailment Tree Generation Framework for Answer Explanation. In Findings of the Association for Computational Linguistics: NAACL, 2022.
Daniel Huang, Prafulla Dhariwal, Dawn Song, and Ilya Sutskever. GamePad: A Learning Environment for Theorem Proving. In Proceedings of the International Conference on Learning Representations, 2019.
Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin, Haiming Wang, Zhenguo Li, Linqi Song, and Xiaodan Liang. MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data. In Proceedings of the International Conference on Learning Representations, 2024.
Geoffrey Irving, Christian Szegedy, Alexander Alemi, Niklas Fén, François Chollet, and Josef Urban. DeepMath Deep Sequence Models for Premise Selection. In Proceedings of the International Conference on Neural Information Processing Systems, 2016.
Jan Jakubüv and Josef Urban. ENIGMA: Efficient Baariing-Based Inference Guiding Ma-
chine. In Proceedings of the International Conference on Intelligent Computer Mathematics, 2017.
Jan Jakubüv and Josef Urban. Hammering Mizar by Learning Clause Guidance. In Proceed- ings of the International Conference on Interactive Theorem Prozing, 2019.
Jan Jakubüv, Karel Chvalovsky, Miroslav Olšák, Bartosz Piotrowski, Martin Suda, and Josef Urban. ENIGMA Anonymous: Symbol-Independent Inference Guiding Machine (System Description). In Proceedings of the International Joint Conference on Automated Reasoning, 2020.
Jan Jakubüv, Karel Chvalovsky, Zarathustra Goertzel, Cezary Kaliszyk, Mirek Olšák, Bartosz Piotrowski, Stephan Schulz, Martin Suda, and Josef Urban. MizAR 60 for Mizar 50. In Proceedings of the International Conference on Interactive Theorem Proving, 2023.
Albert Q Jiang, Wenda Li, Jesse Michael Han, and Yuhuai Wu. LISA: Language Models of Isabelle Proofs. In Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2021.
Albert Q Jiang, Wenda Li, Szymon Tworkowski, Konrad Czechowski, Tomasz Odrzygó2dz, Piotr Mitoš, Yuhuai Wu, and Mateja Jamnik. Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers. In Proceedings of the International Conference on Neural Information Processing Systems, 2022.
Albert Q Jiang, Wenda Li, and Mateja Jamnik. Multilingual Mathematical Autoformalization arviv preprint urXio=2311.03755, 2023a.
Albert Q Jiang, Sean Welleck, Jin Peng Zhou, Wenda Li, Jiacheng Liu, Mateja Jamnik, Timothée Lacroix, Yuhuai Wu, and Guillaume Lample. Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs. In Proceedings of the International Conference on Learning Representations, 2023b.
16

---

Published as a conference paper at COLM 2024
Dongwei Jiang, Marcio Fonseca, and Shay B Cohen. LeanReesoner: Boosting Complex Logical Reasoning with Lean. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024.
Moa Johansson and Nicholas Smallbone. Exploring Mathematical Conjecturing with Large Language Models. In Proceedings of the International Workshop on Neural-Symibn Learning and Reasoning, 2023.
Cezary Kaliszyk and Josef Urban. FEMaLeCoP: Fairly Efficient Machine Learning Connec- tion Prover. In Proceedings of the International Conference on Logic for Programming, Artificial Intelligence and Reasoning, 2015a.
Cezary Kaliszyk and Josef Urban. MizAR 40 for Mizar 40. Journal of Automated Reasoning, 2015b.
Cezary Kaliszyk, Josef Urban, Jiří Vyskočil, and Herman Geuvers. Developing Corpus-Basee Baase Translation Methods between Informal and Formal Mathematics: Project Description. In Proceedings of the International Conference on Intelligent Computer Mathematics, 2014.
Cezary Kaliszyk, Josef Urban, and Jiří Vyskočil. Learning to Parse on Aligned Corpora (Rough Diamond). In Proceedings of the International Conference on Interactive Theorem Proving, 2015.
Cezary Kaliszyk, François Chollet, and Christian Szegedy. HolStep: A Machine Learning Dataset for Higher-Order Logic Theorem Proving. In Proceedings of the International Conference on Learning Representations, 2017.
Cezary Kaliszyk, Josef Urban, Henryk Michalewski, and Miroslav Olšák. Reinforcement Learning of Theorem Proving. In Proceedings of the International Conference on Neural Information Processing Systems, 2018.
Adharsh Kamath, Aditya Senthilnathan, Saikat Chakraborty, Pantazis Deligiannis, Shu- vendu K Lahiri, Akash Lal, Aseem Rastogi, Subhajit Roy, and Rahul Sharma. Finding Inductive Loop Invariants using Large Language Models. arXiv preprint Xii::2311.00744, 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020.
Christoph Kern and Mark R Greenstreet. Formal Verification in Hardware Design: A Survey. ACM Transactions on Design Automation of Electronic Systemss, 1999.
Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, et al. seL4: Formal Verification of an OS Kernel. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles, 2009.
Konstantin Korovin. iProver-An Instantiation-Based Theorem Prover for First-Order Logic (System Description). In Proceedings of the International Joint Conference on Automated Reasoning, 2008
Laura Kovács and Andrei Voronkov. First-Order Theorem Proving and Vampire. In Proceed- ings of the International Conference on Computer Aided Verification, 2013.
Liubov Kovriguina, Roman Teucher, and Robert Wardenga. TextGraphs-Le 16 Natural Lan- guage Premise Selection Task: Zero-Shot Premise Selection with Prompting Generative Language Models. In Proceedings of TextGraphs-16: Graph-Basee Methods for Natural Lan- guage Processing, 2022.
Andrzej Stanislaw Kucik and Konstantin Korovin. Premise Selection with Neural Networks and Distributed Representation of Features. ar Xiv preprint 1xxt:1807.11266 2018
17

---

Published as a conference paper at COLM 2024
Daniel Kühlwein, Twan van Laarhoven, Evgeni Tsivtsivadze, Josef Urban, and Tom Heskes. Overview and Evaluation of Premise Selection Techniques for Large Theory Mathematics. In Proceedings of the International Joint Conference Automated Reasoning, 2012.
Harsh Kumar, David M Rothschild, Daniel G Goldstein, and Jake M Hofman. Math Education with Large Language Models: Peril or Promise? SSRN Electronic Journal, 2023.
Mitsuru Kusumoto, Keisuke Yahata, and Masahiro Sakai. Automated Theorem Proving in Intuitionistic Propositional Logic by Deep Reinforcement Learning. ar preprint arXiv:1811.0079 2018
Sean Lamont, Michael Norrish, Amir Dezfouli, Christian Walder, and Paul Montague. BAIT: Benchmarking (Embedding) Architectures for Interactive Theorem-Provingy In Proceedings of the AAAI Conference Artificial Intelligence, 2024.
Guillaume Lample and Alexis Conneau. Cross-Lingual Language Model Pretraining, In Proceedings of the International Conference Neural Information Processing Systems, 2019.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Phaese-Based & Neural Unsupervised Machine Translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018.
Guillaume Lample, Timothee Lacroix, Marie-Anne Lachaux, Aurelien Rodriguez, Amaury Hayat, Thibaut Lavril, Gabriel Ebner, and Xavier Martinet. Hypertree Proof Search for Neural Theorem Proving. In Proceedings of the International Conference on Neural Information Processing Systems, 2022.
Dennis Lee, Christian Szegedy, Markus N Rabe, Sarah M Loos, and Kshitij Bansal. Mathe- matical Reasoning in Latent Space. In Proceedings of the International Conference on Learning Representations, 2020.
Xavier Leroy, Sandrine Blazy, Daniel Kästner, Bernhard Schommer, Markus Pister, and Christian Ferdinand. CompCert-A Formally Verified Optimizing Compiler. In Proceeding of the European Congress on Embedded Real Time Software and Systems, 2016.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Son Solo, et al. Solving Quantitative Reasoning Problems with Language Models. Proceedings of the International Conference on Neural Information Processing Systems, 2022.
Weixian Waylon Li, Yftah Ziser, Maximin Coavoux, and Shay B Cohen. BERT Is Not The Count: Learning to Match Mathematical Statements with Proofs. In Proceedings of the Conference of the European Chapler of the Association for Compulational Linguistics, 2023.
Wenda Li, Lei Yu, Yuhuai Wu, and Lawrence C Paulson. IsarStep: A Benchmark for High- Level Mathematical Reasoning. In Proceedings of the International Conference on Learning Representations, 2021a.
Zhaoyu Li, Binghong Chen, and Xujie Si. Graph Contrastive Pre- Pre-Training for Effective The- orem Reasoning. International Conference on Machine Learning Workshop on Learning for Reasoning and Perception, 2021b.
Zhenwen Liang, Tianyu Yang, Jipeng Zhang, and Xiangliang Zhang. UniMath: A Founda- tional and Multimodal Mathematical Reasoner. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023.
Haohan Lin, Zhiqing Sun, Yiming Yang, and Sean Welleck. Lean-STaR: Learning to Inter- leave Thinking and Proving. ar Xiv preprint ar XrXi:22407.1044 2024a.
Qika Lin, Jun Liu, Lingling Zhang, Yudai Pan, Xin Hu, Fangzhi Xu, and Hongwei Zeng. Contrastive Graph Representations for Logical Formulas Embedding. IEEE Transactions on Knowledge and Data Engineering, 2021.
18

---

Published as a conference paper at COLM 2024
Xiaohan Lin, Qingxing Cao, Yinya Huang, Haiming Wang, Jianqiao Lu, Zhengying Liu, Linqi Song, and Xiaodan Liang. FVEL: Interactive Formal Verification Environment with Large Language Models via Theorem Proving. arviv preprint ar Sttt:2406.14408 2024b.
Xiaohan Lin, Qingxing Cao, Yinya Huang, Zhicheng Yang, Zhengying Liu, Zhenguo Li, and Xiaodan Liang. ATG: Benchmarking Automated Theorem Generation for Generative Language Models. In Findings of the Association for Computational Linguistics: NAACL, 2024c.
Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, et al. FIMO: A Challenge Formal Dataset for Automated Theorem Proving. arv preprint ar (xXi:2309.00299: 2023.
Qinghua Liu, Yang Xu, and Xingxing He. Attention Recurrent Cross-Graphy Neural Network for Selecting Premises. International Journal of Machine Learning and Cybernetics, 2022a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: Robustly Optimized BERT Pretraining Approach. ar Xiv preprint ar xxxc:1907.11692 2019.
Zhou Liu, Yujun Li, Zhengying Liu, Lin Li, and Zhenguo Li. Learning to Prove Trigonometric Identities. Xrviv preprint (xXx:2207.00679, 2022b.
Evan Lohn and Sean Welleck. miniCodelrops: A Minimal Benchmark for Proving Code Properties. arXiv preprint arXiv:2406.11915, 2024.
Sarah Loos, Geoffrey Irving, Christian Szegedy, and Cezary Kaliszyk. Deep Network Guided Proof Search. In Proceedings of the International Conference on Logic for Programming, Arlificial Intelligence and Reasoning, 2017.
László Lovász and Alexander Schrijver. Cones of Matrices and Set-Functions and 0-1 Optimization. SIAM journal optimization, 1991.
Jianqiao Lu, Zhengying Liu, Yingjaa Wan, Yinya Huang, Haiming Wang, Zhicheng Yang, Jing Tang, and Zhijiang Guo. Process-Driver Autoformalization in Lean 4. arXiv preprint Stt::24060014000 2024.
Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. Neural Machine Translation (seq2seq) Tutorial. http:///ittuub.mmm////ssfffuuu/a 2017.
The mathlib Community. The Lean Mathematical Library. In Proceedings of the ACM SIGPLAN International Conference on Certified Programs and Proofs, 2020.
Jack McKeown and Geoff Sutcliffe. Reinforcement Learning for Guiding the E Theorem Prover. In Proceedings of the International Florida Artificial Intelligence Research Society Conference, 2023.
Norman Megill and David A Wheeler. Metamath: A Computer Language for Mathematical Proofs. Lulu Press, 2019.
Maciej Mikula, Szymon Antoniak, Szymon Tworkowski, Albert Q Jiang, Jin Peng Zhou, Christian Szegedy, tukasz Kucinski, Piotr Mitos, and Yuhuai Wu. Magnushammer: A Transformer-Basee Approach to Premise Selection. In Proceedings of the International Conference on Learning Representations, 2024.
Robin Milner. Implementation and Applications of Scott's Logic for Computable Functions. In Proceedings of the ACM Conference on Proving Assertions about Programs, 1972.
Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. Towards Teachable Reasoning Systems: Using Dynamic Memory of User Feedback for Continual System Improvement. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. ar Xiv preprint arXt::1312.5602. 2013.
19

---

Published as a conference paper at COLM 2024
Guangshuai Mo, Yan Xiong, Wenchao Huang, and Lu Ma. Automated Theorem Proving via Interacting with Proof Assistants by Dynamic Strategies. In Proceedings of the International Conference on Big Data Computing and Communiccations, 2020.
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic. In Proceedings of the International Conference Machine Learning, 2023.
Leonardo de Moura and Sebastian Ullrich. The Lean 4 Theorem Prover and Programming Language. In Proceedings of the International Conference on Automated Deduction, 2021.
Logan Murphy, Kaiyu Yang, Jialiang Sun, Zhaoyu Li, Anima Anandkumar, and Xujie Si. Autoformalizing Euclidean Geometry. In Proceedings of the International Conference on Machine Learning, 2024.
Theo X Olausson, Alex Gu, Benjamin Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua B Tenenbaum, and Roger Levy. LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023.
Miroslav Olšák, Cezary Kaliszyk, and Josef Urban. Property Invariant Embedding for Automated Reasoning. In Proceedings of the European Conference on Artificial Intelligence, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. Xiv preprint ar Xix::1007.00 0037488,0.. 2018.
Jens Otten and Wolfgang Bibel. leanCoP: Lean Connection-Batee Based Theorem Proving. Journal of Symbolic Compulation, 2003.
Aditya Paliwal, Sarah Loos, Markus Rabe, Kshitij Bansal, and Christian Szegedy. Graph Representations for Higher-Order Logic and Theorem Proving. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-LM: Empow- ering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning. In Findings of the Association for Computational Linguistics: EMNLP, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the annual meeting of the
Association for Computational Linguistics, 2002.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. OpenWebMath An
Open Dataset of High-Quallt Mathematical Web Text. In Proceedings of the International Conference on Learning Representations, 2024.
Nilay Patel, Jeffrey Flanigan, and Rahul Saha. New Approach Towards Autoformalization. ar Xiv preprint arXit:2310.07957 2023.
Lawrence C Paulson. Isabelle: A Generic Theorem Prover. Springer, 1994.
Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. Can Large Language Models Reason about Program Invariants? In Proceedings of the International Conference on Machine Learning, 2023.
Kebin Peng and Dianfu Ma. Tree-Structure CNN for Automated Theorem Proving. In Proceedings of the International Conference on Neural Information Processing, 2017.
Jelle Piepenbrock, Tom Heskes, Mikoláš Janota, and Josef Urban. Learning Equational Theorem Proving. In Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2021.
Jelle Piepenbrock, Tom Heskes, Mikoláš Janota, and Josef Urban. Guiding An Automated Theorem Prover with Neural Rewriting. In Proceedings of the International Joint Conference on Automated Reasoning, 2022a.
20

---

Published as a conference paper at COLM 2024
Jelle Piepenbrock, Josef Urban, Konstantin Korovin, Miroslav Olšák, Tom Heskes, and Mikolaš Janota. Machine Learning Meets The Herbrand Universe. ar preprint (rXiv:2210.03550, 2022b.
Bartosz Piotrowski and Josef Urban. Stateful Premise Selection by Recurrent Neural Net- works. In Proceedings of the International Conference Logic for Programming, Artificial Intelligence and Reasoning, 2020a.
Bartosz Piotrowski and Josef Urban. Guiding Inferences in Connection Tableau by Recurrent Neural Networks. In Proceedings of the International Conference on Intelligent Computer Mathemmatics, 2020b.
Gabriel Poesia, David Broman, Nick Haber, and Noah D Goodman. Learning formal mathematics from intrinsic motivation. arXiv preprint ar arXiv:2407.00695 2024.
Auguste Poiroux, Gail Weiss, Viktor Kunčak, and Antoine Bosselut. Improving Autoformal- ization using Type Checking. arXiv preprint ar xrXi::2406.07222, 2024.
Stanislas Polu and Ilya Sutskever. Generative Language Modeling for Automated Theorem Proving. areiv preprint ar XrXivv2009.03393, 2020.
Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever. Formal Mathematics Statement Curriculum Learning. In Proceedings of the International Conference on Learning Representations, 2023.
Seth Poulsen, Sami Sarsa, James Prather, Juho Leinonen, Brett A Becker, Arto Hellas, Paul Denny, and Brent N Reeves. Solving Proof Block Problems Using Large Language Models. In Proceedings of the ACM Technical Symposium on Computer Science Education, 2024.
Krsto Proroković, Michael Wand, and Jürgen Schmidhuber. Improving Stateful Premise Selection with Transformers. In Proceedings of the International Conference on Intelligent Computer Mathematics, 2021.
Xin Quan, Marco Valentino, Louise A Dennis, and André Freitas. Verification and Refine- ment of Natural Language Explanations through LLM-Symbolic Theorem Proving. Xiv preprint arXiv:2405.01339, 2024.
Markus N Rabe, Dennis Lee, Kshitij Bansal, and Christian Szegedy. Mathematical Reasoning via Self-Supervised Skip-Tree Training. In Proceedings of the International Conference on Learning Representations, 2021.
Michael Rawson and Giles Reger. A Neurally-Guided Parallel Theorem Prover. In Proceed- ings of the International Symposium on Frontiers of Combining Systems, 2019.
Michael Rawson and Giles Reger. Directed Graph Networks for Logical Reasoning (Ex- tended Abstract). In Joint Proceedings of the Workshop on Practical Aspects of Automated Reasoning (PAAR) and the Satisfiability Checking and Symbolic Computation Workshop (SC- Square) Workshop, 2020.
Michael Rawson and Giles Reger. lazyCoP: Lazy Paramodulation Meets Neurally Guided Search. In Proceedings of the International Conference on Automated Reasoning with Analytic Tableaux and Related Methods, 2021.
Tom Reichel, R Henderson, Andrew Touchet, Andrew Gardner, and Talia Ringer. Proof Repair Infrastructure for Supervised Models: Building a Large Proof Repair Dataset. In Proceedings of the International Conference Interactive Theorem Proving, 2023.
Stephen Robertson, Hugo Zaragoza, et al. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends@ in Information Retrieval, 2009.
Piotr Rudnicki. An Overview of the Mizar Project. In Proceedings of the Workshop on Types for Proofs and Programs, 1992.
21

---

Published as a conference paper at COLM 2024
Jason Rute, Miroslav Olšák, Lasse Blaauwbroek, Fidel Ivan Schaposnik Massolo, Jelle Piepenbrock, and Vasily Pestun. Graph2Tac: Online Representation Learning of Formal Math Concepts. In Proceedings of the International Conference on Machine Learning, 2024.
Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. PRover: Proof Generation for Interpretable Reasoning over Rules. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020.
Alex Sanchez-Stern, Yousef Alhessi, Lawrence Saul, and Sorin Lerner. Generating Cor- rectness Proofs with Neural Networks. In Proceedings of the ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2020.
Alex Sanchez-Stern, Emily First, Timothy Zhou, Zhanna Kaufman, Yuriy Brun, and Talia Improving Automated Formal Verification Using Identifiers. ACM Transactions on Programming Languages and Systems, 2023.
Ringer. Passport:
Soumya Sanyal, Harman Singh, and Xiang Ren. FaiRR: Faithful and Robust Deductive Reasoning over Natural Language. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2022.
Abulhair Saparov and He He. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. In Proceedings of the International Conference on Learning Representations, 2023.
Gregor vom Scheidt. Experimental Results from Applying GPT-4 to An Unpublished Formal Language. ar arviv preprint arXiv:2305.12196 2023.
Stephan Schulz. E-A Brainiac Theorem Prover. Al Communications, 2002.
Johann M Schumann. Automated Theorem Proving in Software Engineering, Springer Science & Business Media, 2001.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. MeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Xiv preprint ar drXio:2402.03300, 2024.
Boris Shminkee gym-saturation: Gymnasium Environments for Saturation Provers (System description). In Proceedings of the International Conference on Automated Reasoning with Analytic Tableaux and Related Methods, 2023.
David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play. Science, 2018.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MPNet: Masked and Permuted Pre-Training for Language Understanding. In Proceedings of the International Conference on Neural Information Processing Systems, 2020.
Peiyang Song, Kaiyu Yang, and Anima Anandkumar. Towards Large Language Models as Copilots for Theorem Proving in Lean. arXio preprint arXiv:2404.128 125344 2024.
Henrik Kragh Sørensen, Mikkel Willum Johansen, Renee Hoekzema, and Hester Breman. Augmenting the Human Mathematician. In International Conference on Learning Represen- tations Workshop on Mathematical Reasoning in General Artificial Intelligence, 2021.
Martin Suda. Vampire with a Brain a Good ITP Hammer. In Proceedings of the International Symposiun on Frontiers of Combining Systems, 2021a.
Martin Suda. Improving ENIGMA-Style Clause Selection While Learning From History. In Proceedings of the International Conference on Automated Deduction, 2021b.
Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett. Clover: Closed-Loop Verifiable Code Generation. ar Xiv preprint ar Xrt::23100.7800 2023.
22

---

Published as a conference paper at COLM 2024
Geoff Sutcliffe. The TPTP Problem Library and Associated Infrastructure. Journal of Auto- mated Reasoning, 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Networks. In Proceedings of the International Conference on Neural Information Processing Systems, 2014.
Christian Suttner and Wolfgang Ertel. Automatic Acquisition of Search Guiding Heuristics. In Proceedings of the International Conference on Automated Deduction, 1990.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Proceedings of the International Conference on Neural Information Processing Systems, 1999.
Christian Szegedy. A Promising Path Towards Autoformalization and General Artificial Intelligence. In Proceedings of the International Conference on Intelligent Computer Mathematics, 2020.
Christian Szegedy, Markus Rabe, and Henryk Michalewski. Retrieval-Augmented Proof Step Synthesis. In Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2021.
Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. ProofWriter: Generating Impli- cations, Proofs, and Abductive Statements over Natural Language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP, 2021.
Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022.
Kai Sheng Tai, Richard Sucher, and Christopher D Manning. Improved Semantic Repre- sentations From irreSttuuttred Long Short-Term Memory Networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, 2015.
Amitayush Thakur, George Tsoukalas, Yeming Wen, Jimmy Xin, and Swarat Chaudhuri. An In-Context Learning Agent for Formal Theorem-Proving arviv preprint arXio:2310.04333, 2024.
Thi Hong Hanh Tran, Matej Martinc, Antoine Doucet, and Senja Pollak. IJS at TextGraphs-16 Natural Language Premise Selection Task: Will Contextual Information Improve Natural Language Premise Selection? In Proceedings of TextGrapps-16: Graph-Based Methods for Natural Language Processing, 2022.
Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving Olympiad Geometry without Human Demonstrations. Nature, 2024.
Paul Trust, Provia Kadusabe, Haseeb Younis, Rosane Minghim, Evangelos Milios, and Ahmed Zahran. UNLPS TextGraphs 2022 Shared Task: Unsupervised Natural Lan- guage Premise Selection in Mathematical Texts Using Sentence-MPNeet. In Proceedings of TextGraphs-16: Graph- Based Methods for Natural Language Processing, 2022.
George Tsoukalas, Jasper Lee, John Jennings, Jimm Xin, Michelle Ding, Michael Jennings, Amitayush Thakur, and Swarat Chaudhuri. PutnamBench: Evaluating Neural Theorem- Provers on the Putnam Mathematical Competition. ar ariiv preprint ar Xiv:2400 11214, 2024.
Szymon Tworkowski, Maciej Mikula, Tomasz Odrzygóždž, Konrad Czechowski, Szymon Antoniak, Albert Jiang, Christian Szegedy, Lukasz Kuciński, Piotr Milos, and Yuhuai Wu. Formal Premise Selection With Language Models. In Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2022.
Josef Urban and Jan Jakubüv. First Neural Conjecturing Datasets and Experiments. In Proceedings of the International Conference Intelligent Computer Mathematics, 2020.
23

---

Published as a conference paper at COLM 2024
Josef Urban, Jifí Vyskočil, and Petr Štěpánek. MaLeCoP Machine Learning Connection Prover. In Proceedings of the International Conference on Automated Reasoning with Analytic Tableaux and Related Methods, 2011.
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu, et al. WaveNet: A Generative Model for Raw Audio. ar Xie preprint Kiw:1609.0344 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In Proceedings of the International Conference Neural Information Processing Systems, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph Attention Networks. In Proceedings of the International Conference on Learning Representations, 2018.
Rahul Vishwakarma and Subhankar Mishra. Enhancing Neural Theorem Proving through Data Augmentation and Dynamic Sampling Method. ar preprint ar arii:2231..7 14188, 2023.
Haiming Wang, Ye Yuan, Zhengying Liu, Jianhao Shen, Yichun Yin, Jing Xiong, Enze Xie, Han Shi, Yujun Li, Lin Li, et al. DT-Solver: Automated Theorem Proving with Dynamic- Tree Sampling Guided by Proof-level Value Function. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023a.
Haiming Wang, Huajian Xin, Zhengying Liu, Wenda Li, Yinya Huang, Jianqiao Lu, Zhicheng Yang, Jing Tang, Jian Yin, Zhenguo Li, et al. Proving Theorems Recursively. arxiv preprint ar arXiv:2405.14411 2024.
Mingzhe Wang and Jia Deng. Learning to Prove Theorems by Learning to Generate Theo- rems. In Proceedings of the International Conference on Neural Information Processing Systems, 2020.
Mingzhe Wang, Yihe Tang, Jian Wang, and Jia Deng. Premise Selection for Theorem Proving by Deep Graph Embedding. In Proceedings of the International Conference on Neural Information Processing Systems, 2017.
Qingxiang Wang, Cezary Kaliszyk, and Josef Urban. First Experiments with Neural Transla- tion of Informal to Formal Mathematics. In Proceedings of the International Conference on Intelligent Computer Mathematics, 2018.
Qingxiang Wang, Chad Brown, Cezary Kaliszyk, and Josef Urban. Exploration of Neural Machine Translation in Autoformalization of Mathematics in Mizar. In Proceedings of the ACM SIGPLAN International Conference on Certified Programs and Proofs, 2020.
Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative AI for Math: Part I-MathPile: A Billion-Token-Scall Pretraining Corpus for Math. arXio preprint ar xrie:22312.,7120, 2023b.
Chenrui Wei, Mengzhou Sun, and Wei Wang. Proving Olympiad Algebraic Inequalities S without Human Demonstrations. ar Xiv preprint ar Xlo:2406.14219, 2024.
Sean Welleck and Rahul Saha. LLMSTEP: LLM Proofstep Suggestions in Lean. In Interna- tional Conference on Neural Information Processing Systems Workshop on MATH-AI, 2023.
Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. NaturalProofs: Mathematical Theorem Proving in Natural Language. In Proceedings of the International Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. NaturalProver: Grounded Mathematical Proof Generation with Language Models. In Proceedings of the International Conference on Neural Information Processing Systems, 2022.
24

---

Published as a conference paper at COLM 2024
Daniel Whalen. Holophrasm: A Neural Automated Theorem Prover for Higher-Order Logic. arXiv preprint arXio:1608.026: 006444 2016.
Minchao Wu and Yuhuai Wu. Latent Action Space for Efficient Planning in Theorem Proving. In Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2021.
Minchao Wu, Michael Norrish, Christian Walder, and Amir Dezfouli. TacticZero: Learning to Prove Theorems from Scratch with Deep Reinforcement T Learning. Proceedings of the International Conference on Neural Information Processing Systems, 2021a.
Yuhuai Wu, Albert Q Jiang, Jimmy Ba, and Roger Grosse. INT:
An Inequality Benchmark for Evaluating Generalization in Theorem Proving. In Proceedings of the International Conference on Learning Representations, 2021b.
Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning. In Proceedings of the International Conference on Machine Learning, 2021c
Yuhuai Wu, Albert Q Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy, Autoformalization with Large Language Models. In Proceedings of the International Conference on Neural Information Processing Systems, 2022.
Zijian Wu, Jiayu Wang, Dahua Lin, and Kai Chen. Lean-github: Compiling github lean repositories for versatile lean prover. aiv preprint ar arXiv:2407.1722 177227 2024.
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. DeepSeek-Provee Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data. ar Xiu preprint ar XrXi::2405.14333 2024a.
Huajian Xin, Haiming Wang, Chuanyang Zheng, Lin Li, Zhengying Liu, Qingxing Cao, Yinya Huang, Jing Xiong, Han Shi, Enze Xie, et al. LEGO-Prover: Neural Theorem Proving with Growing Libraries. In Proceedings of the International Conference on Learning Representations, 2024b.
Jing Xiong, Jianhao Shen, Ye Yuan, Haiming Wang, Yichun Yin, Zhengying Liu, Lin Li, Zhijiang Guo, Qingxing Cao, Yinya Huang, al. TRIGO: Benchmarking Formal Mathe- matical Proof Reduction for Generative Language Models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In Proceedings of the International Conference on Learning Representations, 2019.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 2022.
Kaiyu Yang and Jia Deng. Learning to Prove Theorems via Interacting with Proof Assistants. In Proceedings of the International Conference on Machine Learning, 2019.
Kaiyu Yang, Jia Deng, and Danqi Chen. Generating Natural Language Proofs with Verifier- Guided Search. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022.
Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem Proving with Serievall-uggmeneed Language Models. In Proceedings of the International Conference on Neural Information Processing Systems, 2023.
Jianan Yao, Ziqiao Zhou, Weiteng Chen, and Weidong Cui. Leveraging Large Language Models for Automated Proof Synthesis in Rust. arx preprint wxXin:2311.03739 2023.
25

---

Published as a conference paper at COLM 2024
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. SATLM: Satisfiabilit--AAdee Language Models using Declarative Prompting. In Proceedings of the International Conference on Neural Information Processing Systems, 2023.
Eric Yeh, Briland Hitaj, Sam Owre, Maena Quemener, and Natarajan Shankar. CoProver: A Recommender System for Proof Construction. In Proceedings of the International Conference on Intelligent Computer Mathematics, 2023.
Huaiyuan Ying, Zijian Wu, Yihan Geng, Jiayu Wang, Dahua Lin, and Kai Chen. Lean Workbook: A Large-Scale Lean Problem Set Formalized from Natural Language Math Problems. ar preprint arXiv:2406. 03847, 2024a.
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Má, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. InternLM-Maah:: Open Math Large Language Models Toward Verifiable Reasoning. arxiv preprint ar XrXiv:2402.06332, 2024b.
Roozbeh Yousefzadeh and Xuenan Cao. Large Language Models' Understanding of Math: Source Criticism and Extrapolation. arXiv preprint arXiv:2311.0066 2023.
Liao Zhang, Lasse Blaauwbroek, Cezary Kaliszyk, and Josef Urban. Learning Proof Trans- formations and Its Applications in Interactive Theorem Proving. In Proceedings of the International Symposium on Frontiers of Combining Systems, 2023a.
Lichen Zhang, Shuai Lu, and Nan Duan. Selene: Pioneering Automated Proof in Software Verification. In Proceedings of the Annual Meeting of the Association for Compulational Linguistics, 2024.
Shizhuo Dylan Zhang, Talia Ringer, and Emily First. Getting More out of Large Language Models for Proofs. In Proceedings of the Conference on Artificial Intelligence and Theorem Proving, 2023b.
Xueliang Zhao, Wenda Li, and Lingpeng Kong. Subgoal-Based Demonstration Learning for Formal Theorem Proving. In Proceedings of the International Conference on Machine Learning, 2024.
Chuanyang Zheng, Haiming Wang, Enze Xie, Zhengying Liu, Jiankai Sun, Huajian Xin, Jianhao Shen, Zhenguo Li, and Yu Li. Lyra: Orchestrating Dual Correction in Automated Theorem Proving. Transactions on Machine Learning Research, 2024.
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. MiniF2F: A Cross-System Benchmark for Formal Olympiad-Level Mathematics. In Proceedings of the International Conference on Learning Representations, 2022.
Jin Peng Zhou, Charles E Staats, Wenda Li, Christian Szegedy, Kilian Q Weinberger, and Yuhuai Wu. Don't Trust: Verify-Grounding L LLM Quantitative Reasoning with Auto- formalization. In Proceedings of the International Conference on Learning Representations, 2024a.
Jin Peng Zhou, Yuhuai Wu, Qiyang Li, and Roger Grosse. REFACTOR: Learning to Ex- tract Theorems from Proofs. In Proceedings of the International Conference on Learning Representations, 2024b.
Zsolt Zombori, Adrián Csiszárik, Henryk Michalewski, Cezary Kaliszyk, and Josef Urban. Towards Finding Longer Proofs. In Proceedings of the International Conference on Automated Reasoning with Analytic Tableaux and Related Methods, 2021a.
Zsolt Zombori, Josef Urban, and Miroslav Olšák. The Role of Entropy in Guiding a Con- nection Prover. In Proceedings of the International Conference on Automated Reasoning with Analytic Tableaux and Related Methods, 2021b.
26

---

Published as a conference paper at COLM 2024
A Lists of Datasets and State-of-the-Art Approaches
Dataset
Languag
Siz
Source
Tas
NI-P5 Ferming & Freites b02
NT.
20
Proofhiki
NaturaPaccs Wellkce HIMMONE
NI.
48,783
Proofhiiki, Stocks, lestbaokk
NaturaPPaois-- Leit Mipllca 11
2002
NL.
-33K
NaturdPrroola
PG
Gamelad [[laane 2t
k1W
Cog
1,602
Fes- Thompson theorem
CogCym (Ang& Lyrt
Cog
70,856
125 open-source projects from Github
Troverbut9 000.000000000000 S0or
1020
Cog
501"
CompCert cett
GraphZTac Redrers 2000
Cog
~500K
open-suurce packages from opam Achiece of Formal Proofs, liahelle standard library
PISA Jann 2i 2u7ta3
lobbllle
~1R3K
<443KK
Achieve Formal Proofs, liabellle standard library
Selene Pu
FVELar
29.125
LeunSley Pourenn
~2UKK
core library, mathiib
TRIDO
427
Iigenomerry problems from tiku
LeanDay Dey Mireerr Al
08,734
mathiin
MMF
Apila
270,647
mathlin, Apda standard libeary, UniMally TypeTopology
miniCiser
Hasseell
127
Tons of Inductivv Progams
GEANGIHHU
28,597
reposilaries on web
NL. Lean
149"
Shortlisted Problems
ProodNec
Lean
371'
undergraduate toxboooks
miniF22
Zhong al.
Metamath, Light
Lean,
IMO. AIME, AMC, MXTH custom
Putnambenni
Leen,
640"
William Lowell
Mathematical Competition
Hobbtep
Kalishraueu
Light
11,410
mullivariate analysis library Kepler conjecture
Light
31,662
Kepler conyecture
METTV207 Pithon Ce
Miza:
Mathematicaal Library
PS.
Miza
Mathematical Library
PS..
M2K (allewyksse 1e
Mizss
]PEF" (Sug0ii
IFTF
11,395
system evalcation
lactic Lo
HOL4
standard library
set.mn (aanooraee
37,091
high goometry problems geometry problems
IMO-AG30 (i
DSL
al,
DSL
1,301K
synthti insquality problems
DSL
soolhtk
MMA [llang
Lean, loablle
352.774
auteialormalin anteialormaiiation Achieve Formal Proafs and mathlib
NL,
outcialarmaliation from mathilb, Arithumo
MUSIARR
1030
NL Lean
autooormalzzation synthetic
DeepSeck
NL, Lean
8,066621
sutooormalization from competition problems antoformalization
Workook
Vinger
Problem Solving
WebMath
Folu Sulskever
7137
mised
358
GilHus, arXiv, Stack Exchangr
PT
Pioof-Plle
Aaetbayey
2025
mived
8.30
arXa,, Stack Exchange, formal libranies, ProofWiki, Wikipedio, books, MATH
MathPile
mixed
9.58
arXin textbooks, Stnck Exchange, Wikipedia, ProofWiki, Web forum educational content, refenence pages, scientriic blogs, and others.
PT
OpenWiebMath
Paster
2024
mised
14:7B
PT
Prod-Pilev22
Aeebarev wr
mised
558
Gillub, Stack, formal proof steps, OpentWehMath, arXiv
FT
DwepSekMaar
mised
1208
OpesWWesaahh Web
PT
InternLMMMMali
mixed
1238
Krowdedge File, Prood Pile-22 syntheti data
PT
Table 1: Summary of existing datasets for theorem proving. For datasets derived from generation methods, we primarily highlight several key examples here.
Language
Arcuracy
Metric
Method
Cog
Corfym
(First 2r00
-CogHammer Crajka Kaliszyk, 2016
(random)
passii timeout
temperature scaling Glooedlest
Lean
ProofNet Ivalid/wsst (kst)
49.2/53.2
iop-1 accuracy
type checking (Potooi
min:P2F (valid/
cummuaative/ pass8164
DxeppeokkProeer
Isabelie
PISA
minuF2F (valid/ fiest)
57.0/512
pass fimeout 500s
passa000//psss/220 pass45200
LEQO-Praver Magnuskarumtrr
Metamath
(valid/ ftest)
61,2/72.4
HTPS (Tample 2020
Mizar
MPTP208
75.5
timeout of 100s
NIAGRA Fokoue et 2023
Table 2: Summary of state-of-the-art methods and their performance on several key datasets.
NL: natural language. DSL: domain-specific language.
We use the number of tokens for pre-training datasets and the number of entries (e.g. definitions, premises, theorems) for other tasks. 6** indicates the size of the festing/validation set. 'e list the main task of each dataset. AF: autoformalization. PS;: premise selection. PG: proofstep generation. PS2: proof search. PT: pre-training. TP: theorem proving (combination of premise selection, proofstep generation, and proof search).
27


Reviews:
[{"review_id": "LZBtbcl0W0", "reviewer": "Reviewer_XYZ8", "summary": "This paper provides a comprehensive survey on deep learning approaches for theorem proving. \nThe survey includes a thorough review of existing tasks and methods, meticulous summary of available datasets and strategies for data generation, a detailed analysis of evaluation metrics and the performance of state-of-the-art, and a critical discussion on the challenges and future directions.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and valuable feedback! In the following, please let us address the concerns and questions you've pointed out.\n\n> References about informal theorem proving via natural language explanation\n\nWe appreciate your suggestion regarding the inclusion of PrOntoQA and EntailmentBank. We will add these references in our revised version to enrich the discussion on informal theorem proving via natural language explanations.\n\n> Missing references and reference versions\n\nThank you for bringing these issues to our attention. We also noticed several recent works (e.g., ATG) published after our initial submission. We will incorporate these references and ensure our discussion reflects the latest advancements. Additionally, we will carefully check and update all references to ensure they are the most current versions.\n\n> Proof generation via LLMs and prompting methods\n\nThe MUSTARD framework's focus on synthesizing theorem and proof data is primarily discussed in Section 4.2 (Data Generation). We will further elaborate on proof generation methods that leverage theorem prover feedback for self-correction, as seen in MUSTARD. Additionally, we will expand our discussion to include more advanced LLM prompting methods in proof generation in our revision.\n\nWe hope these revisions will address your concerns and enhance the comprehensiveness of our submission. Thank you again for your constructive feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a very thorough summary and includes most of the related research works. I believe this paper provides a fundamental reference to the community. This paper is well-organized and very easy to follow. The discussions on current challenges and future directions are insightful.", "reasons_to_reject": "Q1. References are missing in Section 2.1. For example, PrOntoQA [3] and EntailmentBank [4] are some demonstrations of informal theorem proving via natural language explanation. \n\nQ2. Some search works may be missing. For example, ATG [1] introduces an automated theorem generation task and corresponding evaluation metrics to evaluate if a model can generate new theorems for a given theorem proving to achieve the shortest proof as human mathematicians. They construct the benchmark by synthesizing new theorems based on Metamath language and the “set.mm” library. This paper should be included in Section 4.2. \n\nQ3. For proof generation via LLMs with several prompting methods, some of the methods also leverage the theorem prover feedback to prompt the LLMs to self-correct the proofs. For example, MUSTARD [2] uses the error messages from Lean prover to refine the proof generation. \n\nQ4. Many references are not the newest version. For example, “Llemma: An Open Language Model for Mathematics”, “Holist: An environment for machine learning of higher-order theorem proving”, “Lego-prover: Neural theorem proving with growing libraries” are all published in conference proceedings. The authors need to carefully check the reference list to make sure the citations are up to date. \n\n\n[1] Xiaohan Lin, Qingxing Cao, Yinya Huang, Zhicheng Yang, Zhengying Liu, Zhenguo Li, Xiaodan Liang (2024). ATG: Benchmarking Automated Theorem Generation for Generative Language Models. 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2024 Findings).\n\n[2] Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin, Haiming Wang, Zhenguo Li, Linqi Song, Xiaodan Liang (2024). MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data. The Twelfth International Conference on Learning Representations (ICLR 2024).\n\n[3] Abulhair Saparov, He He (2023). Language Models are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. The Eleventh International Conference on Learning Representations (ICLR 2023).\n\n[4] Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, Peter Clark (2021). Explaining Answers with Entailment Trees. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021).", "questions_to_authors": "Please see the questions above.", "extra_scores": {}}, {"review_id": "tSyhaF59tY", "reviewer": "Reviewer_UZhc", "summary": "This paper surveys the field of applying deep learning methods to theorem proving. An extensive summary of 170 papers is made on several tasks such as autoformaliztion, premise selection, proofstep generation, proof search and other tasks such as automated conjecturing. A taxonomy on the datasets involved in automated theorem proving is also made in a proper and comprehensive manner. Finally, a review of evaluations is given and a discussion on the future challenges is made, with an emphasis on data scarcity, evaluation and human-ai interaction.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and valuable suggestions!\n\nGiven the limited formal theorem and proof data, we agree that automated conjecturing (as well as theorem/proof synthesis) is a crucial direction for future research. It is highlighted in Sections 6.1 (Data Scarcity) and 6.2 (Conjecturing). We will emphasize these points further in Section 3.5 (Other Tasks). Besides, we will incorporate the suggested references and include two tables in the Appendix for better visualization: one listing the existing theorem-proving datasets and another benchmarking the performance of various theorem provers.\n\nPlease feel free to follow up if you have further questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The summary is very well-structured and well-suited according to the types of tasks and datasets.\n* The authors have good understanding of the general tendency of the field and its future. The challenges in Section 6 include important directions such as data scarcity.\n* The writing is clear in general and easy to read\n\nIn general, automated conjecturing, automated theorem generation and reasoning data synthesis would be a very important direction in the future and it would be good to either have a separate subsection in Section 3 or emphasize it in 3.5 Other tasks.", "reasons_to_reject": "* Several references are missing in some paragraphs:\n   * “DT-Solver” in 3.3 after “Subsequent advancements”\n   * “DT-Solver” in 3.4 after “Moreover, various works”\n   * “MUSTARD” in 3.5 before “explore automated conjecturing”\n* It would be nice to have some visualizations in the evaluation section for benchmarking and comparison purpose.", "questions_to_authors": "see above", "extra_scores": {"ethics_concerns_details": "n/a"}}, {"review_id": "sfTIYedn60", "reviewer": "Reviewer_pyKY", "summary": "This paper presents a comprehensive survey of deep-learning related exploration in the theorem proving. The survey features a summary of related tasks, datasets and evaluation metrics. At the end, the authors also discuss the challenges and the promising avenues for future exploration.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and valuable feedback! We hope to address your comments in detail as follows:\n\n> Related survey paper\n\nDespite the mentioned [paper](https://arxiv.org/pdf/2212.10535) including some related works about theorem proving (~25 papers), the focus of that paper is primarily on general mathematical reasoning, particularly math word problems, rather than specifically on theorem proving (see their Table 8 for details). It includes very few papers on ATPs, autoformalization, or proof search, and it lacks our classification hierarchy as well as coverage of the most up-to-date methods involving advanced LLMs such as GPT-4. In contrast, our paper presents a comprehensive survey of deep learning approaches specifically for theorem proving, highlighting the latest advancements and unique contributions in this field.\n\n> Reference with SAT/SMT-based approaches in related work\n\nThanks for the suggestion! We also acknowledge the importance of SAT, especially SMT solvers for ATP. Due to the page limit, we did not include them initially. We will add these approaches in our updated paper.\n\n> The trade-off between efficiency, engineering difficulty, and prediction accuracy\n\nAs discussed in Section 6.1 (Evaluation and Human-AI interaction), we agree that there is a trade-off, especially between the efficiency and efficacy of theorem proving. This is also true for premise selection. We will expand this discussion to include more about premise selection in the revised version.\n\n> Discussion about data contamination\n\nThank you for bringing this to our attention. We agree that data contamination (leakage) may occur, especially when using advanced LLMs like GPT-4, which may pre-train on existing testing data. We will add more discussion on this issue in our revision.\n\nWe appreciate your insightful feedback and are committed to addressing these points comprehensively. If you have any further suggestions or questions, please let us know!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well structured and comprehensive enough to include all the relevant papers I am aware of in the field.", "reasons_to_reject": "I am not exactly sure about COLM's criteria for a survey paper. The main downside of this paper is the potential overlap of another survey paper published last year (https://arxiv.org/pdf/2212.10535), though I can see some of the latest work have been added.", "questions_to_authors": "I don't have any specific questions. The followings are merely some comments:\n- page 2, 'saturation-based theorem provers'/'tableau-based', might be worth mentioning SAT/SMT-based approaches, which probably play the most important role in existing ATPs.\n- page 4, 'premise selection': despite the research in deep-learning-based premise selection, most of the working systems are still based on heuristics. This is especially true for ATPs. In exceptions like LeanCopilot we usually don't do fast proof explorations. The trade-off between efficiency, engineering difficulty, and prediction accuracy could be a topic worth discussion.\n- page 8, evaluation: Another (general) challenge in this field is the data contamination during the pretraining stage. This is also the reason that the Graph2Tac paper (https://arxiv.org/pdf/2401.02949) chose not to pretrain their LM baseline. More discussion is probably needed here.", "extra_scores": {}}, {"review_id": "gxLcymzKqU", "reviewer": "Reviewer_FB9a", "summary": "This paper offers a literature review on theorem proving, with a focus on recent deep learning approaches. The paper begins with introducing the informal and formal formulations of theorem proving, followed by presenting five common tasks in theorem proving and existing methods developed for each task, and then summarizes datasets, evaluation metrics and state-of-the-art model performances. The paper concludes with a discussion of outstanding challenges and promising future directions.\n\nIn my opinion this paper has a lot of good value. For a novice, it would be a great entry point for them to learn about the research landscape in ML-based theorem proving. For more senior researchers, it can also help organize their knowledge obtained from the large and scattering volume of work.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and valuable suggestions!\n\n> Presentation of datasets\n\nWe appreciate your suggestion to enrich the presentation of datasets. We will incorporate a detailed table in the appendix that lists the existing theorem-proving datasets, including information such as language, size, source, and other relevant notes. This will enhance the usability of our survey for readers seeking specific data resources.\n\n> Proof search in Section 3.4\n\nWhile classical search methods like best-first search and MCTS are indeed applicable in both ATPs and ITPs, there are nuanced differences in how these methods are integrated with neural networks and their respective tasks. In ATP, search methods are paramount, often utilizing neural networks like RNNs and GNNs for embedding, and treating the process as a classification task for making branching decisions. In ITP, tactic prediction models are also crucial, allowing simple search methods like BFS/DFS to yield effective results. Moreover, existing proof search approaches for ITPs often model the task as language modeling, using language models to predict the proof length or success of a branch as the value function during the search. We will elaborate on these distinctions in the revised version.\n\n> Theorem proving in Section 5\n\nThank you for pointing this out for better clarification. We will revise this paragraph accordingly to reflect the contributions of Thor and to clarify that both powerful LLMs and advanced search methods can significantly enhance theorem proving.\n\nWe appreciate your insightful feedback and will address these points thoroughly in our revised paper. If you have any further suggestions or questions, please let us know!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, I think readers of this paper can gain a comprehensive bird-eye view of the most important recent advances in ML-based theorem proving. As someone who worked in theorem proving, I personally learned a lot from this paper.\n1. The paper is well structured and clearly written.\n1. This review has very good coverage of notable papers in the field of learning-based theorem proving. For those cited papers that I am familiar with, the summary is generally accurate.\n1. It is very refreshing that the paper summarizes theorem proving tasks into five categories: autoformalization, premise selection, proofstep generation, proof search, and others. From the myriad of papers, this structure makes things very clear and easy to understand. Within each task, more fine-grained clustering is made (e.g., formulating the task as classification vs retrieval vs language modeling, early attempts using CNNs/RNNs vs more recent approaches leveraging pretrained LLMs and in-context learning).", "reasons_to_reject": "I wouldn’t say these are reasons to reject, they are more like suggestions to make the paper more clear. Happy to discuss further if there’s any misunderstanding.\n1. The presentation of datasets (Figure 3) can be made richer by including additional details, such as dataset size, data source, and annotation type (e.g. theorems, proofs, annotation of premises, fine-grained proof artifacts), so readers who are looking for data can more easily locate the best resource for them. This might be best presented as a table.\n1. In section 3.4 “proof search”, I’m not sure if I understand why search methods are discussed separately for ATP and ITP. As mentioned in this section, in both domains, classical search methods like best-first search and MCTS are shown to be applicable. Can you maybe clarify how search is different in APT and in ITP?\n1. Nit: In section 5 “theorem proving” paragraph, the Thor method seems to be implied as a “tactic-based model with best-first search” and contrasted with LLM-based methods and search-based methods. However, Thor also employs language models and proof search. It seems to me the takeaway here can be: using powerful LLMs can help, and using advanced search on top can also help. Some nuances could use some adjustment to make the story clearer here.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This survey paper on deep learning for theorem proving is impressively comprehensive and received uniformly positively assessments from reviewers. It's not easy to pack a comprehensive review into a 9-page conference paper the way the authors did. It's also notable that all reviewers brought up important, substantive points in their reviews, that the authors responded succinctly and clearly to all of them, and that all reviewers felt the author response was satisfactory. This is a model of a successful review–response process on both the reviewers' and the authors' parts. The paper should certainly be accepted.
Decision: Accept