DT-Solver: Automated Theorem Proving with Dynamic-Tree Sampling Guided by Proof-level Value Function
Year: 2023

Abstract:
Recent advances in neural theorem-proving resort to large language models and tree searches. When proving a theorem, a language model advises single-step actions based on the current proving state and the tree search finds a sequence of correct steps using actions given by the language model. However, prior works often conduct constant computation efforts for each proving state while ignoring that the hard states often need more exploration than easy states. Moreover, they evaluate and guide the proof search solely depending on the current proof state instead of considering the whole proof trajectory as human reasoning does. Here, to accommodate general theorems, we propose a novel Dynamic-Tree Driven Theorem Solver (DT-Solver) by guiding the search procedure with state confidence and proof-level values. Specifically, DT-Solver introduces a dynamic-tree Monte-Carlo search algorithm, which dynamically allocates computing budgets for different state confidences, guided by a new proof-level value function to discover proof states that require substantial exploration. Experiments on two popular theorem-proving datasets, PISA and Mathlib, show significant performance gains by our DT-Solver over the state-of-the-art approaches, with a 6.65% improvement on average in terms of success rate. And especially under low computing resource settings (11.03% improvement on average).

heading: 1 Introduction
text: Automated theorem proving (ATP) (Harrison et al., 2014) has been considered an essential task of Artificial Intelligence (AI) ever since the birth of modern AI (McCarthy et al., 2006). Besides its remarkable theoretical value and huge potential to accelerate research in mathematics, ATP already demonstrates excellent application value in, for ex-
† Corresponding authors. ∗ Work done during internship in Huawei Noah’s Ark Lab
ample, formal verification (Barras et al., 1997) and code generation (Howard, 1980).
In general, the goal of a theorem proving task is to construct a proof (a sequence of proof steps) that proves a given theorem (often within a given time budget, e.g. 300 seconds). The validity of this proof is verified by a formal environment in which the theorem and the proof are formalized. We show in Fig. 1 an example of a theorem proven in Lean (de Moura et al., 2015), a formal environment we use in this work. Sec.2.1 gives more details on the formal environment.
To deal with this theorem proving problem automatically, many task-specific algorithms have been proposed and are later implemented as stateof-the-art ATP solvers such as Z3 (de Moura and Bjørner, 2008), Vampire (Kovács and Voronkov, 2013), E (Schulz, 2002) and Zipperposition (Bentkamp et al., 2021). These approaches are taskspecific in that they are specific to one given formal environment and rely on solid domain knowledge, symbolic operations, and usually human heuris-
12632
tics. With the exciting development of deep learning (LeCun et al., 2015) over the past decade, approaches (not necessarily for ATP) in the taskagnostic paradigm have been proposed, such as AlphaZero (Silver et al., 2017) and more recently Gato (Reed et al., 2022). As for ATP, task-agnostic approaches are also proposed. For example, GPTf (Polu and Sutskever, 2020) uses a language model to suggest a set of candidates for the next proof step and then search for a complete valid proof within the formal system Metamath (MEGILL and DAVID A, 2019). In contrast to classic ATP algorithms, GPT-f has remarkable generality and can be applied to any ATP tasks in any formal system, such as propositional logic, first-order logic, higherorder logic, typed lambda calculi, etc. With GPT-f, domain knowledge and human heuristics are no more absolutely required. Some works following the workflow of GPT-f are introduced in A.2.
Although approaches such as GPT-f demonstrate impressive generality and performance, they still have two significant drawbacks. On the one hand, GPT-f consumes considerable computational resources. According to (Lample et al., 2022), the original GPT-f requires 2000 GPU days (A100) for training, and their approach HTPS (HyperTree Proof Search (Lample et al., 2022)) also consumes more than 1000 GPU days for one training process, which sets a prohibitive threshold for most researchers. Even worse, the cost for inference with GPT models and search can be 5 to 10 times heavier than that of training. On the other hand, the search process in GPT-f could reach a state of ‘empty queue’, where all proof step candidates (i.e. all leaf nodes in the search tree) generated by the language model turn out to be inapplicable. This can typically take place within the time budget and an premature ‘Failed’ result is returned, potentially resulting in a low pass rate (sometimes we also use the term ‘one-pass success rate’ to be more specific). Also, prior works (Polu and Sutskever, 2020; Han et al., 2021; Polu et al., 2022) often conduct constant computation efforts for each proving state while ignoring that the hard states often need more exploration than easy states. Moreover, they evaluate and guide the proof search solely depending on the current proof state instead of considering the whole proof trajectory as human reasoning does.
In this work, we address the above two issues and propose a novel Dynamic-Tree driven Theorem Solver (DT-Solver). As an automated theorem-
proving algorithm, DT-Solver uses dynamic-tree sampling guided by a proof-level value function. We illustrate these two main components as follows.
• Dynamic-tree sampling. To remedy the issue of premature failure with ‘empty queue’, we allow the nodes in the search tree to be expanded (i.e. generate child nodes using proof steps predicted by the language model) several times, instead of being expanded just once (as proposed by GPT-f). To achieve this, we add an imaginary node or a virtual node to each node in the search tree. When this virtual node is selected, no specific proof step is applied. Instead, we use the language model again to generate possible proof steps and potentially produce new and promising child nodes (proof states). Thus the width of the search tree can be adjusted dynamically. Furthermore, we modify the usual PUCT score in the Monte-Carlo tree search to ensure that the computational resources focus on promising but possibly ‘hard’ nodes. We find this dynamic sampling technique helpful and obtain an increase of 39.9% -> 48.4% in terms of the pass rate.
• Proof-level value function. Different from GPT-f (Polu and Sutskever, 2020) and HTPS (Lample et al., 2022), which use steplevel value function to guide the search, we train another encoder-only transformer, RoBERTa (Liu et al., 2019) to predict whether a new proof state is on the right track to finishing the proof. As we consider the whole proof instead of the current proof step, this RoBERTa model is actually a proof-level value function. We find that this technique can help increase the value function’s accuracy, for instance, from 63.6% to 70.7%.

heading: 2 Background

heading: 2.1 Formal mathematics environments
text: Following (Han et al., 2021; Polu et al., 2022) and (Jiang et al., 2021, 2022), we choose Lean (de Moura et al., 2015) and Isabelle (Paulson, 1994) as our formal environments. As illustrated in Fig. 1, the theorem-proving process in Lean (and similarly in Isabelle) consists of sequentially applying a proof step (also known as a tactic) to a given theorem statement. Specifically,
the formal environment first constructs an initial proof state from the theorem statement, which contains one or multiple goals to be proved. Seeking to solve all goals in the proof state, the user produces a proof step that applies an existing theorem (apply int.eq_of_sub_eq_zero), introduces some new assumptions (intro hap), or tells the formal environment to use techniques like proof by contradiction or mathematical induction. If the produced proof step is applicable, the formal environment applies the proof step and transforms the original proof state into a new proof state. This process is repeated until the proof state reaches ‘no goals’ (or ‘goals accomplished’ as illustrated), which means all goals in the theorem are proven.

heading: 2.2 Language model guided theorem proving
text: Recent approaches that use language models to solve the ATP problem mostly follow the work from GPT-f. Given the current proof state, a causal language model (usually a decoder-only transformer like GPT (Radford et al.)) is used to predict possible proof steps that can be applied. Concretely, the language model is trained on sequences in the following form using a language modeling objective: GOAL $(proof state) PROOFSTEP $(proof step) where $(·) is the placeholder for the actual proof state and proof step. At test time, multiple new proof steps are sampled from the language model giving a prompt as follows: GOAL $(proof state) PROOFSTEP
We denote this atomic operation as expansion. To construct a complete proof for a given theorem, GPT-f adopts the best-first search algorithm that iterates among selecting a best-scoring proof state from the priority queue to expand, doing expansion using the language model with the selected proof state, and adding new states to the priority queue. Each state’s score is either calculated by a value function or uses the prior probability of the preceding proof step. Iterations continue until GPT-f finds the theorem’s proof or reaches the limit of computational budgets.

heading: 3 Methodology
text: Starting from the initial state of the theorem, DT-Solver adapts the language model to suggest forward-moving actions and empowers dynamictree sampling guided by a proof-level value function to find the complete proof path. This is ac-
complished by a three-step pipeline: (1) Following procedures described in Sec. 2.2, DT-Solver trains a language model using supervised data extracted from formal mathematics libraries to predict proof steps based on proof states. The trained language model is called the policy model. (2) Subsequently, dynamic-tree sampling uses the trained policy model to generate steps and searches for theorems proofs in the training set. A data collection procedure collects proof trajectories from successful proof searches. (3) Lastly, DT-Solver trains the proof-level value function (called the critic model) to identify promising proof paths from astray ones on collected data with a classification loss. At test time, to evaluate the performance of our method, we use the trained policy model and critic model to perform dynamic-tree sampling on the test set theorems.
In the following section, we first introduce the dynamic-tree sampling algorithm and the data collection procedure (Sec.3.1). After that, we introduce our proof-level value function (Sec. 3.2).

heading: 3.1 Dynamic-tree sampling
text: The dynamic-tree sampling algorithm strives to maximize the efficiency of the search procedure under limited computational budgets. In practice, it controls the exploration of different proof states according to two criteria: state value and model confidence. The state value estimates whether the state is on the correct proof path. Only high-valued states deserve more exploration. The model confidence is calculated with the prior probabilities of the current state’s proof step. A state is considered to require more exploration if the policy model cannot produce confident steps toward the current state.
To achieve the above objectives, we build the dynamic-tree sampling based on the Monte-Carlo tree search algorithm. As shown in Fig. 2, dynamic tree sampling progressively adds new proof states to construct a proof tree. Specifically, each node in the proof tree represents a proof state denoted as si ∈ {s0, s1..., sM}, where M is the total number of tree nodes. Each edge represents a proof step denoted as aj ∈ {a0, a1..., aM−1}, where M − 1 is the total number of edges. For notational clarity, the edge from si to sj is always denoted as aj , using the same subscript as sj . Similar to the classic Monte-Carlo tree search algorithm, every node-edge pair in the proof tree has a visit count
denoted as N(si, aj) and a value count denoted as W (si, aj). Starting from the root node, dynamictree sampling repeats the following three steps until the theorem’s proof is found or the computational budget is exhausted. (1) The selection phase (Fig. 2(a)). Dynamic-tree sampling computes the PUCT score for child nodes and the virtual child node. We select the highest-scoring child node and proceed according to the following conditions: if a non-virtual child node is selected, dynamic-tree sampling continues the selection phrase from the selected child node. If a virtual node is selected, the selection phase ends and proceeds to expansion & evaluation. (2) The expansion & evaluation phase (Fig. 2(b)(d)(e)). Dynamic-tree sampling performs expansion on the parent node of the selected virtual node. A fixed number (denoted as e) of proof steps are sampled from the trained policy model. The formal environment verifies proof steps and produces new proof states, which are then deduplicated and evaluated by the critic model, and finally added to the tree. (3) The backpropagation phase (Fig. 2(c)). Dynamic-tree sampling backpropagates new state scores to the root state by successively adding the score to the parent’s value count W and accumulating visit count N .
We detailed the selection and backpropagation phrases in the following. The expansion phase is well discussed and we leave the details about evaluating states’ value in Sec. 3.2
Selection. We denote the current node to perform selection as st, and its children as sc ∈ C(st) where C(·) is the function that returns all the children for a given node. The virtual child of st is denoted as s′t. The PUCT score for each child node sc ∈ C(st) is formulated as follow:
PUCTsc = W (st, ac)
N(st, ac) + c · p(ac|st) ·
√ N(st, ·)
N(st, ac) (1)
where c is a constant balancing the exploration and exploitation trade-off, and p(ac|st) is the probability (estimated by the language model) of generating the proof step ac given current state st. The PUCT score for the virtual child node s′t is formulated as follow:
PUCTs′t = vals′t + c · confs′t · √ N(st, ·) |C(st)| (2)
vals′t = 1− maxsc∈C(st) W (st, ac) N(st, ac) (3) confs′t = 1− ∑
sc∈C(st) p(ac|st) (4)
where |C(st)| is the number of children st have. Two parts control the score of selecting the virtual node. vals′t estimates the value of the virtual node. When st finds high-valued promising children, vals′t will have a low score, indicating that
no more exploration is required on st. confs′t estimates the model confidence on st. The value of confs′t remains high until confident proof steps are generated. The model confidence is discounted by the number of children st have and keeps annealing with more exploration performed on st. In the selection phase, we repeatedly select the child node with the highest PUCT score and continue until we reach a virtual node. The selection process empowers dynamic-tree sampling by being able to backtrack to previously expanded states, eliminating the "empty queue" failure and drastically increasing DT-Solver’s stability of finding valid proof.
Backpropagation. The backpropagation follows the classical Monte-Carlo tree search. We denote the function that returns the node’s parent as P (·). The value counts W (P (sc), ac) and visit counts N(P (sc), ac) are both initialized as 0 for newly add child node. Given the newly added leaf state sc and its estimated score vc, dynamic-tree sampling repeats the following steps until the root node is reached: (1) Accumulate the visit count N(P (sc), ac) += 1, and accumulate the value count W (P (sc), ac) += vc (2) Traverse back the tree by changing the current node to the parent sc ⇐ P (sc).
Data collection. To construct a proof-level value function capable of identifying promising states from astray ones, DT-Solver collects supervised training data by performing dynamic-tree sampling in training set theorems1. Specifically, DT-Solver collects trajectories data in the form of ([(s0, a0), (s1, a1), ..., (sl, al)], y), where s0 is the root state and sl is the leaf state. The label y = 1 if the trajectory correctly proves the theorem and y = 0 otherwise. The same data collection procedure is performed on the validation and test theorems for testing the value function.

heading: 3.2 Proof-level value function
text: The performance and efficiency of the dynamictree sampling algorithm depend heavily on correctly evaluated state values. Existing methods like GPT-f only use a representation of the current state to assess the state’s quality. However, according to our close-up observation, states along correct proof paths tend to be more similar and consistent, while states along false proof paths exhibit more diversity.
1At this stage, the proof-level value function is not applicable; thus the state’s preceding step’s prior probability is used as value estimate v.
Thus, it is beneficial to use proof-level information to estimate the value of a state.
Accordingly, shown in Fig. 3, we propose four types of proof-level value functions. Current-stateonly value function construct supervised data formatted as (sl, y) from the collected trajectory data. A Roberta language model is fine-tuned on the current-state-only data with a classification loss. Root-state-and-current-state value function constructs sentence pair data formatted as ([s0, sl], y) from the collected trajectory. The two states are concatenated and fed into the Roberta to predict the state’s value. Conversely, previous-state-tocurrent-state uses more recent state information and formats the training data as ([sl−1, sl], y). Entiretrajectory value function concatenates the entire proof path as follows: GOAL $(s0) PROOFSTEP $(a0) 

 GOAL $(s1) PROOFSTEP $(a1) ...
where the 

 separate different stateaction pair. Empirically, the entire-trajectory value function performs the best in the value function test set, but slightly worse than the Root-state-andcurrent-state value function in end-to-end proof successful rate.

heading: 4 Experiments

heading: 4.1 Experimental setup
text: Implementation details. In this paper, we adopt the same model setup from PACT (Han et al.,
2021)2 in Lean formal environment and Thor(Jiang et al., 2022) in Isabelle formal environment for better comparison. Detailed model configuration and training procedure are described at Sec. A.3 in Appendix.
Dataset. To validate our proposed DT-Solver, we choose the Mathlib dataset in the Lean formal environment and the PISA dataset constructed from Archive of Formal Proofs (AFP) in the Isabelle formal environment. Although both datasets are extracted from the world’s largest formal mathematical libraries, these two datasets have very different characteristics. Mathlib’s proof steps are instructional, such as applying a theorem or simplifying a goal. Contrary, Isabelle’s proof steps are more human-friendly and declarative. The proof step is usually led by a conjecture to proof and followed by short instructions to prove it. Empirically, it is much harder for the model to suggest a correct conjecture. Detailed statics regarding the training dataset are shown in Table.4 and Table.5 in Appendix.
Baseline methods. For the Isabelle formal environment, we compare our DT-Solver with Lisa (Jiang et al., 2021)3, the first work that applies the GPT-f model to the Isabelle environment. In addition, Lisa proposes to add previous proof context to the original proof state to help the lan-
2Unfortunately, we cannot use models with the same pretraining configuration since the data for pre-training have not been released.
3We leave the comparison of Thor (Jiang et al., 2022) in the future work since the use of the Sledgehammer is compatible with our DT-solver and is a guaranteed improvement.
guage model predict the following proof states. For the Lean formal environment, besides the original GPT-f model (Polu and Sutskever, 2020), we compare our DT-Solver with PACT (Han et al., 2021) and Expert iteration (Polu et al., 2022) PACT (Proof Artifact Co-training) built upon the GPT-f model and proposed to co-train the language model with nine auxiliary tasks. Expert iteration bootstraps the language model by training the model with self-generated data from proof searches. All the baseline methods described above use the bestfirst search algorithm described in Sec. 2.2.
DT-Solver follows previous work to construct a policy model. For fair comparisons, we use the same policy models from baseline methods. Specifically, three policy models are trained: Lisa, PACT, and Expert iter. We evaluate our DT-Solver by substituting the best-first search with dynamic-tree sampling and entire-trajectory proof-level value function. For ablation, we substitute dynamic-tree sampling with classic Monte-Carlo tree search but leave the policy and critic models unchanged. All the baseline methods are re-implemented by ourselves since none of them releases the code.

heading: 4.2 Main results

heading: 4.2.1 Comparison with state-of-the-art
text: As shown in Table 1, our proposed DT-Solver dramatically outperforms all the baseline methods under all scenarios. Specifically, Expert iter.+DTSolver improves from 39.9% to 48.4% in the lowresource setting, surpassing the model performance of 45.9% for the Expert iter. method in high-
resource settings. Moreover, in the low-resources setting, PACT+DT-Solver achieves a success rate of 37.3%, only 2.6% worse than the Expert iter. baseline of 39.9%. This result implies that our proposed DT-Solver can substantially close the gap brought by different policy models and empower weaker models to solve more difficult problems. The same improvement applies to the PISA dataset, where the Lisa+DT-Solver model improves from 16.8% to 26.7% in the Lisa baseline. In the high resources setting, the effectiveness of DT-Solver is marginalized by high expansion samples in the best-first search algorithm. Nevertheless, Our DTSolver can still improve upon baseline methods 2.25% on average.
Focusing on how well different search algorithms can find the correct proof state to perform expansion, we calculated the global on-track rate (OTR) and successful on-track rate (SOTR). Ontrack rate averages the rate of selecting the correct nodes to expand in each proof search. As shown in the Table, our proposed DT-Solver has the highest global on-track rate in most scenarios. The result shows that DT-Solver with proof-level value function can locate states requiring more exploration and finds the proof efficiently.
The best success on-track rate and shortest average search time are biased toward weaker policy models. The high successful on-track rate with weak policy models is accomplished by only being able to solve simple theorems within one or two steps long. Meanwhile, the short average search time is because a weak policy model with the best-first search quickly exhausted the plausible action, resulting in a state named "empty queue" that quickly declares the search’s failure. Although DTSolver takes more time to solve a theorem, the algorithm makes maximum utility within the given time limitation. On average, Expert iter.+DT-Solver
only uses 10.2 seconds more to find 5.4% more proofs that Expert iter. in the high-resource setting. Calculating only the successful proof search, Expert iter.+DT-Solver in the low-resources setting uses 75.6 seconds on average to solve a problem, which is 11.6 seconds less than the high-resources setting counterpart Expert iter.

heading: 4.2.2 The effect of dynamic-tree sampling
text: We compare our proposed dynamic tree sampling with the classic Monte-Carlo tree search algorithm (MCTS). MCTS disables dynamic tree sampling by restricting the selection process until a nonexpanded leaf node, denoting no backtracking capabilities. As shown in Table. 1, PACT+MCTS and Expert iter.+MCTS both improved against PACT and Expert iter. 0.95% on average. This shows MCTS a better algorithm to locate correct proof states in proof searches. Compared to DT-Solver, the model’s performance dropped drastically without the ability to backtrack to previous states. In the low-resources setting, MCTS, on average, drops 11.25% in success rate compared to DT-Solver. We observe similar performance drops in the PISA dataset. From these results, we believe that the back-tracking capability in DT-Solver plays a vital role in improving the algorithm effect.

heading: 4.2.3 The effect of proof-level value function
text: To validate the effectiveness of our proof-level value function, we substitute the proof-level value function with outcome value function (Han et al., 2021) and log prob value function (Polu and Sutskever, 2020). Additionally, We calculate the value function’s accuracy on the created value function test set to evaluate the performance of different value functions. As shown in Table 2, the critic model using Roberta as the classifier backbone performs better than the GPT counterpart (Log prob and Outcome). The entire-trajectory value func-
tion substantially outperforms the outcome objective baseline in test set accuracy. Furthermore, the previous-to-current-state value functions provide the best end-to-end result. The result indicates that the state value is better estimated by more recent representations, instead of a remote anchor.
The minor disadvantage of the entire-trajectory value function might be the over-length trajectories, which need to be truncated to feed into the language model for prediction. Specifically, trajectories with lengths exceeding the maximum LM input were truncated from the left, resulting in the absence of the root state being preserved as a prefix. Among all value function queries, truncation occurred at a frequency of 53.00%, covering 53.54% of theorems in the Lean test set. However, when employing the "root-state-and-current-state" strategy, truncation only took place 3.10% of the time. We have extracted a sub-dataset that solely contains theorems with shorter proofs. This subset enables one to calculate the "entire trajectory" value function without resorting to any truncation. Table 3 shows the performance of root-state-and-current-state and
entire-trajectory strategies in the short-proof subset. This outcome validates our initial hypothesis that truncation harms the overall performance of the value function.

heading: 4.2.4 The effect of balancing exploration and exploitation
text: In this section, we seek to understand how c in Eq.1 and Eq.2 affect model performance. As shown in Fig.5, with larger c, the dynamic tree sampling weights more on the model prior to determining which node to select. DT-Solver achieves a better end-to-end success rate with larger c. This result indicates that more exploration and back-tracking are beneficial to find more plausible actions. However, the SOTR achieves the best performance in c = 0.5; this shows better selection accuracy when we focus more on exploiting the value function’s state estimates.

heading: 4.3 Case Study
text: In this section, we conduct a detailed case study for a close-up look at the result DT-solver produced. More examples are shown in Sec. A.4. As shown in Fig. 4, we compare proof from DT-solver with the expert iteration policy model and best-first searchbased expert iteration policy model. The theorem rpow_le_rpow_left_iff_of_base_lt_one aims to prove that for x ∈ (0, 1) and y, z ∈ R, we have xy ≤ xz if and only if z ≤ y. Both approaches can produce (by prediction and search)
the same first 6 proof steps. These steps rewrite (with the tactic rw) the goal with previously proved theorems such as rpow_def_of_pos (if x > 0 then xy = exp(x · log y)) and exp_le_exp (ex ≤ ey if and only if x ≤ y).
Here, we focus on Fig. 4 left. After the tactic application of the first rw [<- not_lt], DT-solver fails to generate the second rw [<- not_lt] subsequently. Ordinarily, such a malfunction in the best-first search algorithm would lead to the persistence of incorrect states without the opportunity to re-explore the node after the first application of rw [<- not_lt]. Nevertheless, the utilization of our Dynamic-tree sampling approach and proof-level value function facilitates the algorithm’s return to the node after 9 attempts to explore other search branches. Upon the decision to re-expand the state, there were 20 unexpanded states, and the total number of different states stood at 41. Thus the prooflevel value function plays an essential role in identifying the appropriate states for re-expansion.
With the best-first search algorithm, we see that the search tree has a width of 1 or 2 and one easily falls into a state of ‘empty queue’, where no candidate generated by the language model is applicable. The approach stops the search process and returns failed before the time budget (300 seconds) is used up. While for DT-Solver, the number of child nodes varies from 1 to 5 and has greater variance than the best-first search. Although the number of child nodes can be relatively larger, the search process of DT-Solver manages to achieve a good balance between exploration and exploitation. Eventually, DT-Solver finds proof of 11 steps within the time budget with higher OTR (36.3% > 21.8%) than the best-first search.

heading: 5 Conclusion
text: In this work, we introduced a new ATP method, DT-Solver, which uses a proof-level value function to guide the dynamic tree sampling algorithm. DTSolver smartly allocates computational budget to states that require more exploration and reduces the cost on easy states. The proof-level value function effectively locates difficult states. Extensive tests show that our method can indeed improve success rates on both PISA and mathlib datasets while being efficient enough to find proofs within a limited time budget. Although our method brings substantial advantages, there remain multiple aspects for improvement in the future, such as utilizing the for-
mal environment’s feedback or reducing enormous search spaces with high-level proof planning.

heading: 6 Limitations
text: Due to the limited ability of the policy model, most theorem still failed to find the proof because of poorly suggested proof steps. Predicting the proof step from the proof state requires substantial reasoning ability. It’s observed in the experiment that the language model tends to produce the same proof step in training data, and is unsatisfactory in generalizing for new states. Another limitation resides in the proof-level value functions. Although the performance of the proof-level value functions shows promising improvement in the value function test set. The end-to-end pass rate diminishes the performance gap. This accounts for two major reasons: 1) weak policy model fails to produce correct action even if our value function correctly located the state to expand. 2) Our value function’s performance is still behind the performance threshold where the value really helps the search drastically. One future direction is to enhance the language model for better reasoning ability by using a larger language model or adding symbolic reasoning into the language model to produce more reasonable proof steps and better evaluate states’ value.

heading: 7 Acknowledgement
text: This work was supported in part by the National Key R&D Program of China under Grant No. 2020AAA0109700, Shenzhen Science and Technology Program (Grant No. RCYX20200714114642083) and Shenzhen Fundamental Research Program(Grant No. JCYJ20190807154211365)

heading: A Appendix
text: A.1 Overview
In the appendix, we first discuss related works in Section A.2. We give more implementation details and dataset statistics of DT-Solver in Section A.3. More examples of theorem proven by DT-Solver are shown in Section A.4.
A.2 Related works
Deep learning has already been applied to automated theorem proving prior to
GPT-f (Polu and Sutskever, 2020). Methods such as DeeMath (Alemi et al., 2016), Holophrasm (Whalen, 2016), HOList (Bansal et al., 2020; Rabe et al., 2020) and MetaGen (Wang and Deng, 2020) all apply one or several neural networks to recommend premises and proof steps. Then the search can either be guided by a neural network or not.
GPT-f (Polu and Sutskever, 2020) uses a finetuned GPT-2 (Radford et al.) as the language model to recommend the next proof step for the current proof state (called tactic state in Lean) and apply a best-first search based on the log-prob of the sequence predicted by the language model. Based on this framework, PACT (Han et al., 2021) provides a multi-tasking training scheme for a slightly larger GPT-2 model. Polu et al. (Polu et al., 2022) further introduce expert iteration (Silver et al., 2017) to achieve a sort of data augmentation to improve the training of the language model further. Most recently, HTPS (Lample et al., 2022) plugs in MonteCarlo Tree Search (Silver et al., 2016) in this framework and applies an online version of expert iteration, which further advances the state-of-the-art performance on Metamath and the Lean benchmark miniF2F (Zheng et al., 2021).
Approaches in a similar paradigm have also been successfully applied to other formal systems such as Isabelle (Paulson, 1994; Li et al., 2020; Jiang et al., 2021, 2022), Lean (de Moura et al., 2015; Han et al., 2021; Polu et al., 2022; Lample et al., 2022) and other customized systems such as INT (Wu et al., 2021; Polu and Sutskever, 2020) and Equations (Lample et al., 2022).
A.3 Experimental details
A.3.1 Experimental setup
Model specification. The policy model is a decoder-only transformer (Vaswani et al., 2017) language model with 774M parameters, 36 layers, 20 attention heads, a hidden dimension of 1280, and a GPT-2 (Radford et al., 2018) tokenizer with 50400 vocabulary size. The model is pre-trained on Github python codes and the arXiv library. For lean, we further pre-train our policy model on the PACT dataset (mix1 and mix2 as denoted in PACT). During per-training, we use a global batch size of 512 with 2500 steps warmup using the AdamW optimizer. The cosine learning rate scheduling strategy is used with a maximum learning rate of 5 ∗ 10−5 and a minimum learning rate 5 ∗ 10−6.
The policy model is pre-trained for 250000 step. For fine-tuning in Lean, we use the tactic dataset in PACT to train the model, with a batch size of 16, and a warmup step of 1000 with a maximum learning rate of 5 ∗ 10−5 and a minimum learning rate of 5 ∗ 10−6. We are early-stoping the finetuning in 15000 steps with a total training budget of 100000 step. For Isabelle, we follow instructions from Lisa (Jiang et al., 2021) and reproduce the AFP dataset. The policy model is further finetuned on the AFP dataset with the same fine-tuning configuration used in Thor.
For the critic model, we use a pre-trained RoBERTa-base model (Liu et al., 2019) as our language model classifier. The critic model is finetuned for one epoch on the dataset generated following procedures described in Sec.3.1. Fine-tuning critic model uses a global batch size of 128, a maximum sequence length of 512, a linear learning secluding strategy with 2000 warmup steps, and a maximum learning rate of 1 ∗ 10−5. We early stop the training with the lowest evaluation loss.
Dyamic tree sampling configuration. We set the balancing factor c = 1 in Eq. 1 and Eq. 2. To sample from the policy model, we use temperature T = 1.2 for Isabelle and T = 1 for Lean. Each proof step has a timeout limit of 10 seconds, and the search is terminated in the following condition: (1) A proof of the theorem has been found (2) Global timeout for 300 seconds (3) A total expansion step of 128 is reached.
Machine Configuration. We use Nvidia V100 GPU with 32GB of GPU memory for pre-training and fine-tuning. The training server has 104 CPU cores and 768GB of CPU memory. For training the policy model and the critic model, we use 8 GPUs with data parallel to speed up the training, with estimated 1100 GPU hours to run the training. For running the dynamic tree sampling algorithm, we use 32 GPUs to speed up the proof-finding procedure. We estimate it takes 64 GPU hours to run a single evaluation on the mathlib test set.
Interactive theorem provers To expediently verify proofs in Lean and Isabelle, we use REPL wrappers that formulate the interactions with ITPs in a REPL style. For lean, we use lean-gym following (Polu et al., 2022). In every step, lean-gym takes a proof state and a proof step as input and outputs a new proof state. For Isabelle, we created a REPL verifier named isabelle-gym based on PISA, with the same IO specification as lean-gym.
With these formal environments, we can check if a proof step is applicable or whether we have proven the theorem.
A.3.2 Dataset statistics We provide statistics for our training dataset. Detailed statics for pre-training are shown in Table.4. We use 159G Python code collected from GitHub and PACT-mix{1,2} dataset for Mathlib pre-train. Moreover, we follow (Jiang et al., 2022) for Isabelle pre-train with GitHub and Arxiv dataset from the-pile (Gao et al., 2020). We also provide the number of theorems for Mathlib and PISA datasets in Table.5. The Mathlib dataset is created with versions of b72300f3455ae73c3ab9ed40fc1f80bbb9c85ba4 and lean core versions of 3.39.1. The PISA dataset is created with Isabelle2021. For running proof searches in the test set in Isabelle, we use the ‘quick test problem’ crated following Thor, which contains 600 theorems randomly sampled from the test split of the PISA dataset.
A.4 Proof examples In this section, we provide more examples of proof found by our DT-Solver. For each proof, we also compare our method with the best-first search counterpart using the same policy model. Specifically, we provide two cases for proofs found in the Lean formal environment shown in Fig.6 and Fig.7. As well as two cases for proofs found in the Isabelle formal environment shown in Fig.8 and Fig.9.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
3 A2. Did you discuss any potential risks of your work? Section 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
3 A4. Have you used AI writing assistants when working on this paper? I use chatGPT and use it to polish the language of my paper and correct the grammar error. This happens in section 4.2.3 and section 4.3
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? References
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section A.3
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section A.3
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section A.3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section A.3.2
C 3 Did you run computational experiments? Section A.3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section A.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section A.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section A.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: One future direction is to enhance the language model for better reasoning ability by using a larger language model or adding symbolic reasoning into the language model to produce more reasonable proof steps and better evaluate states’ value