REFACTOR: Learning to Extract Theorems from Proofs

1 INTRODUCTION . In the history of calculus , one remarkable early achievement was made by Archimedes in the 3rd century BC , who established a proof for the area of a parabolic segment to be 4/3 that of a certain inscribed triangle . In the proof he gave , he made use of a technique called the method of exhaustion , a precursor to modern calculus . However , as this was a strategy rather than a theorem , applying it to new problems required one to grasp and generalize the pattern , as only a handful of brilliant mathematicians were able to do . It wasn ’ t until millennia later that calculus finally became a powerful and broadly applicable tool , once these reasoning patterns were crystallized into modular concepts such as limits and integrals . A question arises – can we train a neural network to mimic humans ’ ability to extract modular components that are useful ? In this paper , we focus on a specific instance of the problem in the context of theorem proving , where the goal is to train a neural network model that can discover reusable theorems from a set of mathematical proofs . Specifically , we work under formal systems where each mathematical proof is represented by a tree called proof tree . Moreover , one can extract some connected component of the proof tree that constitutes a proof of a standalone theorem . Under this framework , we can reduce the problem to training a model that solves a binary classification problem where it determines whether each node in the proof tree belongs to the connected component that the model tries to predict . To this end , we propose a method called theoREm-from-prooF extrACTOR ( REFACTOR ) for mimicking humans ’ ability to extract theorems from proofs . Specifically , we propose to reverse the process of human theorem extraction to create machine learning datasets . Given a human proof T , we take a theorem s that is used by the proof . We then use the proof of theorem s , Ts , to re-write T as T ′ such that T ′ no longer contains the application of theorem s , and replace it by using the proof Ts . We call this re-writing process the expansion of proof T using s. The expanded proof T ′ becomes the input to our model , and the model ’ s task is to identify a connected component of T ′ , Ts , which corresponds to the theorem s that humans would use in T . We implement this idea within the Metamath theorem proving framework – an interactive theorem proving assistant that allows humans to write proofs of mathematical theorems and verify the correctness of these proofs . Metamath is known as a lightweight theorem proving assistant , and hence can be easily integrated with machine learning models ( Whalen , 2016 ; Polu & Sutskever , 2020 ) . It also contains one of the largest formal mathematics libraries , hence providing sufficient background for proving university-level or Olympiad mathematics . While our approach would be applicable to other formal systems ( such as Lean ( de Moura et al. , 2015 ) , Coq ( Barras et al. , 1999 ) , or HOL Light ( Harrison , 1996 ) ) , we chose Metamath for this project because of its features for reduced iteration time in the near term . Our work establishes the first proof of concept using neural network models to extract theorems from proofs . Our best REFACTOR model is able to extract exactly the same theorem as humans ’ ground truth ( without having seeing instances of it in the training set ) about 19.6 % of time . We also observe that REFACTOR ’ s performance improves when we increase the model size , suggesting significant room for improvement with more computational resources . Ultimately , the goal is not to recover known theorems but to discover new ones . To analyze those cases where REFACTOR ’ s predictions don ’ t match the human ground truth , we developed an algorithm to verify whether the predicted component constituent a valid proof of a theorem , and we found REFACTOR extracted 1907 valid , new theorems . We also applied REFACTOR to proofs from the existing Metamath library , from which REFACTOR extracted another 16 novel theorems . Remarkably , those 16 proofs are used very frequently in the Metamath library , with an average usage of 733.5 times . Furthermore , with newly extracted theorems , we show that the human theorem library can be refactored to be more concise : the extracted theorems reduce the total size by approximately 400k nodes . ( This is striking since REFACTOR doesn ’ t explicitly consider compression as an objective . ) Lastly , we demonstrate that training a prover on the refactored dataset leads to a 14-30 % relative improvement on proof success rates in proving new test theorems . Out of all proved test theorems , there are 43.6 % of them use the newly extracted theorems at least once . The usages also span across a diverse set of theorems : 141 unique newly extracted theorems are used , further suggesting diverse utility in new theorems we extracted . Our main contributions are as follows : 1 . We propose a novel method called REFACTOR to train neural network models for the theorem extraction problem , 2 . We demonstrate REFACTOR can extract unseen human theorems from proofs with a nontrivial accuracy of 19.6 % , 3 . We show REFACTOR is able to extract frequently used theorems from the existing human library , and as a result , shorten the proofs of the human library by a substantial amount . 4 . We show newtheorem refactored dataset can improve baseline theorem prover performance significantly with newly extracted theorem being used frequently and diversely . 2 RELATED WORK . Lemma Extraction Our work is generally related to lemma mining in Vyskočil et al . ( 2010 ) ; Hetzl et al . ( 2012 ) ; Gauthier & Kaliszyk ( 2015 ) ; Gauthier et al . ( 2016 ) and mostly related to the work of Kaliszyk & Urban ( 2015 ) ; Kaliszyk et al . ( 2015 ) . The authors propose to do lemma extraction on the synthetic proofs generated by Automated Theorem Provers ( ATP ) on the HOL Light and Flyspeck libraries . They showed the lemma extracted from the synthetic proofs further improves the ATP performances for premise selection . However , their proposed lemma selection methods require human-defined metrics and feature engineering , whereas we propose a novel way to create datasets for training a neural network model to do lemma/theorem selection . Unfortunately , as the Metamath theorem prover is not equipped with ATP automation to generate synthetic proofs , we could not easily compare our method to these past works . We leave more thorough comparisons on the other formal systems to future work . Discovering Reusable Structures Our work also is related to a broad question of discovering reusable structures and sub-routine learning . One line of the work that is notable to mention is the Explore-Compile-style ( EC , EC2 ) learning algorithms ( Dechter et al. , 2013 ; Ellis et al. , 2018 ; 2020 ) . These works focus on program synthesis while trying to discover a library of subroutines . As a subroutine in programming serves a very similar role as a theorem for theorem proving , their work is of great relevance to us . However they approach the problem from a different angle : they formalize sub-routine learning as a compression problem , by finding the best subroutine that compresses the explored solution space . However , these works have not yet been shown to be scalable to realistic program synthesis tasks or theorem proving . We , on the other hand , make use of human data to create suitable targets for subroutine learning and demonstrate the results on realistic formal theorem proving . Another related line of work build inductive biases to induce modular neural networks that can act as subrountines ( Andreas et al. , 2015 ; Gaunt et al. , 2017 ; Hudson & Manning , 2018 ; Mao et al. , 2019 ; Chang et al. , 2019 ; Wu et al. , 2020 ) . These works usually require domain knowledge of sub-routines for building neural architectures hence not suitable for our application . Machine Learning for Theorem Proving Interactive theorem provers have recently received enormous attention from the machine learning community as a testbed for theorem proving using deep learning methods ( Bansal et al. , 2019a ; b ; Gauthier et al. , 2018 ; Huang et al. , 2019 ; Yang & Deng , 2019 ; Wu et al. , 2021 ; Li et al. , 2021 ; Polu & Sutskever , 2020 ) . Previous works demonstrated that transformers can be used to solve symbolic mathematics problems ( Lample & Charton , 2020 ) , capture the underlying semantics of logical problems relevant to verification ( Hahn et al. , 2020 ) , and also generate mathematical conjectures ( Urban & Jakubův , 2020 ) . Rabe et al . ( 2020 ) showed that self-supervised training alone can give rise to mathematical reasoning . Li et al . ( 2021 ) used language models to synthesize high-level intermediate propositions from a local context . Piotrowski & Urban ( 2020 ) used RNNs to solve first-order logic in ATPs . Wang et al . ( 2020 ) used machine translation to convert synthetically generated natural language descriptions of proofs into formalized proofs . Yang & Deng ( 2019 ) augmented theorem prover with shorter synthetic theorems which consist of arbitrary steps from a longer proof with maximum length restriction . This is remotely related to our work where our extraction does not have such restrictions and instead closely mimic what human mathematicians would do . 3 METAMATH AND PROOF REPRESENTATION . In this section , we describe how one represents proof in the Metamath theorem proving environment . We would like to first note that even though the discussion here specializes in the Metamath environment , most of the other formal systems ( Isabelle/HOL , HOL Light , Coq , Lean ) have very similar representations . The fundamental idea is to think of a theorem as a function , and the proof tree essentially represents an abstract syntax tree of a series of function applications that lead to the intended conclusion . Proof of a theorem in the Metamath environment is represented as a tree . For example , the proof of the theorem a1i is shown in Figure 1 ( a ) . Each node of the tree is associated with a name ( labeled as N ) , which can refer to a premise of the theorem , an axiom , or a proved theorem from the existing theorem database . Given such a tree , one can then traverse the tree from the top to bottom , and iteratively prove a true proposition ( labeled as PROP ) for each node by making a step of theorem application . The top-level nodes usually represent the premises of the theorem , and the resulting proposition in the bottom node matches the conclusion of the theorem . In such a way , the theorem is proved . We now define one step of theorem application . When a node is connected by a set of parent nodes , it represents a step of theorem application . In particular , one can think of a theorem as a function that maps a set of hypothesis to a conclusion . Indeed , a node in the tree exactly represents such function mapping , that is to map the set of propositions of the parent nodes , to a new conclusion specified by the theorem . Formally , given a node c whose associated name refers to a theorem T , we denote its parent nodes as Pc . We can then prove a new proposition by applying the theorem T , to all propositions proved by nodes in Pc . The proof of the theorem a1i in Figure 1 ( a ) consists of 3 theorem applications . In plain language , the theorem is a proof of the fact that if ph is true , then ( ps- > ph ) is also true . The top-level nodes are the hypotheses of the theorem . Most of the hypotheses state that some expression is a well-formed formula so that the expression can be used to form a syntactically correct sentence . The more interesting hypothesis is a1i.1 , which states |-ph , meaning ph is assumed to be true . In the bottom node , the theorem invokes the theorem ax-mp , which takes in four propositions as hypotheses , and returns the conclusion |- ( ps- > ph ) .

Summary: The idea of the paper is based on the view of a theorem + proof as a tree, wherein sub-trees are also theorem + proofs. The goal is to extract useful lemmas from existing proof trees. This is then cast as the problem of predicting whether each node should be extracted as a new theorem (or lemma). The main idea of the paper is to construct a training set for this prediction problem by replacing calls to lemmas with their proof tree, and labelling the nodes which root these newly inserted trees as positive examples, i.e. those to be extracted as useful lemmas.