Learning to Prove Theorems by Learning to Generate Theorems

1 INTRODUCTION . Automated theorem proving is a key task in Artificial Intelligence . The goal is to automatically generate a proof , given a conjecture ( the target theorem ) and a knowledge base of known facts , all expressed in a formal language . Automated theorem proving is useful in a wide range of applications , including the verification and synthesis of software and hardware systems ( Gu et al. , 2016 ; Darvas et al. , 2005 ; Kern & Greenstreet , 1999 ) . Automated theorem proving boils down to a search problem : finding the sequence of symbol manipulations that generate a valid proof . A prover typically works backward : starting from the theorem statement , it searches for a path that connects the theorem to known facts in the knowledge base . The fundamental challenge lies in the explosion of search space , in particular with long proofs and large knowledge bases . The success of theorem proving thus relies on effective heuristics that guide the search by deciding the next step the prover should take . Deep learning has emerged as a promising approach to learning search heuristics in a automated theorem prover ( Irving et al. , 2016 ; Yang & Deng , 2019 ; Whalen , 2016 ; Loos et al. , 2017 ; Bansal et al. , 2019a ) . The search process fundamentally reduces to a sequence of actions on manipulating a set of symbols . Thus a deep network can be trained to select the best action at each step . A key challenge is how to train such networks . Prior work has used human-written theorems and proofs to perform imitation learning and has shown promising results ( Loos et al. , 2017 ; Yang & Deng , 2019 ; Whalen , 2016 ; Paliwal et al. , 2019 ) . The training data consists of theorems and proofs manually written by human experts in a formal language , and the prover is trained to imitate the proof steps demonstrated by humans . However , relying on human-written data has a major drawback , that is , such data has limited availability and scalability . Writing theorems and proofs in a formal language requires highly specialized knowledge and skill , including mathematics , computer programming , and proficiency in the particular formal language . For a computer science graduate student , it can take months to master a new formal language such as Mizar , Metamath or HOLight ( Wiedijk , 2003 ) , after which it can take days to formalize a single page of a math textbook . This makes it impractical to crowdsource human-written proofs at large scale . An alternative to imitation learning is reinforcement learning , which requires only formalized theorem statements but not their proofs . During training , the prover estimates the value of each action through exploration . This reinforcement learning approach substantially reduces the amount of manual formalization needed , but at the expense of sample efficiency . The prover needs positive rewards to assess past attempts , but positive rewards are only available when the prover finds a com- plete proof , which is rare because it involves a combination of multiple correct steps . This leads to extremely sparse positive rewards , and in turn very low sample efficiency . In this paper , we propose to learn search heuristics using synthetic data . The basic idea is to construct a generator that automatically synthesizes new theorems and their proofs , which are then used to augment human-written data . To generate a new theorem and its proof , the generator applies an inference rule on a set of existing theorems and combines their proofs to form the proof of the new theorem . Similar to the prover , the generator performs a sequence of symbol manipulations , albeit in the inverse direction , going forward from existing theorems to a new theorem instead of from a target theorem to existing ones . A key question is how to construct a generator such that the generated data is useful . The space of new theorems and proofs is infinite , but a prover can only process a finite amount of data during training . Thus , to maximize the utility of the generate data , we make the generator learnable by parametrizing it with deep networks . We hypothesize that the generated data will be more useful if they are similar to human-written data . Thus we use human-written data to train a generator . We consider two scenarios . If the humanwritten data consists of both theorem statements and their proofs , we train the generator to follow the proof steps in the forward direction , so that a well-trained generator would derive theorems humans tend to derive . If the human-written data consists of only theorem statements but not their proofs , we use reinforcement learning to train the generator such that the generated theorems are similar to the human-written theorems . We measure similarity using the language model trained on the human-written theorem . We instantiate our approach in Metamath ( Megill , 2019 ) , a popular language for formal mathematics , and with Holophrasm ( Whalen , 2016 ) , a Metamath neural prover . We propose a neural theorem generator we call “ MetaGen “ , which synthesizes new theorems and their proofs expressed in the formalism of Metamath . To the best of our knowledge , MetaGen is the first neural generator of synthetic training data for theorem proving . Experiments on real-world Metamath tasks demonstrate that synthetic data from MetaGen helps the prover prove more human-written theorems , achieving state of the art results . Experiments also show that our approach can synthesize useful data , even when there are only human-written theorems but zero proofs during training . 2 RELATED WORK . Automated theorem proving Our work is related to prior work on learning to prove theorems ( Whalen , 2016 ; Gauthier et al. , 2018 ; Bansal et al. , 2019a ; Yang & Deng , 2019 ; Loos et al. , 2017 ; Balunovic et al. , 2018 ; Kaliszyk et al. , 2018 ; Bansal et al. , 2019b ) . Our work directly builds off of Holophrasm ( Whalen , 2016 ) , a neural-augmented theorem prover for Metamath . It contains three deep networks to generate actions and initial values to guide proof search following the UCT algorithm ( Kocsis & Szepesvári , 2006 ) . TacticToe ( Gauthier et al. , 2018 ) , DeepHOL ( Bansal et al. , 2019a ) and ASTactic ( Yang & Deng , 2019 ) are learning-based theorem provers for higher-order logic based on various interactive theorem provers , including HOL4 ( Slind & Norrish , 2008 ) , HOL Light ( HOLLight ) and Coq ( Bertot & Castéran , 2004 ) . Paliwal et al . ( 2019 ) improves DeepHOL by representing formulas as graphs . Loos et al . ( 2017 ) propose to learn clause selection by deep learning inside the first-order logic prover E ( Schulz , 2002 ) . FastSMT ( Balunovic et al. , 2018 ) learns to compose search heuristics as programs with branches for the SMT solver ( De Moura & Bjørner , 2008 ) . All of these methods are othogonal to our approach because all of their provers are learned from human-written training data , whereas our contribution is on training a neural generator of synthetic training data for theorem proving . Kaliszyk et al . ( 2018 ) ; Bansal et al . ( 2019a ; b ) use reinforcement learning to train provers with only human-written theorems but not their proofs . During training , a prover only collects rewards only upon finding full proofs . In contrast , we always train our prover using imitation learning . Under the same setting with only human-written theorems but not proofs , we use reinforcement learning to train our generator , whose reward is the similarity between a generated theorem and a humanwritten theorem , as measured by a language model of human-written theorems . Our reinforcement learning task is much easier because the reward is continuous and there are many ways to generate theorems similar to human-written ones . Automatic goal generation by self-play Our work is similar to the line of work in reinforcement learning ( Florensa et al. , 2018 ; Sukhbaatar et al. , 2017 ; 2018 ; Durugkar & Stone , 2018 ) that deploys one agent to generate tasks for another agent to accomplish . Sukhbaatar et al . ( 2017 ) ; Florensa et al . ( 2018 ) propose to train these two agents by adversary self-play , where the generation agent learns to produce difficult goals for another agent . With self-play , the generator learns to increase the difficulty of goals and build a learning curriculum automatically . We pursue similar ideas in the new context of theorem proving by learning to generate synthetic theorems to train the prover . Also of note is that we have no adversarial self-play . The goal of the generator is to discover novel theorems similar to human-written ones , not to beat the prover . Recently , Huang ( 2019 ) introduced a two-player game which encourages players to learn to predict the consistency of formulas in first-order logic by self-play . These two players behave symmetrically and complete with each other in the game . In contrast , our generator and prover execute different tasks , and are co-operative . In addition , their game remains a theoretical proposal without any empirical validation , whereas we have performed experiments on large-scale data . 3 BACKGROUND ON METAMATH . Metamath is a language for developing formal mathematics . It is one of the simplest formal systems . It has only one inference rule , called substitution , but is universally applicable in formalizing a large portion of mathematics 1 and different types of logic ( Megill , 2019 ) . A knowledge base in Metamath consists of a set of theorems including axioms , which are admitted to be true , and others that are derived from proofs . Each theorem has one or more expressions , including one assertion and zero or more hypotheses . The hypotheses provide the preconditions , such as x = y2 and y is an even number , to prove the assertion , such as x is divisible by 4 . Following Whalen ( 2016 ) , an expression is represented as a tree of tokens , whose nodes are either constants or variables . A constant node has a fixed number of children ( including zero ) and a variable has no children . Therefore , we represent each expression as a unique sequence of tokens by traversing its parse tree in pre-order . A proof is a sequence of steps using substitution . A proof step has two parts , a theorem that is declared earlier than the current theorem in the knowledge base , and a substitution that maps a variable in this theorem to a new expression . For example , we have a theorem t , hypotheses : A = B ( 1 ) assertion : CFA = CFB ( 2 ) { A , B , C , F } is the set of variables in t. Let φ be a substitution to map each variable in t to a new expression , A→ 2 B → ( 1 + 1 ) C → 2 F → + ( 3 ) By replacing variables in the t with their corresponding expressions from φ , we have a new assertion and a set of new hypotheses , new hypotheses : 2 = ( 1 + 1 ) ( 4 ) new assertion : 2 + 2 = 2 + ( 1 + 1 ) ( 5 ) and this proof step ( t , φ ) demonstrates that the new assertion 2 + 2 = 2 + ( 1 + 1 ) is entailed by the new hypothesis 2 = ( 1 + 1 ) . Note that we need to substitute all occurrences of the same variable with the same expression in both the assertion and hypotheses 2 . Formally , let e ∈ E be an expression ( a tree of tokens ) with l unique variables fe = ( f1 , f2 , ... , fl ) ∈ F l , and let φ ∈ F → E be a substitution . Let e ( φ ) denote the new expression obtained by replacing 1Its largest knowledge base , set.mm ranks 3rd in the ” Formalizing 100 Theorems ” challenge ( Wiedijk ) . 2Variables in Metamath are called metavariables , which are different from variables bound by quantifiers in the first-order and higher-order logic . fi in e with φ ( fi ) , for i = 1 , 2 ... , l. And given k expressions e = ( e1 , e2 , ... , ek ) ∈ Ek , let e ( φ ) = ( e1 ( φ ) , e2 ( φ ) , ... , ek ( φ ) ) represent applying the substitution to all k expressions . Given a theorem t , let at be its assertion and ht = ( ht,1 , ht,2 , ... , ht , m ) ∈ Em be its hypotheses . Let φt be a substitution for variables in t. A proof step s = ( t , φt ) ∈ Em+1 × ( F → E ) demonstrates an entailment of assertion at ( φt ) by hypotheses ht ( φt ) . In this formal system , proving a theorem τ means finding a tree such that ( 1 ) the root node is the assertion of τ , ( 2 ) each leaf node is either empty or one of the hypotheses of τ , and ( 3 ) each internal node is an expression associated with a proof step that demonstrates an entailment of the node by its children . To prove a target theorem τ , it is the most straightforward to reason backward . We start by selecting a proof step that will demonstrate an entailment of the assertion of the target theorem , that is , at ( φ ) = aτ . It is worth noting that if we pick a particular theorem t , if a valid φ exists , φ ( f ) is uniquely determined for any variable f that occurs in the assertion ( recall that each expression is a tree of tokens ) . But φ ( f ) is not uniquely determined if f only occurs in the hypotheses ( f is called a hypothesis variable , or a assertion variable if it only occurs in the assertion ) , because it can be replaced with anything . Once this initial proof step ( t , φ ) is fully specified , the assertion aτ is entailed by a set of new hypotheses hτ ( φ ) , and the goal of proving theorem τ has now been decomposed to the subgoals of finding entailment of the new hypotheses hτ ( φ ) by the original hypotheses hτ .

Summary: This paper focuses on the problem of developing deep learning systems that can prove theorems in a mathematical formalism -- in this case, MetaMath. This has been a rapidly growing topic in the past few years, as evidenced by the numerous cited works. What sets this work apart from others is its focus on the instrumental task of generating data to train a prover, rather than directly training the prover on human theorems (via reinforcement learning) or human proofs (via imitation learning).