TacticZero: Learning to Prove Theorems from Scratch with Deep Reinforcement Learning

1 Introduction . Interactive theorem proving ( ITP ) is the process of humans interacting with a computer system to develop formal proofs of mathematical theorems . In ITP , a human user can define mathematical objects , and then guide the computer to prove theorems about them using commands called tactics . Tactics are programs that embody high-level proof strategies such as simplification and induction . Successful tactic application converts the current target goal ( a theorem to be proved ) into zero or more subgoals that remain to be proved . Proofs written in an ITP system are ultimately checked by a machine , and therefore , they are much more trustworthy than pencil-and-paper proofs . For this reason , ITP has gained success in both formalizing mathematics , and in verifying computer programs [ Leroy , 2009 , Hales et al. , 2017 ] . Nevertheless , because a great amount of formal detail needs to be spelled out when using an ITP system , substantial human inputs are required to fill the gaps between steps in a proof . In particular , in terms of tactic-based theorem proving , human guidance in the selection of both tactics , and the arguments to tactics is crucial to a successful proof . This requires expert knowledge both of the relevant mathematics , and the particular ITP system being used . This requirement for expertise has in turn limited the application of ITP systems . One promising line of work to address this limitation has been to use machine learning methods to replace the role of human experts in ITP . Existing learning-based approaches have two noticeable 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . characteristics : learning from human examples for tactic prediction and using a fixed proof search strategy ( e.g . breadth/best-first search ( BFS ) ) that is separate from the learning process to find proofs . The dependency on human examples limits the application of this kind of techniques depending on the availability and quality of the human proofs in the domain that the proof is required . There are potentially infinitely many proofs of a theorem , but there are only one or two of them exist in the library without any gurantee of being optimal . For this reason , it is essential for an agent to be able to explore different proofs by itself in order to generalize well and capture the underlying mathematical knowledge . On the other hand , fixed proof search strategies tend to be artificial and suboptimal . Search strategies such as BFS are usually computationally expensive in time and space ( due to needless proof states resulting from redundant expansion of previous proof states ) , and are thus not suitable for learning algorithms that require fast iteration . We propose a reinforcement-learning ( RL ) framework to learning ITP that addresses the above limitations of the existing approaches . The RL agent interacts ( Figure 1a ) with the HOL4 interactive theorem prover [ Slind and Norrish , 2008 ] and mimics the behavior of human ITP experts . Given a proof state , a human expert often decides not only what tactic and arguments to apply , but also whether or not to go back to a previous proof state and restart the proof from there . This kind of situation occurs frequently in ITP as careless tactic applications can quickly result in unmanageable proof states . An experienced ITP expert would have the intuition on when and where to backtrack . Our agent learns proof search strategies in a similar manner by learning to choose promising derivations and subgoals to attack , as well as the tactics and their arguments to apply to the goals , without using the limited examples in the library . Our Contributions are as follows . • We provide a formulation of ITP as a Markov decision process ( MDP ) [ Bellman , 1957 ] in which flexible state representations enable tracking multiple subproofs during the search process . The structure of the action space allows for tactics to have both theorem names and terms as input arguments . • We propose an RL architecture using multiple recurrent and feed-forward neural network modules to solve the above MDP . We use policy gradients [ Williams , 1992 ] to jointly learn to apply actions at the backtracking , goal , tactic and argument levels . • We implement the framework in the HOL4 theorem prover and show that it outperforms state-of-the-art automated theorem provers ( i.e . hammers [ Gauthier and Kaliszyk , 2015 ] ) available in HOL4 including E [ Schulz et al. , 2019 ] , Z3 [ de Moura and Bjørner , 2008 ] and Vampire [ Kovács and Voronkov , 2013 ] . We further use ablation studies to establish the contribution of key components of our architecture in achieving the overall performance . 2 Interactive Theorem Proving with Reinforcement Learning . Theorem example : p ∧ q ⇒ p ∧ q Proof rpt strip_tac > - ( * Induct_on ‘ p ’ * ) simp [ ] > - simp [ ] QED Figure 2 : A Simple HOL4 Proof . We begin with an overview of interactive proving in HOL4 by a simple example ( corresponding to the script in Figure 2 ) . Suppose we want to prove the theorem p ∧ q ⇒ p ∧ q where p and q are propositional variables ( i.e. , boolean ) . This expression is called our initial goal . Since it is an implication , we will assume the antecedent , and then use that to prove the conclusion . This can be done by applying a tactic , rpt strip_tac to the goal . This tactic does not take any arguments . HOL4 tells us that the tactic application results in two new subgoals . In this case , they are p⇒ q ⇒ p and p⇒ q ⇒ q.1 At this stage , both subgoals need to be proved in order for the proof to succeed . Suppose we choose to first prove p⇒ q ⇒ p. We notice that because p and q are boolean variables , it should be possible to prove the goal by cases . This can be done by applying the Induct_on tactic to the goal . The Induct_on tactic takes a single term argument , that is , an arbitrary HOL4 expression . In this case , it will be the variable p. 1HOL4 actually moves p and q to what it calls the goal ’ s assumption-list ; for simplicity here , we represent that list with chained implications in one formula . After applying Induct_on ‘ p ’ to p ⇒ q ⇒ p , two more subgoals are generated , with the variable p replaced by truth values T and F respectively . At this stage , we have three subgoals to deal with . One is the earlier goal that we chose to defer : p⇒ q ⇒ q . The other two are the newly generated T⇒ q ⇒ T and F⇒ q ⇒ F. One might decide at this stage that the previous tactic application of Induct_on ‘ p ’ has made the situation worse because it has left us with more goals to prove . A human expert may now look back at the remaining subgoals before the application of Induct_on ‘ p ’ and try another tactic . Fortunately , there is a tactic called simp that can prove the goal p⇒ q ⇒ p directly . The simp tactic takes a list of theorem names as its argument . In this case , an empty argument list is sufficient to prove the goal . Similarly , simp [ ] proves the remaining p⇒ q ⇒ q goal as well . 2.1 Modeling ITP with MDP . Tactic-based theorem proving is naturally understood as a tree search problem where a tree is composed of sets of goals and can be updated by applying tactics to a node . Below we give a formulation that reflects such intuition — states are the search trees and actions are the tactic applications to nodes . States A proof attempt starts with a main goal g ∈ G. At any point in a proof attempt , there is a set of goals that all need to be proved . We refer to these finite sets of goals as fringes . In the context of our framing as an MDP , multiple fringes will be generated and maintained in such a way that the main goal will be proved if everything in any one fringe is proved . Thus a fringe represents a particular path along which we may continue in an attempt to complete a single valid HOL4 proof of the main goal . In contrast , always choosing the most recently generated fringe would be equivalent to never backtracking in HOL4 . The MDP state s is therefore a finite sequence of fringes and we denote the i-th fringe in state s by s ( i ) . See Figure 1b for an example state sequence . Actions An action is a 4-tuple ( i , j , t , c ) ∈ N×N×T ×A∗ . Intuitively , given a state s , i selects the i-th fringe in s and j selects the j-th goal within fringe s ( i ) . Then , t is the HOL4 tactic that we select from the set T of possible HOL4 tactics , and c is the possibly empty list of arguments that accompany t ( see Figure 1b ) . An argument in A is either a theorem name , or a HOL4 term . The existence of arguments makes the action space arbitrarily large—terms may be arbitrarily defined , and there are thousands of theorems that can be given as arguments to the tactic . This necessitates an RL algorithm that can handle large action and state spaces , for which we select policy gradients [ Williams , 1992 ] . Rewards Feedback is received from HOL4 after each tactic application . If successful , it generates a set of new ( sub- ) goals such that proving all of them proves g ; if this set is empty , then g is itself proven . A tactic application may fail , either with an error indicating that it is not applicable , or with a timeout caused by HOL4 exceeding a maximum computation time which we set . In terms of our framing as an MDP , different numeric rewards are associated with the different cases described above . MDP State Transition A proof attempt always starts with a single main goal g , and so the initial state is a single fringe containing one element , g. Given a state s , performing an action ( i , j , t , c ) may not change the state . This happens when the tactic times out , has no effect , or is not applicable . Otherwise , the application of the tactic generates a set of new subgoals . This set G of new subgoals may be empty , indicating that goal s ( i ) ( j ) is immediately proved by tactic t. In any case , a new fringe is then constructed by first copying fringe s ( i ) , and then replacing the goal s ( i ) ( j ) with the new subgoals G. Then we construct a new state s′ by adding the new fringe to state s. Each state constructed in this way maintains all possible partial proof attempts . The size of each state is linear in the number of timesteps so that the manipulation and storage of states are efficient . Termination The MDP terminates when an empty fringe is constructed , which implies that we can recover a proof of the main goal . In this case , the agent receives a positive reward . The process is also terminated if the timestep budget is exceeded , in which case the proof attempt is unsuccessful and the agent receives a negative reward . The state s4 in Figure 1b is an example of a terminal state .

Summary: TacticZero proposes a reinforcement learning approach to theorem proving in interactive theorem provers (HOL4 in this paper, but the approach could be applied to others). The RL agent selects both a node in the search tree to expand and a proof tactic to apply in this state. This means the RL agent learns to generate good proof steps, but also learns how to search effectively. The paper formalizes this process as an MDP and demonstrates experimentally, that we can learn good search strategies in this way (compared to various BFS and DFS algorithms). The paper also shows that their approach compares favorably to existing theorem provers (not relying on deep learning). The RL agent is trained with policy gradient.