Proving Theorems using Incremental Learning and Hindsight Experience Replay

1 Introduction . I believe that to achieve human-level performance on hard problems , theorem provers likewise must be equipped with soft knowledge , in particular soft knowledge automatically gained from previous proof experiences . I also suspect that this will be one of the most fruitful areas of research in automated theorem proving . And one of the hardest . Schulz ( 2017 , E ’ s author ) Automated theorem proving ( ATP ) is an important tool both for assisting mathematicians in proving complex theorems as well as for areas such as integrated circuit design , and software and hardware verification ( Leroy , 2009 ; Klein , 2009 ) . Initial research in ATP dates back to 1960s ( e.g. , Robinson ( 1965 ) ; Knuth & Bendix ( 1970 ) ) and was motivated partly by the fact that mathematics is a hallmark of human intelligence . However , despite significant research effort and progress , ATP systems are still far from human capabilities ( Loos et al. , 2017 ) . The highest performing ATP systems ( e.g. , Cruanes et al . ( 2019 ) ; Kovács & Voronkov ( 2013 ) ) are decades old and have grown to use an increasing number of manually designed heuristics , mixed with some machine learning , to obtain a large number of search strategies that are tried sequentially or in parallel . Recent advances ( Loos et al. , 2017 ; Chvalovskỳ et al. , 2019 ) build on top of these provers and used modern machine learning techniques to augment , select or prioritize their heuristics , with some success . However , these machinelearning based provers usually require initial training data in the form of proofs , or positive and negative examples ( provided by the high-performing existing provers ) from which to bootstrap . Recent works do not build on top of other provers , but still require existing proof examples ( e.g. , Goertzel ( 2020 ) ; Polu & Sutskever ( 2020 ) ) . Our perspective is that the reliance of current machine learning techniques on high-end provers limits their potential to consistently surpass human capabilities in this domain . Therefore , in this paper we start with only a basic theorem proving algorithm , and develop machine learning methodology for bootstrapping automatically from this prover . In particular , given a set of conjectures without proofs , our system trains itself , based on its own attempts and ( dis ) proves an increasing number of conjectures , an approach which can be viewed as a form of incremental learning . A particularly interesting recent advance is rlCop ( Kaliszyk et al. , 2018 ; Zombori et al. , 2020 ) , which is based on the minimalistic leanCop theorem prover ( Otten & Bibel , 2003 ) , and is similar in spirit to our approach . It manages to surpass leanCop ’ s performance—but falls short of better competitors such as E , likely because it is based on a ‘ tableau ’ proving style rather than a saturation-based one . This motivated Crouse et al . ( 2021 ) to build on top of a saturation-based theorem prover and indeed see some improvement , while still not quite getting close to E. However , all previous approaches using incremental learning have a blind spot , because they learn exclusively from the proofs of successful attempts : If the given set of conjectures have large gaps in their difficulties , the system may get stuck at a suboptimal level due to the lack of new training data . This could in principle even happen at the very start , if all theorems are too hard to bootstrap from . To tackle this issue , Aygün et al . ( 2020 ) propose to create synthetic theorem generators based on the axioms of the actual conjectures , so as to provide a large initial training set with diverse difficulty . Unfortunately , synthetic theorems can be very different from the target conjectures , making transfer difficult . In this paper , we adapt instead hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) to ATP : clauses reached in proof attempts are turned into goals in hindsight . This generates a large amount of “ auxiliary ” theorems with proofs for the learner , even when no theorem from the original set can be proven . We compare our approach on a subset of TPTP ( Sutcliffe , 2017 ) with the state-of-the-art E prover ( Schulz , 2002 ; Cruanes et al. , 2019 ) , which performs very well on this dataset . Our learning prover eventually reaches equal or better performance on 16 domains out of 20 . In addition , it finds shorter proofs than E in approximately 98 % of the cases . We perform an ablation experiment to highlight specifically the role of hindsight experience replay . In the next sections , we explain our incremental learning methodology with hindsight experience replay , followed by a description of the network architecture and experimental results . 2 Methodology . For the reader unfamiliar with first-order logic , we give a succinct primer in Appendix A . From an abstract viewpoint , in our setting the main object under consideration is the clause , and two operations , factoring and resolution . These operations produce more clauses from one or two parent clauses . Starting from a set of axiom clauses and negated conjecture clauses ( which we will call input clauses ) , the two operations can be composed sequentially to try to reach the empty clause , in which case its ancestors form a refutation proof of the input clauses and correspond to a proof of the non-negated conjecture . We call the tree_size of a clause the number of symbols ( with repetition ) appearing in the clause ; for example tree_size ( p ( X , a , X , b ) ∨ q ( a ) ) is 7 . We start by describing the search algorithm , which allows us then to explain how we integrate machine learning and to describe our overall incremental learning system . 2.1 Search algorithm . To assess the incremental learning capabilities of recent machine learning advances , we have opted for a simple base search algorithm ( see also Kaliszyk et al . ( 2018 ) for example ) , instead of jump-starting from an existing high-performance theorem prover . Indeed , E is a state-of-art prover that incorporates a fair number of heuristics and optimizations ( Schulz , 2002 ; Cruanes et al. , 2019 ) , such as : axiom selection , simplifying the axioms and input clauses , more than 60 literal selection strategies , unit clause rewriting , multiple fast indexing techniques , clause evaluation heuristics ( tree size preference , age preference , preference toward symbols present in the conjecture , watch lists , etc . ) , strategy selection based on the analysis of the prover on similar problems , multiple strategy scheduling with 450 strategies tuned on TPTP 7.3.0,1 integration of a PicoSAT solver , etc . Other machine learning provers 1See http : //www.tptp.org/CASC/J10/SystemDescriptions.html # E -- -2.5. based on E ( e.g. , Jakub̊uv et al . ( 2020 ) ; Loos et al . ( 2017 ) ) automatically take advantage of at least some of these improvements ( but see also Goertzel ( 2020 ) ) . Like E and many other provers , we use a variant of the DISCOUNT loop ( Denzinger et al. , 1997 ) , itself a variant of the given-clause algorithm ( McCune & Wos , 1997 ) ( See Algorithm 3 in Appendix B ) . The input clauses are initially part of the candidates , while the active_clauses list starts empty . A candidate clause is selected at each iteration . New factors of the clause , as well as all its resolvents with the active clauses are pushed back into candidates . The given clause is then added to active_clauses , and we say that it has been processed . We also use the standard forward and backward subsumptions , as well as tautology deletion , which allow to remove too-specific clauses that are provably unnecessary for refutation . If candidates becomes empty before the empty clause can be produced , the algorithm returns `` saturated '' , which means that the input clauses are actually countersatisfiable ( the original conjecture is dis-proven ) . Given-clause-based theorem provers often use several priority queues to sort the set of candidates ( e.g. , McCune & Wos ( 1997 ) ; Schulz ( 2002 ) ; Kovács & Voronkov ( 2013 ) ) . We also use three priority queues : the age queue , ordered from the oldest generated clause to the youngest , the weight queue , ordered by increasing tree_size of the clauses , and the learnedcost queue , which uses a neural network to assign a score to each generated clause . The age queue ensures that every generated clause is processed after a number of steps that is at most a constant factor times its age . The weight queue ensures that small clauses are processed early , as they are “ closer ” to the empty clause . The learned-cost queue allows us to integrate machine learning into the search algorithm , as detailed below . 2.2 Clause-scoring network and hindsight experience replay . The clause-scoring network can be trained in many ways so as to find proofs faster . A method utilized by Loos et al . ( 2017 ) and Jakubuv & Urban ( 2019 ) turns the scoring task into a classification task : a network is trained to predict whether the clause to be scored will appear in the proof or not . In other words , the probability predicted by an ‘ in-proofness ’ classifier is used as the score . To train , once a proof is found , the clauses that participate in the proof ( i.e. , the ancestors of the empty clause ) are considered to be positive examples , while all other generated clauses are taken as negative examples.2 Then , given as input one such generated clause x along with the input clauses Cs , the network must learn to predict whether x is part of the ( found ) proof . There are two main drawbacks to this approach . First , if all conjectures are too hard for the initially unoptimized prover , no proof is found and no positive examples are available , making supervised learning impossible . Second , since proofs are often small ( often a few dozen steps ) , only few positive examples are generated . As the number of available conjectures is often small too , there is far too little data to train a modern high-capacity neural network . Moreover , for supervised learning to be successful , the conjectures that can be proven must be sufficiently diverse , so the learner can steadily improve . Unfortunately , there is no guarantee that such a curriculum is available . If the difficulty suddenly jumps , the learner may be unable to improve further . These shortcomings arise because the learner only uses successful proofs , and all the unsuccessful proof attempts are discarded . In particular , the overwhelming majority of the generated clauses become negative examples , and most need to be discarded to maintain a good balance with the positive examples . To leverage the data generated in unsuccessful proof attempts , we adapt the concept of hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) from goal-conditioned reinforcement learning to theorem proving . The core idea of HER is to take any “ unsuccessful ” trajectory in a goal-based task and convert it into a successful one by treating the final state as if it were the goal state , in hindsight . A deep network is then trained with this trajectory , by contextualizing the policy with this state instead of the original goal . This way , even in 2These examples are technically not necessarily negative , as they may be part of another proof . But avoiding these examples during the search still helps the system to attribute more significance to the positive examples . the absence of positive feedback , the network is still able to adapt to the domain , if not to the goal , thus having a better chance to reach the goal on future tries . Inspired by HER , we use the clauses generated during any proof attempt as additional conjectures , which we call hindsight goals , leading to a supply of positive and negative examples . Let D be any non-input clause generated during the refutation attempt of Cs . We call D a hindsight goal.3 Then , the set Cs∪ { ¬D } can be refuted . Furthermore , once the prover reaches D starting from Cs ∪ { ¬D } , only a few more resolution steps are necessary to reach the empty clause ; that is , there exists a refutation proof of Cs ∪ { ¬D } where D is an ancestor of the empty clause . Hence , we can use the ancestors of D as positive examples for the negated conjecture and axioms Cs ∪ { ¬D } . This generates a very large number of examples , allowing us to effectively train the neural network , even with only a few conjectures at hand . Since each domain has its own set of axioms , and a separate network is trained per domain , axioms are not provided as input to the scoring network . Although the set of active clauses is an important factor in determining the usefulness of a clause , we ignore it in the network input to keep the network size smaller .

Summary: The paper applies hindsight experience replay (HER) to automatic theorem proving (ATP). ATP attempts to prove logic statements by (roughly speaking) showing that there are no counterexamples, i.e. showing that nothing resolves the negation of a statement. HER is a technique from reinforcement learning which mitigates sparse rewards by treating failed attempts as successful attempts on a different problem, namely that with the end state of the attempt as the goal.  The paper is not quite an HER method however ; it does not use reinforcement learning but rather incremental learning, where clauses generated during proof attempts are used as additional data for (incremental) supervised learning.