Towards Finding Longer Proofs

We present a reinforcement learning ( RL ) based guidance system for automated theorem proving geared towards Finding Longer Proofs ( FLoP ) . FLoP focuses on generalizing from short proofs to longer ones of similar structure . To achieve that , FLoP uses state-of-the-art RL approaches that were previously not applied in theorem proving . In particular , we show that curriculum learning significantly outperforms previous learning-based proof guidance on a synthetic dataset of increasingly difficult arithmetic problems . 1 INTRODUCTION . In 1958 B. F. Skinner , a pioneer of modern behaviorism , in the article “ Teaching Machines ” ( Skinner , 1958 ) noticed that “ in acquiring complex behavior the student must pass through a carefully designed sequence of steps , often of considerable length . Each step must be so small that it can always be taken , yet in taking it the student moves somewhat closer to fully competent behavior ” . His study extended also to the teaching of arithmetic : “ The student is expected to arrive at 9 · 7 = 63 , not by memorizing it as he would memorize a line of poetry , but by putting into practice such principles as that nine times a number is the same as ten times the number minus the number . . . ” . The idea of learning using a curriculum of problems is also widely used in machine learning ( Bengio et al. , 2009 ; Elman , 1993 ; Resnick et al. , 2018 ; Salimans & Chen , 2018 ) and in this work we apply curriculum learning to automatic theorem proving focusing on arithmetic . Our work has the following contributions . ( 1 ) We introduce a new theorem proving algorithm FLoP ( Section 4 ) based on reinforcement learning and the connection tableau calculus . FLoP uses a meta-learning variation of the curriculum learning algorithms presented by Resnick et al . ( 2018 ) and Salimans & Chen ( 2018 ) . ( 2 ) We introduce a synthetic dataset of increasingly difficult arithmetic problems organized as RL environments ( Section 5 ) . ( 3 ) We use this benchmark to compare ( Section 6 ) the performance of our system with state-of-the-art saturation provers Vampire ( Kovács & Voronkov , 2013 ) and E ( Schulz , 2013 ) guided by human-designed strategies , and with rlCoP ( Kaliszyk et al. , 2018 ) – a recently developed RL-based connection tableau prover . FLoP significantly outperforms the other provers on harder problems , demonstrating its ability to find longer proofs . Our datasets presented in Section 5 seem to be particularly suited for machine learning methods : problems are simple , solutions are long , repetitive and rather predictable for humans . Still , state-ofthe-art systems struggle with solving some of the problems – see Section 6 for details . Other works using machine learning to guide a prover ( Chvalovský et al. , 2019 ; Jakubuv & Urban , 2017 ; Kaliszyk & Urban , 2015b ; Kaliszyk et al. , 2018 ; Loos et al. , 2017 ; Urban et al. , 2011 ) usually deal with large mathematical corpora , while we focus on a fragment of Robinson Arithmetic , which is a limited and simple theory . Our reasons behind this narrower focus : ( a ) We wanted to create a scalable RL benchmark with emphasis on the length of proofs . ( b ) A symbolic method based on human-designed sets of hints ( Veroff , 1996 ) was previously successfully applied in abstract algebra by Kinyon et al . ( 2013 ) to discover long proofs and we wanted to check whether learning of long proofs is feasible using the state-of-the-art ML toolset . ( c ) We wanted interpretable failure modes . In the case of large mathematical corpora , the interpretation of failures may be a hard task because of multiple failures and the complicated structure of the corpora , requiring specialized domain knowledge both in mathematics and with regard to the inner workings of the proof system . Our code , datasets and all experiment configuration files are available at http : //bit.ly/code_ atpcurr1 . Supplementary materials including screencasts with gameplays performed in our environments are available at the project webpage http : //bit.ly/site_atpcurr . 2 RELATED WORK . Machine learning datasets and RL environments involving mathematics and logic . The arithmetic dataset which we introduce in Section 5 is geared towards longer proofs and is structurally much simpler than other theorem proving datasets which we list below . One can think about this suite of RL problems as gridworlds of theorem proving ( see ( Sutton & Barto , 2018 , Example 3.5 ) for a broader explanation of importance of gridworlds in RL ) . Our dataset is intended to become a general purpose testing ground for theorem proving and reinforcement learning methods , in particular for meta-learning and hierarchical learning algorithms . TPTP ( Sutcliffe , 2017 ) consists of 22507 problems in 53 domains collected over several decades . A large dataset for developing machine learning for theorem proving based on the Mizar Mathematical Library ( MML ) ( Grabowski et al. , 2010 ) was introduced by Urban ( 2006a ) in the MPTP project . The dataset was used e.g . by Alemi et al . ( 2016 ) ; Kaliszyk & Urban ( 2015a ) ; Urban ( 2007 ) ; Urban et al . ( 2008 ) . Similar datasets based on the Isabelle/HOL , HOL Light/Flyspeck and HOL4/CakeML systems and projects ( Blanchette et al. , 2016 ; Gauthier & Kaliszyk , 2015 ; Kaliszyk & Urban , 2014 ) were introduced in the last decade and used for the CASC LTB ( large theory ) ATP competition ( Sutcliffe & Urban , 2016 ) and other system evaluations . Such datasets cover large areas of mathematics and computer science and contain diverse axioms , lemmas , theorems , definitions , and symbols . Smaller subsets of lemmas leading to the Bolzano-Weierstrass theorem were selected from the MPTP dataset to form the MPTP Challenge ( Urban , 2006b ) and the MPTP2078 benchmark ( Alama et al. , 2014 ) . HOLStep ( Kaliszyk et al. , 2017 ) introduced a dataset based on 11400 proofs , including a proof of the Kepler Conjecture ( Hales et al. , 2015 ) , formalized using HOL Light ( Harrison , 1996 ) . In HOLStep and in FormulaNet ( Wang et al. , 2017 ) the dataset was used as a benchmark for various neural architectures . The recent HOList project ( Bansal et al. , 2019a ; b ; Paliwal et al. , 2019 ) uses 29462 theorems formalized in HOL Light and instruments them for experiments oriented towards tactic selection , where a tactic is a human-designed program which aggregates multiple proof steps . GamePad ( Huang et al. , 2019 ) introduced a dataset based on a formalization of the Feit-Thompson Theorem ( Gonthier et al. , 2013 ) along with a generated algebra problems . It is intended for learning tactic selection together with an auxiliary task of predicting the number of proof steps left . A dataset based on theorems proved in HOL4 ( Slind & Norrish , 2008 ) was used for developing the TacticToe ( Gauthier et al. , 2018 ) learning-guided tactical prover . Saxton et al . ( 2019 ) proposed a dataset of simple algebraic problems expressed in English . Arithmetic problems without a natural language context were tackled by Neural GPUs ( Kaiser & Sutskever , 2016 ) and its improved successors ( Freivalds & Liepins , 2017 ) . Supervised learning was also applied to various instances of propositional satisfiability in NeuroSAT ( Selsam et al. , 2018 ) and NeuroCore ( Selsam & Bjørner , 2019 ) as well as in ( Chvalovsky , 2019 ; Evans et al. , 2018 ) . Datasets introduced in Section 5 are OpenAI-gym ( Brockman et al. , 2016 ) compliant and can be tested with modern RL algorithms . Previous work on theorem proving and RL includes TacticToe , HOList and rlCoP ( Kaliszyk et al. , 2018 ) . TacticToe and rlCoP use guided Monte Carlo Tree Search ( MCTS ) and HOList proposes a custom RL algorithm . Machine learning systems for guidance of theorem provers . Our current work focuses on providing guidance for the fCoP ( Kaliszyk et al. , 2015b ) theorem prover . fCoP is an OCaml implementation of the very compact lean connection tableau prover ( Otten & Bibel , 2003 ) . fCoP was used as the proof engine in the guided provers FEMaLeCoP ( Kaliszyk & Urban , 2015b ) and rlCoP ( Kaliszyk et al. , 2018 ) . FEMaLeCoP learns only from positive data using two simple , but fast machine learning models ( custom nearest neighbour and naive Bayes ) . In rlCoP , the value and policy functions of the guided MCTS algorithm are learned similar to ( Anthony et al. , 2017 ; Silver et al. , 2017 ) , using gradient boosted trees as implemented in the XGBoost ( Chen & Guestrin , 2016 ) library . In contrast , we use neural network models instead of trees and the Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) algorithm instead of MCTS . In a longer run we believe that these methods should be combined , see ( François-Lavet et al. , 2018 , Section 6.2 ) , but in this work we 1This distribution does not include the fCoP theorem prover , which can not yet be publicly released , however a binary can be obtained upon request . propose to investigate how much can be achieved directly via rollouts and without a search algorithm like MCTS . A distinctive feature of our approach is the ability to perform very long rollouts both in training and evaluation . We demonstrate this in Section 6 , see Figure 4 . Chvalovský et al . ( 2019 ) ; Jakubuv & Urban ( 2017 ; 2019 ) ; Loos et al . ( 2017 ) added learning-based guidance to E prover ( Schulz , 2013 ) . These are supervised experiments which learn from saturation-style proof traces . Meta-learning suites of RL environments . Meta-learning algorithms in the context of RL can be tested using a suite of simulated robotic tasks ( Finn et al. , 2017 ; Rakelly et al. , 2019 ) , one of discrete environments proposed in ( Cobbe et al. , 2018 ; Juliani et al. , 2019 ; Menashe & Stone , 2018 ; Nichol et al. , 2018 ; Roderick et al. , 2017 ) or a new MineRL ( Guss et al. , 2019 ) suite of problems with mixed continuous and discrete actions . Our suite of tasks involves discrete actions . Curriculum learning and reinforcement learning . Our algorithm FLoP is an adaptation of the curriculum learning algorithms presented in ( Resnick et al. , 2018 ; Salimans & Chen , 2018 ) in a context of a suite of reinforcement learning environments presented in Section 5 . 3 FCOP AND THE CONNECTION TABLEAU CALCULUS In this section , we give a brief overview of the connection tableau method , as implemented by the fCoP system . We assume basic first-order logic and theorem proving terminology ( Robinson & Voronkov , 2001 ) . The input is a ( mathematical ) problem consisting of axioms and conjectures formally stated in first-order logic ( FOL ) . The calculus searches for refutational proofs , i.e . proofs showing that the axioms together with the negated conjectures are unsatisfiable . The FOL formulas are first translated to clause normal form ( CNF ) , producing a set of first-order clauses consisting of literals ( atoms or their negations ) . Figure 1 shows a closed connection tableau , i.e. , a finished proof tree where every branch contains complementary literals ( literals with opposite polarity ) . Since all branches contain a pair of contradictory literals , this shows that the set of clauses is unsatisfiable . Proof search starts with a start clause as a goal and proceeds by building a connection tableau by repeatedly applying extension steps and reduction steps . The extension step connects ( unifies ) the current goal ( a selected tip of a tableau branch ) with a complementary literal of a new clause . This extends the current branch , possibly splitting it into several branches if there are more literals in the new clause , and possibly instantiating some variables in the tableau . The reduction step connects the current goal to a complementary literal of the active path , thus closing the current branch . The proof is finished when all branches are closed . The extension and reduction steps are nondeterministic , requiring backtracking in the standard connection calculus . Brute force search such as iterative deepening can be used to ensure completeness , i.e. , making sure that the proof search finds a proof if there is any . fCoP represents theorem proving as a one-person game . The game ends with a success if a proof is found . The prover has many choices to make along the way , hence it typically has to explore a search space that is exponentially large in the length of the proof . In fCoP , the action space is roughly correlated with the size of the axiom set . While this can be large for large problems , typically only a few actions are available in any particular state . 4 FLOP – THE MAIN ALGORITHM The FLoP algorithm combines the connection tableau calculus with guidance based on PPO and curriculum learning Resnick et al . ( 2018 ) ; Salimans & Chen ( 2018 ) . Actions in our theorem proving game consist of selecting an extension step as defined in Section 3 ( reduction steps are performed automatically by the game engine ) . Figures 2 and 3 show how actions interact with other components of FLoP . Each extension step involves selecting one of the clauses , however , not all clauses are applicable as actions at a given proof step , due to the unification condition . The full information about the game state consists of all previous proof steps , the partial proof tree ( proof state ) and the current goal . The state and actions ( formulas ) are represented using previously developed features Kaliszyk & Urban ( 2015b ) ; Kaliszyk et al . ( 2015a ; 2018 ) . The features mainly include ( suitably hashed ) triples of adjacent nodes in the formula trees and in the partial proof trees . This means that the proof states and the actions are presented as ( sparse ) fixed length vectors , see the inputs to the policy and value networks in Figure 2 . These features have proved useful but are not free from problems . See the discussion in Appendix A. Curriculum Learning on Proofs . In theorem proving we are dealing with sparse rewards and we tackle this with the help of curriculum learning as implemented in Algorithm 1 . First , in line 6 of Algorithm 1 we sample a problem . In lines 7-20 we play an episode : if we have a proof then we start from the state dictated by the global curriculum ( lines 7-9 ) . If we do not have a proof then we start from the beginning . If the policy succeeds in finding a proof of a yet unproven problem then we reset the global curriculum to 1 in line 20 . We sample k episodes repeating k times the loop in lines 6-20 and finally decide whether to increase the global curriculum in lines 23-24 . We can advance curriculum globally ( as in Algorithm 1 ) or independently for each problem . We found that global advancement makes learning more stable , so that is our default approach . We can start learning with or without training proofs . It does not change the processing of Algorithm 1 . In Section 6 we provide experimental evidence with regard to both approaches .

Summary: This work introduces a method for learning to prove theorems which can leverage prior proving experience in order to discover very long proofs. At its core it works by inputting a corpus of training problems (which can also be annotated with solutions i.e. proofs), training a policy to solve these training problems by curriculum learning. The curriculum works by first supervising on the trace of an entire solution, and then once the system can solve a particular problem, decreasing the amount of the trace that it supervises on. The authors claim that this is a kind of analogical reasoning, because the system's policy is implicitly learning to represent the state/action space on the basis of prior experience.