Towards Finding Longer Proofs

We present a reinforcement learning ( RL ) based guidance system for automated theorem proving geared towards Finding Longer Proofs ( FLoP ) . FLoP focuses on generalizing from short proofs to longer ones of similar structure . To achieve that , FLoP uses state-of-the-art RL approaches that were previously not applied in theorem proving . In particular , we show that curriculum learning significantly outperforms previous learning-based proof guidance on a synthetic dataset of increasingly difficult arithmetic problems . 1 INTRODUCTION . In 1958 B. F. Skinner , a pioneer of modern behaviorism , in the article “ Teaching Machines ” ( Skinner , 1958 ) noticed that “ in acquiring complex behavior the student must pass through a carefully designed sequence of steps , often of considerable length . Each step must be so small that it can always be taken , yet in taking it the student moves somewhat closer to fully competent behavior ” . His study extended also to the teaching of arithmetic : “ The student is expected to arrive at 9 · 7 = 63 , not by memorizing it as he would memorize a line of poetry , but by putting into practice such principles as that nine times a number is the same as ten times the number minus the number . . . ” . The idea of learning using a curriculum of problems is also widely used in machine learning ( Bengio et al. , 2009 ; Elman , 1993 ; Resnick et al. , 2018 ; Salimans & Chen , 2018 ) and in this work we apply curriculum learning to automatic theorem proving focusing on arithmetic . Our work has the following contributions . ( 1 ) We introduce a new theorem proving algorithm FLoP ( Section 4 ) based on reinforcement learning and the connection tableau calculus . FLoP uses a meta-learning variation of the curriculum learning algorithms presented by Resnick et al . ( 2018 ) and Salimans & Chen ( 2018 ) . ( 2 ) We introduce a synthetic dataset of increasingly difficult arithmetic problems organized as RL environments ( Section 5 ) . ( 3 ) We use this benchmark to compare ( Section 6 ) the performance of our system with state-of-the-art saturation provers Vampire ( Kovács & Voronkov , 2013 ) and E ( Schulz , 2013 ) guided by human-designed strategies , and with rlCoP ( Kaliszyk et al. , 2018 ) – a recently developed RL-based connection tableau prover . FLoP significantly outperforms the other provers on harder problems , demonstrating its ability to find longer proofs . Our datasets presented in Section 5 seem to be particularly suited for machine learning methods : problems are simple , solutions are long , repetitive and rather predictable for humans . Still , state-ofthe-art systems struggle with solving some of the problems – see Section 6 for details . Other works using machine learning to guide a prover ( Chvalovský et al. , 2019 ; Jakubuv & Urban , 2017 ; Kaliszyk & Urban , 2015b ; Kaliszyk et al. , 2018 ; Loos et al. , 2017 ; Urban et al. , 2011 ) usually deal with large mathematical corpora , while we focus on a fragment of Robinson Arithmetic , which is a limited and simple theory . Our reasons behind this narrower focus : ( a ) We wanted to create a scalable RL benchmark with emphasis on the length of proofs . ( b ) A symbolic method based on human-designed sets of hints ( Veroff , 1996 ) was previously successfully applied in abstract algebra by Kinyon et al . ( 2013 ) to discover long proofs and we wanted to check whether learning of long proofs is feasible using the state-of-the-art ML toolset . ( c ) We wanted interpretable failure modes . In the case of large mathematical corpora , the interpretation of failures may be a hard task because of multiple failures and the complicated structure of the corpora , requiring specialized domain knowledge both in mathematics and with regard to the inner workings of the proof system . Our code , datasets and all experiment configuration files are available at http : //bit.ly/code_ atpcurr1 . Supplementary materials including screencasts with gameplays performed in our environments are available at the project webpage http : //bit.ly/site_atpcurr . 2 RELATED WORK . Machine learning datasets and RL environments involving mathematics and logic . The arithmetic dataset which we introduce in Section 5 is geared towards longer proofs and is structurally much simpler than other theorem proving datasets which we list below . One can think about this suite of RL problems as gridworlds of theorem proving ( see ( Sutton & Barto , 2018 , Example 3.5 ) for a broader explanation of importance of gridworlds in RL ) . Our dataset is intended to become a general purpose testing ground for theorem proving and reinforcement learning methods , in particular for meta-learning and hierarchical learning algorithms . TPTP ( Sutcliffe , 2017 ) consists of 22507 problems in 53 domains collected over several decades . A large dataset for developing machine learning for theorem proving based on the Mizar Mathematical Library ( MML ) ( Grabowski et al. , 2010 ) was introduced by Urban ( 2006a ) in the MPTP project . The dataset was used e.g . by Alemi et al . ( 2016 ) ; Kaliszyk & Urban ( 2015a ) ; Urban ( 2007 ) ; Urban et al . ( 2008 ) . Similar datasets based on the Isabelle/HOL , HOL Light/Flyspeck and HOL4/CakeML systems and projects ( Blanchette et al. , 2016 ; Gauthier & Kaliszyk , 2015 ; Kaliszyk & Urban , 2014 ) were introduced in the last decade and used for the CASC LTB ( large theory ) ATP competition ( Sutcliffe & Urban , 2016 ) and other system evaluations . Such datasets cover large areas of mathematics and computer science and contain diverse axioms , lemmas , theorems , definitions , and symbols . Smaller subsets of lemmas leading to the Bolzano-Weierstrass theorem were selected from the MPTP dataset to form the MPTP Challenge ( Urban , 2006b ) and the MPTP2078 benchmark ( Alama et al. , 2014 ) . HOLStep ( Kaliszyk et al. , 2017 ) introduced a dataset based on 11400 proofs , including a proof of the Kepler Conjecture ( Hales et al. , 2015 ) , formalized using HOL Light ( Harrison , 1996 ) . In HOLStep and in FormulaNet ( Wang et al. , 2017 ) the dataset was used as a benchmark for various neural architectures . The recent HOList project ( Bansal et al. , 2019a ; b ; Paliwal et al. , 2019 ) uses 29462 theorems formalized in HOL Light and instruments them for experiments oriented towards tactic selection , where a tactic is a human-designed program which aggregates multiple proof steps . GamePad ( Huang et al. , 2019 ) introduced a dataset based on a formalization of the Feit-Thompson Theorem ( Gonthier et al. , 2013 ) along with a generated algebra problems . It is intended for learning tactic selection together with an auxiliary task of predicting the number of proof steps left . A dataset based on theorems proved in HOL4 ( Slind & Norrish , 2008 ) was used for developing the TacticToe ( Gauthier et al. , 2018 ) learning-guided tactical prover . Saxton et al . ( 2019 ) proposed a dataset of simple algebraic problems expressed in English . Arithmetic problems without a natural language context were tackled by Neural GPUs ( Kaiser & Sutskever , 2016 ) and its improved successors ( Freivalds & Liepins , 2017 ) . Supervised learning was also applied to various instances of propositional satisfiability in NeuroSAT ( Selsam et al. , 2018 ) and NeuroCore ( Selsam & Bjørner , 2019 ) as well as in ( Chvalovsky , 2019 ; Evans et al. , 2018 ) . Datasets introduced in Section 5 are OpenAI-gym ( Brockman et al. , 2016 ) compliant and can be tested with modern RL algorithms . Previous work on theorem proving and RL includes TacticToe , HOList and rlCoP ( Kaliszyk et al. , 2018 ) . TacticToe and rlCoP use guided Monte Carlo Tree Search ( MCTS ) and HOList proposes a custom RL algorithm . Machine learning systems for guidance of theorem provers . Our current work focuses on providing guidance for the fCoP ( Kaliszyk et al. , 2015b ) theorem prover . fCoP is an OCaml implementation of the very compact lean connection tableau prover ( Otten & Bibel , 2003 ) . fCoP was used as the proof engine in the guided provers FEMaLeCoP ( Kaliszyk & Urban , 2015b ) and rlCoP ( Kaliszyk et al. , 2018 ) . FEMaLeCoP learns only from positive data using two simple , but fast machine learning models ( custom nearest neighbour and naive Bayes ) . In rlCoP , the value and policy functions of the guided MCTS algorithm are learned similar to ( Anthony et al. , 2017 ; Silver et al. , 2017 ) , using gradient boosted trees as implemented in the XGBoost ( Chen & Guestrin , 2016 ) library . In contrast , we use neural network models instead of trees and the Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) algorithm instead of MCTS . In a longer run we believe that these methods should be combined , see ( François-Lavet et al. , 2018 , Section 6.2 ) , but in this work we 1This distribution does not include the fCoP theorem prover , which can not yet be publicly released , however a binary can be obtained upon request . propose to investigate how much can be achieved directly via rollouts and without a search algorithm like MCTS . A distinctive feature of our approach is the ability to perform very long rollouts both in training and evaluation . We demonstrate this in Section 6 , see Figure 4 . Chvalovský et al . ( 2019 ) ; Jakubuv & Urban ( 2017 ; 2019 ) ; Loos et al . ( 2017 ) added learning-based guidance to E prover ( Schulz , 2013 ) . These are supervised experiments which learn from saturation-style proof traces . Meta-learning suites of RL environments . Meta-learning algorithms in the context of RL can be tested using a suite of simulated robotic tasks ( Finn et al. , 2017 ; Rakelly et al. , 2019 ) , one of discrete environments proposed in ( Cobbe et al. , 2018 ; Juliani et al. , 2019 ; Menashe & Stone , 2018 ; Nichol et al. , 2018 ; Roderick et al. , 2017 ) or a new MineRL ( Guss et al. , 2019 ) suite of problems with mixed continuous and discrete actions . Our suite of tasks involves discrete actions . Curriculum learning and reinforcement learning . Our algorithm FLoP is an adaptation of the curriculum learning algorithms presented in ( Resnick et al. , 2018 ; Salimans & Chen , 2018 ) in a context of a suite of reinforcement learning environments presented in Section 5 . 3 FCOP AND THE CONNECTION TABLEAU CALCULUS In this section , we give a brief overview of the connection tableau method , as implemented by the fCoP system . We assume basic first-order logic and theorem proving terminology ( Robinson & Voronkov , 2001 ) . The input is a ( mathematical ) problem consisting of axioms and conjectures formally stated in first-order logic ( FOL ) . The calculus searches for refutational proofs , i.e . proofs showing that the axioms together with the negated conjectures are unsatisfiable . The FOL formulas are first translated to clause normal form ( CNF ) , producing a set of first-order clauses consisting of literals ( atoms or their negations ) . Figure 1 shows a closed connection tableau , i.e. , a finished proof tree where every branch contains complementary literals ( literals with opposite polarity ) . Since all branches contain a pair of contradictory literals , this shows that the set of clauses is unsatisfiable . Proof search starts with a start clause as a goal and proceeds by building a connection tableau by repeatedly applying extension steps and reduction steps . The extension step connects ( unifies ) the current goal ( a selected tip of a tableau branch ) with a complementary literal of a new clause . This extends the current branch , possibly splitting it into several branches if there are more literals in the new clause , and possibly instantiating some variables in the tableau . The reduction step connects the current goal to a complementary literal of the active path , thus closing the current branch . The proof is finished when all branches are closed . The extension and reduction steps are nondeterministic , requiring backtracking in the standard connection calculus . Brute force search such as iterative deepening can be used to ensure completeness , i.e. , making sure that the proof search finds a proof if there is any . fCoP represents theorem proving as a one-person game . The game ends with a success if a proof is found . The prover has many choices to make along the way , hence it typically has to explore a search space that is exponentially large in the length of the proof . In fCoP , the action space is roughly correlated with the size of the axiom set . While this can be large for large problems , typically only a few actions are available in any particular state . 4 FLOP – THE MAIN ALGORITHM The FLoP algorithm combines the connection tableau calculus with guidance based on PPO and curriculum learning Resnick et al . ( 2018 ) ; Salimans & Chen ( 2018 ) . Actions in our theorem proving game consist of selecting an extension step as defined in Section 3 ( reduction steps are performed automatically by the game engine ) . Figures 2 and 3 show how actions interact with other components of FLoP . Each extension step involves selecting one of the clauses , however , not all clauses are applicable as actions at a given proof step , due to the unification condition . The full information about the game state consists of all previous proof steps , the partial proof tree ( proof state ) and the current goal . The state and actions ( formulas ) are represented using previously developed features Kaliszyk & Urban ( 2015b ) ; Kaliszyk et al . ( 2015a ; 2018 ) . The features mainly include ( suitably hashed ) triples of adjacent nodes in the formula trees and in the partial proof trees . This means that the proof states and the actions are presented as ( sparse ) fixed length vectors , see the inputs to the policy and value networks in Figure 2 . These features have proved useful but are not free from problems . See the discussion in Appendix A. Curriculum Learning on Proofs . In theorem proving we are dealing with sparse rewards and we tackle this with the help of curriculum learning as implemented in Algorithm 1 . First , in line 6 of Algorithm 1 we sample a problem . In lines 7-20 we play an episode : if we have a proof then we start from the state dictated by the global curriculum ( lines 7-9 ) . If we do not have a proof then we start from the beginning . If the policy succeeds in finding a proof of a yet unproven problem then we reset the global curriculum to 1 in line 20 . We sample k episodes repeating k times the loop in lines 6-20 and finally decide whether to increase the global curriculum in lines 23-24 . We can advance curriculum globally ( as in Algorithm 1 ) or independently for each problem . We found that global advancement makes learning more stable , so that is our default approach . We can start learning with or without training proofs . It does not change the processing of Algorithm 1 . In Section 6 we provide experimental evidence with regard to both approaches .

Summary: In the paper, the authors present a new algorithm for training neural networks used in an automated theorem prover using theorems with or without proofs as training data. The algorithm casts this training task as a reinforcement learning problem, and employs curriculum learning and the Proximal Policy Optimization algorithm to find appropriate neural network parameters, in particular, those that make the prover good at finding long proofs. The authors also propose a new dataset for theorems and proofs for a simple equational theory of arithmetic, which is again suitable for improving (via learning) and testing the ability of the prover for finding long proofs. The proposed prover is tested against existing theorem provers, and for the authors' dataset, it outperforms those provers.