IsarStep: a Benchmark for High-level Mathematical Reasoning

A well-defined benchmark is essential for measuring and accelerating research progress of machine learning models . In this paper , we present a benchmark for high-level mathematical reasoning and study the reasoning capabilities of neural sequence-to-sequence models . We build a non-synthetic dataset from the largest repository of proofs written by human experts in a theorem prover . The dataset has a broad coverage of undergraduate and research-level mathematical and computer science theorems . In our defined task , a model is required to fill in a missing intermediate proposition given surrounding proofs . This task provides a starting point for the long-term goal of having machines generate human-readable proofs automatically . Our experiments and analysis reveal that while the task is challenging , neural models can capture non-trivial mathematical reasoning . We further design a hierarchical transformer that outperforms the transformer baseline . The dataset and models are available from : https : //github.com/ Wenda302/IsarStep . 1 INTRODUCTION . Neural networks have achieved outstanding performance on a wide range of problems in natural language processing , computer vision , and speech recognition . However , research investigating their capacity of doing mathematical reasoning is still limited , with earlier attempts focusing on simple arithmetic tasks like integer addition and multiplication ( Zaremba & Sutskever , 2014 ; Kaiser & Sutskever , 2016 ; Trask et al. , 2018 ) . More recently , there has been work on solving school-level mathematical problems ( Saxton et al. , 2019 ) , logical reasoning ( Evans et al. , 2018 ) , and problems of function integration , ordinary differential equations ( Lample & Charton , 2020 ) , and properties of differential systems ( Charton et al. , 2020 ) . While these are valuable contributions to the machine learning community , they focused on generating answers to questions from a specific domain and were carried out on synthetic datasets with small vocabulary ( e.g . up to 100 unique tokens ) . In this paper , we consider general undergraduate and research-level mathematical proofs as a target for neural networks . When humans prove a theorem , a crucial step is to propose an intermediate proposition to bridge the gap between the goal and the currently known facts . This step requires complicated reasoning capabilities such as creative thinking , inference , understanding existing conditions , and symbolic manipulation of rules . For example , consider the following proof of the irrationality of √ 2 : Proof of irrationality of √ 2 . Assume √ 2 is rational . Then there exists a pair of coprime integers a and b such that √ 2 = a/b , and it follows that 2 = a2/b2 and then 2b2 = a2 . Hence a is even . Thus there exists an integer c such that a = 2c , which combined with 2b2 = a2 yields 2c2 = b2 : hence b is also even . So a and b are both even although they are coprime , contradiction . To derive ∃c ∈ Z. a = 2c from 2b2 = a2 , the intermediate proposition “ a is even ” would reduce the gap and lead to a successful proof . We would like to simulate the way humans prove theorems by proposing an intermediate proposition synthesis task — IsarStep . Instead of having primitive steps like 3 + 5 = 8 , the proof steps in IsarStep are at a higher-level , with much bigger steps as basic . Therefore it usually can not be simply solved by pattern matching and rewriting . To succeed in this task , a model is required to learn the meaning of important mathematical concepts ( e.g . determinant in linear algebra , residue in complex analysis ) , how they are related to each other through theorems , and how they are utilised in proof derivations . Solving the IsaStep task will be potentially helpful for improving the automation of theorem provers , because proposing a valid intermediate proposition will help reduce their search space significantly . It is also a first step towards the long-term goal of sketching complete human-readable proofs automatically . We have built the IsarStep dataset by mining arguably the largest publicly-hosted repository of mechanised proofs : the Achieve of Formal Proofs ( AFP ) .1 The AFP is checked by the Isabelle proof assistant ( Paulson , 1994 ) and contains 143K lemmas . Combining the AFP with the standard library of Isabelle/HOL yields a dataset of 204K formally-proved lemmas . The dataset covers a broad spectrum of subjects , including foundational logic ( e.g . Gödel ’ s incompleteness theorems ) , advanced analysis ( e.g . the Prime Number Theorem ) , computer algebra , cryptographic frameworks , and various data structures . A nice property of the mined formal proofs is that they are mostly declarative proofs , a proof style very close to human prose proofs.2 Fig . 1 illustrates the proof of irrationality of √ 2 in Isabelle . We can see that the proof is actually legible ( even to people who are not familiar with the system ) and and it captures high-level structures like those in human proofs . We further explore the reasoning capabilities of neural models . We frame the proposed task as a sequence-to-sequence ( seq2seq ) prediction problem . Beyond evaluating the existing neural seq2seq model baselines—the seq2seq with attention ( Bahdanau et al. , 2015 ) , the transformer ( Vaswani et al. , 2017 ) —we also propose a new architecture , the hierarchical transformer ( §4 ) . The architecture is motivated by the way humans reason about propositions ; it consists of a set of local transformer layers , modelling the representation of each proposition , and a set of global layers , modelling the correlation across propositions . Experiments ( §5 ) show that these neural models can solve 15–25 % of problems on the test set , and the hierarchical transformer achieves the best result . Further analysis ( §6 ) on the output of these models shows that while the proposition synthesis task is hard , the neural models can indeed capture mathematical reasoning . We find that the embeddings of closely related mathematical concepts are close in cosine space ; models can reason about the relation between set , subset , and member , and perform more complex multi-step reasoning that is even hard for humans . Our contributions are summarised as follows : 1 . We mine a large non-synthetic dataset of formal proofs and propose a task for evaluating neural models ’ mathematical reasoning abilities . The dataset contains 820K training examples with a vocabulary size of 30K . 1https : //www.isa-afp.org 2A comparison of proofs in different systems is available in Wiedijk ( 2006 ) . The declarative style proof is also available in Mizar ( Grabowski et al. , 2010 ) , where the style originates . 2 . We evaluate existing neural seq2seq models on this task . 3 . We introduce a hierarchical transformer model , which outperforms the baseline models . 4 . We provide a comprehensive analysis of what has been learned by the neural models . 5 . We provide a test suite to check the correctness of types and the validity of the generated propositions using automatic theorem provers . 2 THE ISARSTEP TASK . In this section , we define the task of intermediate proposition generation more concretely . We again take the proof of irrationality of √ 2 as an example . We will have the following derivation : 2b2 = a2︸ ︷︷ ︸ ( 1 ) ⇒ a is even︸ ︷︷ ︸ ( 2 ) ⇒ ∃c ∈ Z. a = 2c︸ ︷︷ ︸ ( 3 ) . In our proposed task , we would like to generate ( 2 ) given ( 1 ) and ( 3 ) . When humans prove a theorem , they implicitly assume certain background knowledge , as lemmas . For example , in this case we assume that we can trivially prove ( 1 ) ⇒ ( 2 ) based on the fact that the product of two numbers are even iff at least one of them is even . In Isabelle ( Paulson , 1994 ) , these relevant lemmas ( e.g . even_mult_iff : even ( ? a * ? b ) = ( even ? a ∨ even ? b ) corresponding to line 10 in Fig . 1 ) can be found automatically by its built-in automation Sledgehammer ( Blanchette et al. , 2011 ) . In our task , we optionally provide these lemmas as extra information in addition to ( 1 ) and ( 3 ) . The derivation of ( 2 ) ⇒ ( 3 ) in the proof above is a simple step , because only ( 2 ) is needed to arrive at ( 3 ) . In most cases , multiple propositions have to be used together in order to infer a proposition , for example P1 , P2 , P3 ⇒ P4 . For these more general cases , we also include the additional propositions ( e.g . P2 and P1 ) as part of the source propositions . To summarize , each example in the IsarStep dataset is formed by five parts : F.1 a target proposition ( e.g . a is even ) , F.2 a set of used local propositions to derive F.1 ( e.g . 2b2 = a2 ) , F.3 a local proposition derived from the target proposition F.1 ( ∃c ∈ Z. a = 2c ) , F.4 other local propositions and ( library ) lemmas used to justify F.3 , F.5 a set of used ( library ) lemmas to justify F.1 ( e.g . even_mult_iff : even ( ? a * ? b ) = ( even ? a ∨ even ? b ) ) . We want to synthesise F.1 given F.2 – F.4 with F.5 optional : the named lemmas in F.5 are common knowledge and can be used as additional hints . The propositions are generated as a sequence of tokens and therefore the search space is Σ∗ : search over 30K actions ( §3.3 , vocabulary size for seq2seq models ) at every timestep without a predefined maximum output length . IsarStep can be considered as single step reasoning , which can be repeated to sketch more complex proofs . Good performance on this task is a crucial step for designing models that can automatically prove theorems with minimal human assistance . 3 DATASET PREPROCESSSING AND STATISTICS . The mined raw dataset has long propositions and a large number of unique tokens . To alleviate the performance deterioration of machine learning models due to the aforementioned problems , we propose tricks to preprocess the raw dataset , including free variable normalisation and removing unnecessary parentheses . These tricks substantially reduce the sequence lengths and vocabulary size . 3.1 THE LOGIC AND TOKENS . The core logic of Isabelle/HOL is simply-typed λ-calculus with de Bruijn indices for bound variables ( Wenzel , 2020 , Chapter 2.2 ) . A local proposition or a ( library ) lemma/theorem is essentially a term in the calculus . As types can be inferred automatically , we drop types in terms ( to reduce the size of the vocabulary ) and encode a term as a sequence of tokens that include lambda term constructors : CONST , FREE , VAR , BOUND , ABS ( function abstraction ) , and $ ( function application ) . Additionally , parentheses have been used in the sequence to represent the tree structure . To give an example , we encode the proposition even a as the following sequence of tokens separated by a white space : CONST HOL.Trueprop $ ( CONST Parity.semiring_parity_class.even $ FREE < X0 > ) where CONST HOL.Trueprop is a boilerplate function that converts from type bool to prop ; CONST Parity.semiring_parity_class.even is the even predicate ; FREE < X0 > encodes the Skolem constant a in even a . Since a is a user-introduced local constant that can be arbitrary , we normalised it to the algorithmically generated name < X0 > in order to reduce the vocabulary size ( see §3.2 ) . Overall , every local proposition and library lemma/theorem is encoded as a sequence of tokens , and can be mostly decoded to the original term with type inference .

Summary: This paper presents a non-synthetic dataset generated from the Isabelle AFP, the largest mechanised proof repository for the task of filling in a missing intermediate proposition given surrounding proofs. Together with the dataset the paper presents a hierarchical transformer model (HAT). Top-10 accuracy, which is the percentage of target proposition appearing in the top 10 generated propositions, is 37.2 for their HAT model. The task this paper addresses is important for theorem proving practitioners, and synthesising propositions appears to be a more difficult challenge compared to other ML tasks in theorem proving (choosing useful lemmas or choosing tactics).