IsarStep: a Benchmark for High-level Mathematical Reasoning

A well-defined benchmark is essential for measuring and accelerating research progress of machine learning models . In this paper , we present a benchmark for high-level mathematical reasoning and study the reasoning capabilities of neural sequence-to-sequence models . We build a non-synthetic dataset from the largest repository of proofs written by human experts in a theorem prover . The dataset has a broad coverage of undergraduate and research-level mathematical and computer science theorems . In our defined task , a model is required to fill in a missing intermediate proposition given surrounding proofs . This task provides a starting point for the long-term goal of having machines generate human-readable proofs automatically . Our experiments and analysis reveal that while the task is challenging , neural models can capture non-trivial mathematical reasoning . We further design a hierarchical transformer that outperforms the transformer baseline . The dataset and models are available from : https : //github.com/ Wenda302/IsarStep . 1 INTRODUCTION . Neural networks have achieved outstanding performance on a wide range of problems in natural language processing , computer vision , and speech recognition . However , research investigating their capacity of doing mathematical reasoning is still limited , with earlier attempts focusing on simple arithmetic tasks like integer addition and multiplication ( Zaremba & Sutskever , 2014 ; Kaiser & Sutskever , 2016 ; Trask et al. , 2018 ) . More recently , there has been work on solving school-level mathematical problems ( Saxton et al. , 2019 ) , logical reasoning ( Evans et al. , 2018 ) , and problems of function integration , ordinary differential equations ( Lample & Charton , 2020 ) , and properties of differential systems ( Charton et al. , 2020 ) . While these are valuable contributions to the machine learning community , they focused on generating answers to questions from a specific domain and were carried out on synthetic datasets with small vocabulary ( e.g . up to 100 unique tokens ) . In this paper , we consider general undergraduate and research-level mathematical proofs as a target for neural networks . When humans prove a theorem , a crucial step is to propose an intermediate proposition to bridge the gap between the goal and the currently known facts . This step requires complicated reasoning capabilities such as creative thinking , inference , understanding existing conditions , and symbolic manipulation of rules . For example , consider the following proof of the irrationality of √ 2 : Proof of irrationality of √ 2 . Assume √ 2 is rational . Then there exists a pair of coprime integers a and b such that √ 2 = a/b , and it follows that 2 = a2/b2 and then 2b2 = a2 . Hence a is even . Thus there exists an integer c such that a = 2c , which combined with 2b2 = a2 yields 2c2 = b2 : hence b is also even . So a and b are both even although they are coprime , contradiction . To derive ∃c ∈ Z. a = 2c from 2b2 = a2 , the intermediate proposition “ a is even ” would reduce the gap and lead to a successful proof . We would like to simulate the way humans prove theorems by proposing an intermediate proposition synthesis task — IsarStep . Instead of having primitive steps like 3 + 5 = 8 , the proof steps in IsarStep are at a higher-level , with much bigger steps as basic . Therefore it usually can not be simply solved by pattern matching and rewriting . To succeed in this task , a model is required to learn the meaning of important mathematical concepts ( e.g . determinant in linear algebra , residue in complex analysis ) , how they are related to each other through theorems , and how they are utilised in proof derivations . Solving the IsaStep task will be potentially helpful for improving the automation of theorem provers , because proposing a valid intermediate proposition will help reduce their search space significantly . It is also a first step towards the long-term goal of sketching complete human-readable proofs automatically . We have built the IsarStep dataset by mining arguably the largest publicly-hosted repository of mechanised proofs : the Achieve of Formal Proofs ( AFP ) .1 The AFP is checked by the Isabelle proof assistant ( Paulson , 1994 ) and contains 143K lemmas . Combining the AFP with the standard library of Isabelle/HOL yields a dataset of 204K formally-proved lemmas . The dataset covers a broad spectrum of subjects , including foundational logic ( e.g . Gödel ’ s incompleteness theorems ) , advanced analysis ( e.g . the Prime Number Theorem ) , computer algebra , cryptographic frameworks , and various data structures . A nice property of the mined formal proofs is that they are mostly declarative proofs , a proof style very close to human prose proofs.2 Fig . 1 illustrates the proof of irrationality of √ 2 in Isabelle . We can see that the proof is actually legible ( even to people who are not familiar with the system ) and and it captures high-level structures like those in human proofs . We further explore the reasoning capabilities of neural models . We frame the proposed task as a sequence-to-sequence ( seq2seq ) prediction problem . Beyond evaluating the existing neural seq2seq model baselines—the seq2seq with attention ( Bahdanau et al. , 2015 ) , the transformer ( Vaswani et al. , 2017 ) —we also propose a new architecture , the hierarchical transformer ( §4 ) . The architecture is motivated by the way humans reason about propositions ; it consists of a set of local transformer layers , modelling the representation of each proposition , and a set of global layers , modelling the correlation across propositions . Experiments ( §5 ) show that these neural models can solve 15–25 % of problems on the test set , and the hierarchical transformer achieves the best result . Further analysis ( §6 ) on the output of these models shows that while the proposition synthesis task is hard , the neural models can indeed capture mathematical reasoning . We find that the embeddings of closely related mathematical concepts are close in cosine space ; models can reason about the relation between set , subset , and member , and perform more complex multi-step reasoning that is even hard for humans . Our contributions are summarised as follows : 1 . We mine a large non-synthetic dataset of formal proofs and propose a task for evaluating neural models ’ mathematical reasoning abilities . The dataset contains 820K training examples with a vocabulary size of 30K . 1https : //www.isa-afp.org 2A comparison of proofs in different systems is available in Wiedijk ( 2006 ) . The declarative style proof is also available in Mizar ( Grabowski et al. , 2010 ) , where the style originates . 2 . We evaluate existing neural seq2seq models on this task . 3 . We introduce a hierarchical transformer model , which outperforms the baseline models . 4 . We provide a comprehensive analysis of what has been learned by the neural models . 5 . We provide a test suite to check the correctness of types and the validity of the generated propositions using automatic theorem provers . 2 THE ISARSTEP TASK . In this section , we define the task of intermediate proposition generation more concretely . We again take the proof of irrationality of √ 2 as an example . We will have the following derivation : 2b2 = a2︸ ︷︷ ︸ ( 1 ) ⇒ a is even︸ ︷︷ ︸ ( 2 ) ⇒ ∃c ∈ Z. a = 2c︸ ︷︷ ︸ ( 3 ) . In our proposed task , we would like to generate ( 2 ) given ( 1 ) and ( 3 ) . When humans prove a theorem , they implicitly assume certain background knowledge , as lemmas . For example , in this case we assume that we can trivially prove ( 1 ) ⇒ ( 2 ) based on the fact that the product of two numbers are even iff at least one of them is even . In Isabelle ( Paulson , 1994 ) , these relevant lemmas ( e.g . even_mult_iff : even ( ? a * ? b ) = ( even ? a ∨ even ? b ) corresponding to line 10 in Fig . 1 ) can be found automatically by its built-in automation Sledgehammer ( Blanchette et al. , 2011 ) . In our task , we optionally provide these lemmas as extra information in addition to ( 1 ) and ( 3 ) . The derivation of ( 2 ) ⇒ ( 3 ) in the proof above is a simple step , because only ( 2 ) is needed to arrive at ( 3 ) . In most cases , multiple propositions have to be used together in order to infer a proposition , for example P1 , P2 , P3 ⇒ P4 . For these more general cases , we also include the additional propositions ( e.g . P2 and P1 ) as part of the source propositions . To summarize , each example in the IsarStep dataset is formed by five parts : F.1 a target proposition ( e.g . a is even ) , F.2 a set of used local propositions to derive F.1 ( e.g . 2b2 = a2 ) , F.3 a local proposition derived from the target proposition F.1 ( ∃c ∈ Z. a = 2c ) , F.4 other local propositions and ( library ) lemmas used to justify F.3 , F.5 a set of used ( library ) lemmas to justify F.1 ( e.g . even_mult_iff : even ( ? a * ? b ) = ( even ? a ∨ even ? b ) ) . We want to synthesise F.1 given F.2 – F.4 with F.5 optional : the named lemmas in F.5 are common knowledge and can be used as additional hints . The propositions are generated as a sequence of tokens and therefore the search space is Σ∗ : search over 30K actions ( §3.3 , vocabulary size for seq2seq models ) at every timestep without a predefined maximum output length . IsarStep can be considered as single step reasoning , which can be repeated to sketch more complex proofs . Good performance on this task is a crucial step for designing models that can automatically prove theorems with minimal human assistance . 3 DATASET PREPROCESSSING AND STATISTICS . The mined raw dataset has long propositions and a large number of unique tokens . To alleviate the performance deterioration of machine learning models due to the aforementioned problems , we propose tricks to preprocess the raw dataset , including free variable normalisation and removing unnecessary parentheses . These tricks substantially reduce the sequence lengths and vocabulary size . 3.1 THE LOGIC AND TOKENS . The core logic of Isabelle/HOL is simply-typed λ-calculus with de Bruijn indices for bound variables ( Wenzel , 2020 , Chapter 2.2 ) . A local proposition or a ( library ) lemma/theorem is essentially a term in the calculus . As types can be inferred automatically , we drop types in terms ( to reduce the size of the vocabulary ) and encode a term as a sequence of tokens that include lambda term constructors : CONST , FREE , VAR , BOUND , ABS ( function abstraction ) , and $ ( function application ) . Additionally , parentheses have been used in the sequence to represent the tree structure . To give an example , we encode the proposition even a as the following sequence of tokens separated by a white space : CONST HOL.Trueprop $ ( CONST Parity.semiring_parity_class.even $ FREE < X0 > ) where CONST HOL.Trueprop is a boilerplate function that converts from type bool to prop ; CONST Parity.semiring_parity_class.even is the even predicate ; FREE < X0 > encodes the Skolem constant a in even a . Since a is a user-introduced local constant that can be arbitrary , we normalised it to the algorithmically generated name < X0 > in order to reduce the vocabulary size ( see §3.2 ) . Overall , every local proposition and library lemma/theorem is encoded as a sequence of tokens , and can be mostly decoded to the original term with type inference .

Summary: This paper proposes a benchmark for high-level mathematical reasoning and study the reasoning capabilities of neural sequence-to-sequence models. This is a non-synthetic dataset from the largest repository of proofs written by human experts in a theorem prover, which has a broad coverage of undergraduate and research-level mathematical and computer science theorems. Based on this dataset, the model need to fill in a missing intermediate proposition given surrounding proofs, named as IsarStep. It's a very interesting task. This task provides a starting point for the long-term goal of having machines generate human-readable proofs automatically. The experiments and analysis also reveal that neural models can capture non-trivial mathematical reasoning.