Mathematical Reasoning via Self-supervised Skip-tree Training

We demonstrate that self-supervised language modeling applied to mathematical formulas enables logical reasoning . To measure the logical reasoning abilities of language models , we formulate several evaluation ( downstream ) tasks , such as inferring types , suggesting missing assumptions , and completing equalities . For training language models for formal mathematics , we propose a novel skip-tree task . We find that models trained on the skip-tree task show surprisingly strong mathematical reasoning abilities , and outperform models trained on standard skipsequence tasks . We also analyze the models ’ ability to formulate new conjectures by measuring how often the predictions are provable and useful in other proofs . 1 INTRODUCTION . Language modeling using Transformers ( Vaswani et al. , 2017 ) has been hugely successful for applications like translation and text generation . Models like GPT are able to generate news articles and stories given just an abstract ( Radford et al. , 2018 ) . These models are usually ( pre- ) trained on a proxy task , such as predicting missing words in the case of BERT ( Devlin et al. , 2019 ) , before fine tuning the models on more specific ( downstream ) tasks such as machine translation and questionanswering . These proxy tasks are not reliant on labels , and thus can be trained on large corpora of unlabeled data . Recently , however , we have seen successful demonstrations of language modeling using only self-supervised training without any fine tuning ( Brown et al. , 2020 ) . In this work , we extend this line of thought and demonstrate that purely self-supervised training can even lead to mathematical reasoning abilities . This represents a major departure from prior work in deep learning for mathematics , which has focused on learning directly on logical reasoning tasks , such as predicting the proof steps or premises or assignments . These approaches require labeled data , which is hard to come by and typically very limited in size . In contrast , our language modeling approach to mathematics allows us to train on unlabeled mathematical expressions . We start with the HOList dataset ( Bansal et al. , 2019 ) , which spans a wide range of mathematical topics , including topology , multivariate calculus , real and complex analysis , geometric algebra , and measure theory , formalized in the HOL Light proof assistant ( Harrison , 1996 ) . We find that training a language model on all mathematical expressions in this dataset leads to surprisingly strong mathematical reasoning capabilities . We believe that this opens the door to different kinds of neural theorem provers , which do not only search through a well-defined search space of tactics and premises , but which are capable to generating their own lemmas and could even come up with a new Ansatz requiring a creative substitution . For self-supervised training on mathematical expressions , we propose a novel skip-tree task , which is a specialization of the skip-sequence task that respects the tree structure of expressions . We show that models trained on the skip-tree task significantly outperform those trained on the skip-sequence task , which is the state of the art for sequence to sequence models for natural language . Reasoning can refer to a wide range of abilities , and thus we measure the mathematical reasoning abilities of language models on a variety of tasks , including mechanical derivations , such as type inference , and also creative tasks , such as predicting under which assumptions a statement is true . As we want to study what reasoning capabilities can be acquired just through self-supervised training , we do not employ fine-tuning on these tasks . Instead , we designed the tasks to be syntactically similar to the training task , such that the language model may produce correct answers . An advantage of formal language compared to natural language is that we can attempt to automatically evaluate statements . That is , we can let our language models produce conjectures , which we then try to prove using the DeepHOL theorem prover ( Bansal et al. , 2019 ; 2020 ) . Besides evaluating the provability of the produced statements , we go one step further and evaluate their usefulness , by measuring how many times they are used as premises in proofs of other theorems . Our contributions are as follows : 1 . We show that self-supervised training on mathematical formulas alone leads to logical reasoning capabilities . 2 . We introduce a new skip-tree training task that outperforms the state-of-the-art skip-sequence training . We also introduce several evaluation tasks that are subsumed by skip-tree training ( i.e . predict a missing subexpression ) , but test specific logical reasoning abilities to make the performance of the models interpretable . 3 . We suggest a way to create and evaluate mathematical conjectures using existing neural theorem provers . The remainder of this paper is structured as follows : First , we review related work on language modeling and deep learning for mathematics in Section 2 . Then , in Section 3 we discuss the source corpus of formal mathematical statements from which we generate our training data . In Section 4 , we present the skip-tree training task , as well as several variations that we used in our ablation studies . We present the evaluation tasks in Section 5 , discuss our experimental findings in Section 6 , and conclude in Section 7 . 2 RELATED WORK . Recently , we have seen a series of rapid improvements in language modeling stemming from better pretraining tasks ( Devlin et al. , 2019 ; Zhang et al. , 2019 ; Song et al. , 2019 ; Dong et al. , 2019 ; Raffel et al. , 2019 ; Conneau and Lample , 2019 ) . BERT ( Devlin et al. , 2019 ) is a pretraining task for Transformers ( Vaswani et al. , 2017 ) , which masks out a certain fraction of the input tokens that the model then has to predict . UniLM uses multiple pretraining tasks ( Dong et al. , 2019 ) . One of them is a sequence-to-sequence task ; to predict the next sentence from the previous sentence . MASS and SpanBERT consider a generalized sequence-to-sequence pretraining task , which is to predict a masked out subsequence of the input ( Song et al. , 2019 ; Joshi et al. , 2020 ) . However , both MASS and SpanBERT reveal the length of the sequence to predict as they replace it by a number of mask tokens equal to the length of the sequence . T5 introduced a generalization of sequence-to-sequence pretraining tasks that is crucial to our work ( Raffel et al. , 2019 ) . They replace the subsequence ( or multiple subsequences ) to be predicted by a single token ( not a number of mask tokens equal to the length of the subsequence , as in MASS ) . Zhang et al . ( 2019 ) additionally exploit the sentence structure of natural language . They suggest the pretraining task Pegasus , which masks out entire sentences of a given text , and additionally masks out randomly selected tokens in the remaining text ( or alternatively replace them by other tokens ) . In a similar way Pegasus ’ exploitation of the sentence structure of natural language , our skip-tree task exploits the tree structure of formal expressions . Zhang et al . ( 2019 ) also suggest sampling the sentences to be masked with the help of ROUGE1-F1 ( Lin , 2004 ) . We work with the HOList dataset by Bansal et al . ( 2019 ) , which is closely related to the Flyspeck dataset by Kaliszyk and Urban ( 2014 ) . There are other datasets which might be suitable for our approach as well , including proofs extracted from HOL4 ( Gauthier et al. , 2017 ) , and from Coq ( Huang et al. , 2019 ; Yang and Deng , 2019 ; Sanchez-Stern et al. , 2019 ) . Most previous works that apply sequence-to-sequence models to logics have focused on specific logical tasks in supervised training settings ( e.g . Piotrowski and Urban ( 2020a ) ) . In contrast , we train language models on a self-supervised proxy task that does not require labeled data and can thus be applied to almost any source of mathematical expressions . Lample and Charton ( 2020 ) use a Transformer model for symbolic integration . They train their model directly on the the task to produce the integral of a given expression . To generate training data , their approach needs a classical algorithm to compute the derivative of the expressions . Finkbeiner et al . ( 2020 ) explore the generalization properties of the Transformer architecture predicting the solutions to SAT formulas and temporal logic , but require a data generator that can solve formulas , which is currently not feasible for higher-order logic . Piotrowski et al . ( 2019 ) train RNNs on individual logical reasoning steps , such as substitutions , using a dataset of rewrites on polynomials extracted from Prover9 . Wang et al . ( 2018 ) translate between synthetic descriptions in natural language and formal mathematics on a dataset generated with Mizar . Self-supervised training techniques for formal mathematics have received much less attention . Wang et al . ( 2020 ) apply recent self-supervised translation techniques by Lample et al . ( 2018 ) to align formal and informal statements . Very recently , Li et al . ( 2020 ) and Polu and Sutskever ( 2020 ) applied language modeling to proofs of formal mathematics . In contrast , this work focuses on measuring reasoning abilities on mathematical statements ( not necessarily proofs ) achieved through self-supervised training only . Independently from our work , Urban and Jakubův ( 2020 ) presented initial experiments on applying self-supervised language modeling to formal mathematics in order to produce conjectures . However , they only evaluate the learned models through the truth of the produced conjectures , while we also consider several reasoning tasks and measure the usefulness of conjectures . Earlier methods to produce conjectures were limited in scope . For example , Piotrowski and Urban ( 2020b ) propose a method to predict the next literal in an automated theorem prover using recurrent neural networks after supervised training . Prior to that Gauthier et al . ( 2016 ) relied only on statistical approaches to produce conjectures . Applying natural language techniques to formal mathematics has a long history . Already in 2004 , Cairns ( 2004 ) applied information retrieval based on latent semantics to improve over search for keywords , and Urban ( 2004 ) formulated the intention to learn from large amounts of data in formalized mathematics . Transformer models for program understanding have focused on providing inductive biases in the architecture ( Shiv and Quirk , 2019 ; Hellendoorn et al. , 2020 ) , whereas this work suggests to use a modified language modeling proxy task . 3 DATASET . We start from the HOList dataset introduced by Bansal et al . ( 2019 ) . The complete dataset includes 29465 theorems and their proofs . We here consider only the “ core ” and “ complex ” datasets which comprise 18943 theorems , 637 definitions and 603,950 proof steps . The theorems and proofs were written ( by humans ) using the HOL Light proof assistant , and span various areas of mathematics such as set theory , arithmetic , linear algebra , topology , and multivariate complex analysis . Each proof starts with the theorem statement as a proof goal . For each goal , the dataset contains a tactic that a human applied to it , which then produces a list of subgoals . Most tactics have arguments , such as previously proven theorems ( also called premises ) , which are invoked by that tactic . From this dataset we extract all theorem statements as well as all intermediate proof goals . We use S-expressions to represent statements . For example , consider x = y , which in s-expression syntax is represented as follows : ( a ( a ( c ( fun ( A ) ( fun ( A ) ( bool ) ) ) = ) ( v A x ) ) ( v A y ) ) Each subexpression here is either a leaf or a triple . The first element of these triples indicates their kind : a indicates function applications , c indicates constants ( i.e . symbols that have been defined in the formal system ) , v indicates a variable , and finally fun indicates a function type . The equality operator “ = ” is represented by ( c ( fun ( A ) ( fun ( A ) ( bool ) ) ) = ) , which indicates that it is a constant that has type ( fun ( A ) ( fun ( A ) ( bool ) ) ) . This type indicate it is a curried function with two arguments of arbitrary type , indicated by the generic type variable A , and returns a bool . The variables x and y are represented as ( v A x ) and ( v A y ) . The v indicates that this subexpression is a variable , and A is a type variable , which indicates that x and y could have any type . Since both x and y use the same type variable A , they must have the same type . In our dataset , we wrap each S-expression derived from a theorem in a < theorem > node , and all other expression in a < goal > node . That is , an expression ( c ( fun ... ) ... ) is represented now as ( < goal > ( c ( fun ... ) ... ) ) . We use the same split as used in HOList ( Bansal et al. , 2019 ) . They first split the theorems into training , validation , and test , and then assign all statements in entire proof of each theorem to the same split as the theorem . This means that we have used the proof of 11,655 theorems in the training split of the core and complex libraries . This avoids partially revealing the proofs of theorems in the validation and test sets during training . We derive all training data from the theorems and proofs in the training set , and use only the theorems ( not the proofs ) for the evaluation tasks . This addresses the possibility that some proof steps for training theorems and for validation theorems might be shared . In Figure 1 we depict our choice of training and evaluation data .

Summary: This paper extends the idea of language-model style self-supervised learning approach to training logical reasoning models from unlabeled mathematical expressions. The main idea is to develop a skip-tree proxy task (self-supervision) for training the encoder-decoder architecture.  The skip-tree method masks out a complete sub-tree in the input and linearizes it into a sequence in the form of S-expression. The model is required to predict the masked subtree at the decoder end. The paper also proposes several new reasoning tasks for evaluating the model performance. Experimental results show that models learned from this task significantly outperform those trained on the skip-sequence task. Furthermore, the model also exhibits good conjecturing ability in generating quite reasonable amount of new theorems that are provable and useful, which is quite encouraging and impressive.