miniF2F: a cross-system benchmark for formal Olympiad-level mathematics

1 INTRODUCTION . Shared benchmarks and datasets have historically played a crucial role in driving advances in largescale applications of deep learning , e.g . in computer vision ( Deng et al. , 2009 ) and natural language processing ( Wang et al. , 2019 ; Rajpurkar et al. , 2016 ; Paperno et al. , 2016 ) . Neural theorem proving is a rapidly developing area which aims to apply techniques from deep learning to interactive theorem proving . To date , most contributions in this area have focused on individual theorem proving systems , each with a separately-implemented mathematics library and with results reported on a dataset-specific test split ; examples include the HOList ( Bansal et al. , 2019a ) , CoqGym ( Yang & Deng , 2019 ) and LeanStep ( Han et al. , 2021 ) theorem proving environments and benchmarks . However , benchmarks from this paradigm are not ideal for measuring the mathematical reasoning ability of neural theorem provers for several reasons . Library-specific train/test splits are siloed by construction , dependent on how theorems and lemmas are split in these libraries , and as such are not directly comparable across systems . Moreover , formal mathematics libraries are closer to software repositories than informal mathematical exposition , and many lemmas are implementation-specific artifacts without precise informal mathematical or cross-system translations . To date , the neural theorem proving community has not organized its efforts around a cross-system benchmark . To address this need and to provide a common resource to research groups working on formal theorem proving , we present miniF2F , a unified cross-system benchmark of formal mathematics of progressively increasing difficulty , centering around Olympiad-level problem statements ( AMC , AIME , IMO ) as well as high-school and undergraduate maths classes . Both the content and name of miniF2F are inspired by the IMO Grand Challenge ( Selsam et al. , 2019 ) : to build an AI that can win a gold medal in the International Mathematical Olympiad in a formal-to-formal ( F2F ) format . More precisely , the agent must receive IMO problems written in a formal mathematical format , and must produce a formal ( i.e . machine-checkable ) proof for that problem . We intend for miniF2F to serve as a stepping stone for different formal systems towards the IMO Grand Challenge ( Selsam et al. , 2019 ) , as it is end-to-end verifiable , cross-platform and spans a wide range of difficulty . While we report baseline results on miniF2F using GPT-f , a language model based on GPT-3 which has been finetuned for theorem proving , language models are not a mandatory approach for Olympiad problems and this assumption is not reflected in miniF2F , preserving the generality and widespread applicability of the benchmark to systems similar to DeepHOL ( Bansal et al. , 2019a ) or Holophrasm ( Whalen , 2016 ) . 2 BACKGROUND AND RELATED WORK . BENCHMARKS In the closely related field of ( first-order ) automated theorem proving ( ATP ) , the TPTP ( Sutcliffe , 2017 ) benchmark is a library of test problems in a unified format for ATP systems . In interactive theorem proving , the ” Freek 100 ” ( Wiedijk , 2008 ) tracks progress across various interactive theorem provers on a list of 100 mathematical theorems . Wu et al . ( 2021 ) built a simplified formal proof environment INT with an associated synthetic inequality benchmark . Competitions and communal challenges have also spurred development in formal theorem proving . The CADE ATP System Competition ( CASC ) ( Sutcliffe , 2016 ) is a competition that evaluates the performance of first-order automated theorem proving systems . Proof Ground ( Haslbeck et al. , 2019 ) , part of the ITP conference , is an interactive proving contest ( for humans ) that supports Coq , Isabelle , and Lean , which focuses on evaluating the formalization effort of proof to given problems within limited time . Finally , the IMO Grand Challenge ( Selsam et al. , 2019 ) , a proposal from researchers working on the interactive proof assistant Lean , aims to build a system capable of solving IMO problems in the formal-to-formal format . Due to its convenient framing as a natural language processing task , the domain of informal mathematical reasoning has received more attention than the formal one . MATH ( Hendrycks et al. , 2021 ) is a mathematics benchmark comprising 12,500 statements in natural language where exercises are classified into 5 levels of difficulty across various domains . Each exercise is combined with a detailed step-by-step proof in natural language . Scaling state-of-the-art models shows little amelioration on MATH , which requires advanced mathematical reasoning capabilities . miniF2F includes a number of formalized statements from MATH . NaturalProofs ( Welleck et al. , 2021 ) is another benchmark of natural proof in mathematics , containing 32k theorem statements and proofs . It essentially contains the proofs in ProofWiki and other resources . While MATH is more oriented towards mathematics exercises , NaturalProofs is focused on proofs of general mathematics theorems . Saxton et al . ( 2019 ) built a mathematics dataset with 2 × 106 training data and 104 test data , presented in a question-answering format where each statement is paired with a question written in natural language and a direct answer without proof . NEURAL THEOREM PROVING HOList ( Bansal et al. , 2019a ; b ; Paliwal et al. , 2020 ) provides an environment as well as a benchmark for HOL Light . They also proposes various deep reinforcement learning approaches for theorem proving and report a pass rate of 59.91 % on their benchmark . Yang & Deng ( 2019 ) built CoqGym , a large-scale dataset , which comes also with a learning environment , of 71k human-written proofs in Coq proof assistant . They report a 30.0 % pass rate on the held-out test theorems in CoqGym . Polu & Sutskever ( 2020 ) applied a decoder-only transformer similar to GPT-3 ( Brown et al. , 2020 ) to proof steps prediction in Metamath combined with a log-probability based proof search . They also proposed a methodology to train a value function to further guide proof search , achieving a 56.22 % pass rate on the held-out test set . Large language models were applied to Lean by Han et al . ( 2021 ) . They created an environment around the Lean prover targeted to machine learning and propose a dataset extracted from low level proof artifacts that is shown to boost performance when used as a self-supervised co-training objective . They report a 48.4 % pass rate on held-out test statements from mathlib , Lean ’ s mathematical library ( mathlib Community , 2020 ) . 3 MINIF2F BENCHMARK . miniF2F is a dataset of manually formalized statements of Olympiad type problems , aligned in Lean , Metamath , and Isabelle ( partial at the time of writing ) , providing a cross-platform benchmark for formal mathematical reasoning . Olympiad type problems are of particular interest to compare automated provers across different formal systems as the theories required to solve them are well identified and they generally do not require the definition of new mathematical concepts ( a capability that remains beyond the current neural theorem proving state of the art ) . The formalized statements in miniF2F are drawn from multiple sources , ranging from high school and undergraduate level exercises to Olympiad problems . miniF2F also covers different subsubjects in mathematics as well as proof strategies , focusing on the types of exercises whose statements are expressible in most formal systems . This leads to a systemic focus on algebra , number theory and inequalities because , for example , geometry and combinatorial problems are generally challenging to formalize due to only nascent efforts in these areas in most formal systems . The statements in miniF2F are all manually formalized and selected to cover a variety of difficulty levels for both humans and machines . Formal proofs for these statements are optionally attached . miniF2F draws from AIME , AMC , IMO problems as well as problems from the MATH ( Hendrycks et al. , 2021 ) informal dataset . Formalizing problems from the MATH dataset serves two purposes . First , problems in MATH are segmented by difficulty level ( from 1 to 5 ) , randomly selecting a subset from each of these difficulty levels allows miniF2F to cover a wider range of difficulty . Second , it provides the community an opportunity to compare capabilities of formal automated prover to their informal counter-parts as discussed in later sections . miniF2F comprises a test set and a validation set , which are a stratified random split from the statements we formalized such that each set equally covers each problem type and difficulty ( when available ) . Table 1 shows a detailed distribution of these statements . Versioning miniF2F is an evolving effort and new statements will continuously be added . Periodically , we will freeze versions of the benchmark . The current version of the benchmark is v11 and results in this paper are reported using this version . v1 comprises 244 test and 244 valid statements . The set of statements of each version is guaranteed to remain stable , only allowing fixes in case errors are later discovered . Rules of engagement and License miniF2F is meant to serve as a shared resource for research groups working on applying deep learning to formal theorem proving . There is no formal process to submit evaluation results and researchers are simply invited to cite miniF2F indicating the version used in their evaluations . We also encourage them to contribute proofs found by their approaches back to the benchmark . The parts of the benchmark associated with each theorem prover ( Metamath , 1https : //github.com/openai/miniF2F/tree/v1 Lean , Isabelle ) are meant to be licensed in a way that is aligned with the licensing usage associated with the theorem prover ’ s main library . As a result , the Metamath version of the benchmark is released under the MIT License , while the Lean and Isabelle versions are released under the Apache License . Formalization effort and challenges We found that , for trained practitioners ( but not necessarily experts , including students recently introduced to formal systems ) , formalizing a statement takes about 15 minutes on average , and reviewing a formalized statement , about half of that on average . Note that not all exercises are directly or naturally formalizable . In particular , multi-choice questions , word problems , and exercises that require to explicit a witness or a set as part of the answer present interesting challenges : multi-choice questions2 these problems are generally straightforwardly formalizable by reformulating the statement using the right answer only , and could be made “ fair ” in a competitive setup by formalizing all possible choices and running automated provers on all of them , attributing points only if a proof of the correct answer is provided . word problems3 where significant information is presented in natural language generally require non-trivial efforts to be formalized . We generally formalized them by explicitly modeling the mathematics concepts and expression presented in natural language while attempting as best as possible to preserve the mathematical difficulty of the original problem . Sometime the formalization work is most of the difficulty associated with the original question ; in such cases we would discard the problem entirely . problems that require to explicit a set or witness4 ( e.g . find all ... such that ... ) are not directly formalizable . The best approximation we relied on for these was to formalize the statement with the witness or answer provided , turning such exercises into the generation of a proof that the answer is correct , and if needed , that it is the unique one–which is , at times , a much easier exercise . A non negligible portion of IMO problems are as such , which we foresee could become a challenge in the future , to fairly compare humans to automated proving systems in a competitive setup . Porting effort In addition to Metamath , Lean , Isabelle ( work in progress ) and HOL Light ( work in progress ) , we are eager to extend the coverage of miniF2F to Coq , and will welcome any effort in that direction or to extend miniF2F to further systems .

Summary: This paper presents miniF2F, a test suite of Olympiad-level problems of theorem proving that is implemented in Metamath, Lean and Isabelle. MiniF2F contains 488 individual theorem statements that are formalized from Olympiad math contests. GPT-f models trained on Metamath and Lean are evaluated on this test suite.