SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andrej Bauer
Faculty of Mathematics and Physics
Institute for Mathematics, Physics and Mechanics
andrej.bauer@fmf.uni-lj.si
Matej Petković
Department of Knowledge Technologies
Jožef Stefan Institute
matej.petkovic@fmf.uni-lj.si
Ljupco Todorovski
ljupco.todorovski@fmf.uni-lj.si
We introduce MLFMF, a collection of data sets for benchmarking recommendation systems used to support formalization of mathematics with proof assistants. These systems help humans identify which previous entries (theorems, constructions, datatypes, and postulates) are relevant in proving a new theorem or carrying out a new construction. Each data set is derived from a library of formalized mathematics written in proof assistants Agda or Lean. The collection includes the largest Lean 4 library Mathlib, and some of the largest Agda libraries: the standard library, the library of univalent mathematics Agda-unimath, and the TypeTopology library. Each data set represents the corresponding library in two ways: as a heterogeneous network, and as a list of s-expressions representing the syntax trees of all the entries in the library. The network contains the (modular) structure of the library and the references between entries, while the s-expressions give complete and easily parsed information about every entry. We report baseline results using standard graph and word embeddings, tree ensembles, and instance-based learning algorithms. The MLFMF data sets provide solid benchmarking support for further investigation of the numerous machine learning approaches to formalized mathematics. The methodology used to extract the networks and the s-expressions readily applies to other libraries, and is applicable to other proof assistants. With more than 250 000 entries in total, this is currently the largest collection of formalized mathematical knowledge in machine learnable format.
Applications of artificial intelligence to automation of mathematics have a long history, starting from early approaches based on a collection of hand-crafted heuristics for formalizing new mathematical concepts and conjectures related to them [Lenat, 1977]. In the last decade, there has been a growing interest in formalization of mathematics with proof assistants, which verify the formal correctness of
mathematical proofs and constructions, and help automate the tedious parts. The trend is correlated with the interest of machine learning community in aiding formalization efforts with its expertise.
Machine learning methods are often used to address premise selection, i.e., recommendation of theorems that are useful for proving a given statement. DeepMath [Irving et al., 2016] proposes using convolutional and recurrent neural networks to predict the relevance of a premise for proving the given statement. While many other approaches [Polu and Sutskever, 2020, Welleck et al., 2022] use transformers and general language models, Paliwal et al. [2020] have shown that taking into account the higher-order structure of logical expressions used in formalizing mathematics can greatly improve the performance of premise selection and automated proving. Indeed, many approaches use graph neural networks to learn from the higher-order structures, e.g., [Wang et al., 2017]. More recently, graph neural networks have also been proven useful for explorative, unsupervised approaches to automated theorem proving with reinforcement learning [Bansal et al., 2020, Lample et al., 2022]. Some of these approaches address alternative tasks, such as recommending or automatically selecting suitable proof tactics, i.e., routines for performing a series of proof steps, applying a decision procedure, or for carrying out proof search.
Data sets of different origins have been used to evaluate the proposed approaches. Welleck et al. [2022] evaluate their approach on a selection of three hundred proofs included in the ProofWiki [proofwiki] library of mathematical proofs written in a combination of natural language and Latex. Polu and Sutskever [2020] use a standard library of the Metamath proof assistant. Lample et al. [2022] combine proofs from the Metamath library with proofs from the Mathlib library [Mathlib] of the Lean proof assistant. The latter has also been used for evaluating the approaches in [Han et al., 2022]. Wang et al. [2017], Paliwal et al. [2020], Bansal et al. [2020] evaluate their models within the HOL Light proof assistant based on higher-order logic [Harrison, 2009]. The formalized proofs in standard HOL libraries have been transformed into a HOLStep data set for machine learning, where examples correspond to more than 2 million steps from 11400 proofs [Kaliszyk et al., 2017]. The training set includes proof steps in context (local hypotheses and the current statement being proved) and the library entry used in the step. Descriptions of the library entries are included in human-readable and machine-readable, tokenized versions. The data set has been recently upgraded to the interactive benchmark environment HoList for training automated proof systems with reinforcement learning [Bansal et al., 2019].
We present a collection of data sets, MLFMF, based on libraries of formalized mathematics encoded in two proof assistants, Agda and Lean. It supports evaluation and benchmarking of machine learning approaches for recommendation systems in the context of formalized mathematics.
ID	entry
N	N:
zero: N
succ(n): N → N
+	0 + n = n, for all n ∈ N
succ(m) + n = succ(m+n), for all m, n ∈ N
Lemma 1 (L1)	m + 0 = m for all m ∈ N.
This is proved by induction on m.
Lemma 2 (L2)	m + succ(n) = succ(m+n), for all m, n ∈ N.
This is proved by induction on m.
Theorem (T)	m + n = n + m, for all m, n ∈ N.
This is proved by induction on m. In the base case (m = 0), we need L1. In the induction step (m = succ(ℓ)), we need L2.
Table 1: An example formalization of proof that the addition of natural numbers is commutative.
We transform each library into a directed multi-graph whose nodes represent library entries (theorems, lemmas, axioms, and definitions), while edges represent the references between them. Consider the example in Table 1. It starts with a definition of the set of natural numbers with two simple constructors that define the first natural number 0 and constructs all the others inductively by asserting that a successor  $\operatorname{succ}(\mathfrak{n})$  of a natural number  $\mathfrak{n}$  is also a natural number. The definition of the addition of natural numbers follows their definition by asserting two simple rules for the left addition of 0 and the left addition of a successor. Note that the definition of  $+$  references the definition of  $\mathbb{N}$ . Next, the first lemma establishes the rule for the right addition of zero as the first simple commutativity case.
The second lemma establishes the right addition of a successor as the second case. The theorem at the end references the two lemmas to show (and prove) the commutativity of adding natural numbers.
The entries from Table 1 are transformed into a multi-graph depicted in Figure 1a. It contains five nodes, each corresponding to a table row. The multi-graph includes an edge from the node  $+$  to the node  $\mathbb{N}$ , indicating the reference to the set of natural numbers in the definition of addition. It also contains the self-reference of  $+$ , since the second case of this definition is recursive. Similarly, there are four edges from the theorem node to the two lemma nodes and the two nodes defining natural numbers and addition thereof. The obtained data allows us to approach premise selection as a standard edge prediction machine learning task.
Furthermore, we transform each formalized entry into a directed acyclic graph that retains complete information about the entry, see Figure 1b. By including the entire entry structures in the data sets, we make them suitable for further exploration of the utility of the state-of-the-art approaches to graph-based machine learning. A detailed description of the format is given in Sections 3.3 and 3.4.
Figure 1: The two-part representation of a library. Library as a whole is represented as a network of references (a). Additionally, every entry is represented as a DAG which is shown here in its textual s-expression format (b). Note that some nodes of DAG were replaced by  $(\dots)$  for better readability.
Our approach is general and can be applied to other proof assistants based on type theory. Moreover, even though Agda and Lean have quite different internal representations, the corresponding data sets use a common format that requires little or no knowledge about the inner workings of proof assistants. Thus our collection provides the machine learning community with easy access to a large amount of formalized mathematics in familiar formats that allow immediate application of machine learning algorithms. To our knowledge, MLFMF is the first and most extensive collection of data sets featuring more than one proof assistant and providing access to the higher-order structured representation of more than 250 000 mathematical formalization entries.
2 Formalized Mathematics
Formalized mathematics is mathematics written in a format that allows algorithmic checking of correctness of mathematical proofs and constructions. The programs that perform such checking are called proof assistants or proof checkers. An early proof checker was AUTOMATH [Bruijn, 1970], while today the most prominent assistants are Isabelle/HOL [Isabelle, HOL, Harrison], Coq [Coq], Agda [Agda] and Lean [de Moura et al., 2015]. They are all interactive: As the user develops a piece of formalized mathematics the assistant keeps track of unfinished proof goals, displays information about the current goal, checks the input on the fly, and provides search and automation facilities.
The level of automation varies between different proof assistants. In Agda, which supports little automation, the user directly writes down proofs and constructions in abridged type-theoretic syntax that Agda checks and algorithmically elaborates to fully formal constructions. On the other end of the spectrum are Isabelle/HOL and Lean, where the user relies heavily on tactics, which are routines that automatically perform various tasks, such as running a domain-specific decision procedure, applying a heuristic, or carrying out proof search.
The mathematical formalism most commonly used as the underpinning of a proof assistant is type theory, of which there are many variants [Church, 1940, Martin-Lof, 1975, Coquand and Huet, 1988]. The proof assistant processes the user input by disambiguating mathematical notations, applying tactics and other meta-level processing commands, and internally stores the resulting proofs, theorems, constructions, definitions, and types as expressions, or syntax trees, of the chosen type theory. These are typically quite verbose, so that checking their correctness is straightforward, but contain many more details than a user may wish to look at.
Libraries of formalized mathematics comprise units, organized hierarchically with a module system or namespaces, each of which contains a number of entries: definitions of types, constructions of elements of types, statements and proofs of theorems, unproved postulates (axioms), as well as meta-level content, such as embedded documentation, definitions of tactics, hints for heuristics, and other automation mechanisms.
In the last decade the libraries of formalized mathematics have grown considerably, most recently with the rise of the popularity of the Lean proof assistant and the Mathlib library [Mathlib, community, 2019], around which a mathematical community of several thousand mathematicians has formed. Such growth presents its own challenges, many of which are of the software engineering kind and can be so addressed. In our work we addressed the specific problem of recommendation: given a large body of formalized mathematical knowledge, how can the proof assistant competently recommend theorems or constructions that are likely useful in solving the current goal? There are two typical scenarios: the user knows which theorem they would like to use but have a hard time finding it in the library, or the user is not aware of the existence of a potentially useful theorem that is already available. Both are obvious targets for machine-learning methods.
3 MLFMF Data Sets
In this section we describe our data sets in detail. We first explain the semantic content of the data extracted from libraries of formalized mathematics, describe the format and information content of the data sets, continue by reviewing the machine learning tasks for which the data sets were built, and finish with an overview of the technical aspects of the library-to-data-set transformation process.
3.1 The Extracted Data
Formalized mathematics is written by the user in a domain-specific language, often called the vernacular or the meta-language. The proof assistant evaluates the source code, which involves executing tactics, decision procedures, etc., verifies that the proofs and constructions so generated are mathematically valid, and stores the results using an internal type-theoretic format. One may apply machine learning techniques directly on the vernacular, as written by the user, or on the formal representation of mathematics. The former approach roughly corresponds to learning how to do formalized mathematics, and the latter what formalized mathematics is.
We took the latter approach, namely learning on the formalized mathematics itself, for two reasons. First, because we aimed at a uniform approach that is applicable to most popular proof assistants, it made sense to use the internal type-theoretic representations, which are much more uniform across proof assistants than the vernaculars. Second, the vernacular contains meta-level information, such as what tactics to use, from which one cannot discern directly which theorems are actually used in a given proof. Without this information, one can hardly expect a recommendation system to work well.
Every data set that we prepared is generated from a library of formalized mathematics. Most libraries, and all that we incorporated, are organized hierarchically into modules and sub-modules, each of which is a unit of vernacular code that, once evaluated by the proof assistant, results in a list of entries: definitions of types, constructions of elements of types, theorems and their proofs, and unproved postulates. The entries refer to each other and across modules, possibly cyclically in case of mutually recursive definitions.
The internal representations of entries vary across assistants, but all have certain common features:
1. Each entry has a qualified name  $M_1.M_2 \ldots M_k.N$  by which it is referred to, where  $M_1.M_2 \ldots M_k$  is a reference to a module in the module hierarchy and  $N$  is the local name of the entry, for example Algebra.Group.FirstIsomorphismTheorem.
Figure 2: The two stages of the data transformation. First, a language-dependent (i.e., Agda or Lean) command line tool is used to transform the library entries into s-expressions. In the second stage, Python scripts are used to explicitly construct the directed multi-graph, which contains library modules, entries, and references among them.
2. Each entry has an associated type  $T$ , which specifies the information content of the body of the entry. For example, the type List(N) specifies that the entry is a list of natural numbers. Importantly, logical statements are just a special sort of types, so that the type of a proof is the logical statement that it proves. (This is to be contrasted with first-order logic, where logical statements are strictly separated from types.)
3. An entry has a body, which is an expression of the given entry type. In some cases the body may be missing, for instance if the user declares an axiom.
4. Depending on the proof assistant, various meta-level information is included, such as which arguments to functions are implicit (need not be provided by the user).
In this section, we describe the data set. We start with a brief description of data transformation process, continue with the detailed description of the resulting pair of computational graphs for the entries in the library, and the directed, multi graph of references in the library (see 3.3) and 3.4).
Every data set consists of two parts. The first part is a set  $\mathcal{T}$  of abstract syntax trees (AST) that correspond to the entries in the library, while the second is a directed multi-graph  $G(V,E)$ , where  $V$  is a set of library entries, and  $E$  includes the references among them. ASTs are actually trees in the case of Agda libraries. However, in Lean, they are directed acyclic graphs (DAGs) due to memory optimization and node-sharing: all the parents that would potentially reference their own copy of a node (or a subtree), rather reference the same node. For this reason, we refer to them as computational graphs. They provide the full information about every entry in the library and are given in the s-expression format that is much easier to parse, as compared to the typically very flexible syntax of proof assistants that allows for implicit arguments, mix-fix notation, etc. For example, the function if_then_also x y z in Agda can be called as if x then y else z. Learning from the source code would put an additional burden on the machine learning algorithm. Learning directly from computational graphs, on the other hand, is much easier.
Figure 3: The DAG representing an entry has a single root node with three children: a node containing the entry name, a DAG containing the entry declaration, and a DAG representing the entry body.
3.3 The Computational Graphs
During compile time, the full type of every entry in the library is computed and the source code of the entry is converted to a (directed acyclic) computational graph. We intercept this procedure and export every entry as a Lisp  $s$ -expression, which is defined recursively as:
1. A literal is an s-expression, and
2. A list of s-expressions is an s-expression.
For example, the literals 12 and "foo" are s-expressions, and the list ("foo" ("bar" 12) "baz") is also an s-expression with three elements: "foo", ("bar" 12) (which contains two s-expressions) and "baz". Every s-expression that is obtained from the entries in a library is three-part, as shown in Fig. 3. In consists of the name of the entry, the s-expression that describes the declaration, and the s-expression that describes the body of the entry.
Even though the entries were manually encoded and mostly take at most a few kilobytes of space, their computational graphs can be much larger (more than a gigabyte), mostly due to the type checking and the expansion of the declared type of the entry.
3.4 The Multi-Graph
For simplicity reasons, we will refer to the directed multi-graph  $G(V,E)$  simply as a graph. Its meta-structure is shown in Fig. 4a. In the description below, we follow this structure (in the bottom-up manner) and the concrete example of a subgraph for Agda's standard library in Fig. 4b.
Entry nodes. Every module in a library defines at least one entry (shown as green circles), e.g., bijection DEFINES id, B bijection DEFINES Bijection (these are two different nodes), and Injection DEFINES injective. We further differentiate between different kinds of entries, as shown in Tab. 2. Most of the nodes in the graph are entries (and most of them are functions), and most of the edges are of type REFERENCE FROM DECLARATION/BODY).
Library and module nodes. The only nodes with no incoming edges (root nodes) are the library nodes (shown as blue squares). Every graph contains at least one library node—the one that corresponds to the library itself. In Fig. 4b, this is the node std1ib. However, the graph might contain
(a) The meta-structure of the graph.
(b) An excerpt from Agda's standard library. The entry id (indentity function) is defined in the module B bijection, which is a submodule of the module Function, which is one of the top modules of the library. In the declaration of id, the entry B bijection is referenced. In the body of id, the entry injective is referenced. The two referenced entries are defined in the submodules B bijection and Injection, respectively.
Figure 4: A meta-graph of libraries (a) and a subgraph of the graph that was created from Agda's standard library (b) that follows the prescribed meta-structure.
Table 2: Tags of the nodes in s-expressions.
kind	description
:data	inductive data type (natural numbers, lists and trees)
:constructor	data-type constructor (successor, cons)
:function	function (including constants as nullary functions)
:record	record type (a structure with named fields or attributes)
:axiom	postulated type or statement (no inhabitant or proof given)
:primitive	built-in (primitive) function
:sort	the sort of a type (proposition, universe at a given level)
:recursor	the induction/recursive principle associated with an inductive data-type
:abstract	entry whose body is hidden
an additional node outer library if any of the library entries reference some external entries that are not part of the library (for example, built-in types). The library nodes are directly connected to the nodes representing modules (shown as blue diamonds) via the edges of the type CONTAINS, e.g., stdlib CONTAINS Function. Every module can contain zero or more (sub)modules, e.g., Function CONTAINS B bijection.
In the case of Agda libraries, the module nodes correspond to the modules that are actually present in the library and resemble the file system of the library. Lean, however, supports the use of namespaces. If the file a/b/c.lean defines an entry foo.bar.F, and the file d/e.lean defines an entry foo.bar.G, those two entries are part of the same namespace foo.bar and the exact location in the file system where these two entries were defined, is irrelevant. Therefore, module nodes for Lean's library Mathlib4 correspond to namespaces in the library. Following the previous example, we create module nodes foo and bar, together with the edge foo CONTAINS bar.
3.5 Machine Learning Tasks
The main motivation for the creation of the data set was the development of machine learning algorithms that would enhance current proof assistants and help mathematicians using them. This translates to the following two machine learning tasks.
Link prediction. Given the current state of the multi-graph of references among the entries, learn a model that predicts the future, novel links (references) among the library entries. Formally, we learn a model  $M:(u,v)\mapsto M(u,v)\in [0,1]$  that given two nodes  $u$  and  $v$  outputs the model confidence in the presence of the edge  $(u,v)$ . The (current) computational graphs of the entries can be used as additional information for learning such a model. If learning from the multi-graph only, one can use standard node- or edge-embedding approaches as well as graph neural networks.
Recommendation. The problem of predicting the future references among the entries could be understood as a recommendation task as well. Given a specific unfinished entry (possibly with some additional context, such as the list of lemmas/ claims that were used last), the task is to recommend the candidates that could be referenced in the current computational graph of the entry to complete it.
Note that the two tasks are equivalent, i.e., solving one solves the other. A link prediction model  $M$  (see above) can be converted into a recommendation system by fixing the entry  $u$  and recommending the entries  $v \in V$  with the highest confidence levels  $M(u, v)$ . Vice versa, given a recommendation model  $M': u \mapsto M'(u) \subseteq V$ , we can define a corresponding link-prediction model  $M$  as  $M(u, v) = 1$  if  $v \in M'(u)$ , and  $M(u, v) = 0$  otherwise.
Since the essential part of the MLFMF data set is a directed multi-graph (which represents a heterogeneous network), other standard learning tasks for graphs/ networks might also be interesting. Here, we mention two example instances of the common node classification task.