SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Entry class detection. A straight-forward instance of node classification task would be classifying the entries into their types from Table 2, e.g., function or axiom. This does not require additional manual labeling and should not be too hard, especially when computational graphs are taken into account, since the structure of a function is quite different from the structure of, e.g., record.
Claim detection. A more challenging instance of node classification is predicting whether a function entry is a claim (e.g., a lemma, corollary, theorem, etc.) or not, since some of the entries are simply definitions of, for example, the addition of natural numbers. Approaching this task, however, would require additional (manual) labeling of the entries.
3.6 License
We make MLFMF publicly available under the Creative Commons Attribution 4.0 International $^{1}$  (CC BY 4.0) license at https://github.com/ul-fmf/mlfmf-data.
In this section, we first introduce the experimental setup for the baseline experiments (how to prepare the train and test part of the data set, and which standard metrics can be used), and then, after briefly introducing the baseline methods, we report the experimental results.
4.1 Train-test split
When splitting the graph into train and test data sets, we should split the multi-graph  $G(V,E)$  and the set of computational graphs of the entries. In the case of the link prediction and recommendation tasks, we should focus on function nodes, since these are the only nodes that correspond to a computational graph whose body contains a proof of a claim formalized in the declaration part of the computational graph.
In our baseline experiments from Sec. 4.4, we follow a generic approach to creating a train-test split. The approach takes two parameters:  $p_{\mathrm{test}} \in (0,1)$ ,  $p_{\mathrm{body}} \in [0,1)$ . First, we randomly choose the proportion  $p_{\mathrm{test}}$  of function nodes. We assume that those correspond to partially written entries whose computational graphs have completely specified type, i.e., the user knew how to formalize a claim, and partially known body, i.e., the proof of the claim is not finished yet. Note that, often, proofs are not written linearly and might contain so-called holes at problematic parts where the right lemmas are yet to be applied (possibly with already known arguments). Thus, we need to modify the computational graphs of the test nodes to reflect the changes in the multi-graph.
We simulate the applications of the missing lemmas by keeping only the proportion of  $p_{\mathrm{body}}$  of the references in the body. Since our graph contains a weighted edge u REFERENCE FROM BODY v, which we either remove or keep intact, we remove all references to v from the body of u or none of them. Then, the unfinished proofs are simulated by keeping the proportion of  $p_{\mathrm{body}}$  of the body of u, which is done by iterative pruning of the leaves of the body. At each iteration, a leaf is chosen uniformly at random. If the chosen leaf is a reference that we have to keep, the leaf is not pruned and we continue with the next iteration.
The removed edges represent positive test examples, and the negative test examples for learning predictive models need to be sampled. In the baseline experiments, the negative test examples were sampled uniformly at random.
4.2 Evaluation metrics
For link prediction, one can use standard classification metrics, such as accuracy, precision, recall, and  $F_{1}$ -score. If the model returns its confidence  $M(u,v) \in [0,1]$  instead of the class value  $(M(u,v) \in \{0,1\})$ , one could additionally consider area under the receiver-operating-characteristic curve. Similarly goes for the recommendation models: one can use precision and recall.
If the recommendation model returns the relevance score of a candidate entry to the current context, we can rank candidates according to the score values, with the top recommendation having a rank of one (1). We can then compute the minimal (and the mean) rank of the actual references and average them over the testing examples. This is an important metric, since it counts the number of false recommendations with better ranks than any of the actual references. Ideally, the minimal rank is close to one, i.e., the top-ranked recommendation mostly matches the missing entry to be referenced.
<sup>1</sup>https://creativecommons.org/licenses/by/4.0/
4.3 Baseline Methods
Dummy recommender. This recommender ignores the current context and always recommends the  $k$  nodes of the multi-graph with the highest in-degree.
Bag of Words/TFIDF recommenders. Bag of Words (BoW) recommender converts every computational graph  $g(u)$  of an entry  $u$  in a library into a bag of words  $\mathrm{BoW}(u)$ . We compute the relevance  $M(u, v)$  of the candidate entry  $v$  for the current context  $u$  using the Jaccard similarity between the corresponding bag-of-words:
$$
J (\mathrm {B o W} (u), \mathrm {B o W} (v)) = \frac {| \mathrm {B o W} (u) \cap \mathrm {B o W} (v) |}{| \mathrm {B o W} (u) \cup \mathrm {B o W} (v) |}.
$$
Similarly, TFIDF-recommender embeds  $g(u)$  into a term-frequency-inverse-document-frequency vectors (obtained from the corresponding bags-of-words) as implemented in Scikit-Learn 1.2.2 [Pedregosa et al., 2011]. The relevance of the candidate entry  $v$  is computed as a Manhattan or a cosine distance between the TFIDF-vectors of  $u$  and  $v$ .
FastText embedding recommender. It embeds every computational graph  $g(u)$  into a vector  $\vec{\varphi}(g(u)) = \sum_{\mathrm{word} \in g(u)} w(\mathrm{word}, g(u)) \cdot \varphi_{\mathrm{cc}}(\mathrm{word})$ , where  $\varphi_{\mathrm{cc}}(\mathrm{word})$  is the vector of word obtained from the fastText model trained of Common Crawl [Mikolov et al., 2018], and  $w(\mathrm{word}, g(u))$  is the TFIDF weight of the word in the entry  $u$ .
Recommendations via analogies. We design a recommender that is based on fastText analogies property, i.e., the fact that  $x =$  queen is one of the approximate solutions of  $\varphi_{\mathrm{cc}}(\mathrm{king}) - \varphi_{\mathrm{cc}}(x) = \varphi_{\mathrm{cc}}(\mathrm{man}) - \varphi_{\mathrm{cc}}(\mathrm{woman})$ . We design the analogy recommender that for a given entry  $u$  recommends the nodes  $v$ , for which a good analogy  $u' \to v'$  of the edge  $u \to v$  can be found. The relevance of the candidate entry  $v$  in a given context  $u$  is defined in terms of the Manhattan distance as
$$
r(u,v) = 1\quad \big{/}\min_{u^{\prime}\to v^{\prime}\in E(G)}\| [\varphi_{\mathrm{cc}}(u) - \varphi_{\mathrm{cc}}(v)] - [\varphi_{\mathrm{cc}}(u^{\prime}) - \varphi_{\mathrm{cc}}(v^{\prime})]\|_{1}.
$$
Node2vec-based link prediction. We train a node2vec [Grover and Leskovec, 2016] model (as implemented in Gensim 4.3.1 [Rehurek and Sojka, 2011] on the multi-graph to obtain node embeddings. We obtain the embedding of the edge  $(u,v)$  by concatenation of the node embeddings for  $u$  and  $v$ . A tree-bagging classifier  $M:\varphi (u\to v)\mapsto M(\varphi (u\to v))\in [0,1]$  is trained on the tabular data obtained with using the edge embeddings as inputs and the edge presence as the target to be predicted. We selected bag of trees ensemble since it is a robust classifier, working well on tabular data, when using the recommended settings of 100 fully grown (not pruned) classification trees.
We selected baseline methods that are not computationally expensive and are robust to hyperparameter settings: if not mentioned otherwise, the methods use the default parameter settings. We can combine multiple embeddings (e.g., those from node2vec together with those from fastText) as the input to the similarity measure of the recommender or classifier for the link prediction task. However, as noted in the next section, this did not improve the best results. In all the experiments, we generated the train-test split with the parameters  $p_{\mathrm{test}} = 0.2$  and  $p_{\mathrm{body}} = 0.1$ . The results here are reported for  $k = 5$  recommended items and the threshold  $\vartheta = 0.5$  for classification.
The experiments on Lean were run on a computer with 4 Intel Core i7-6700K CPU cores and 64 GB of RAM. The experiments on Agda were run on a smaller machine (2 Intel Core i7-5600U CPU cores, 12 GB of RAM). Experiments that would last more than a week were not carried out (analogies on the Type Topology and Mathlib4 libraries, and fastText on the Mathlib4 library).
Tab. 3 reports the results of the experiments: for extended report including other evaluation metrics (accuracy@k, area under the ROC curve, etc.), and ablation study of node2vec on Agda stldib, check the supplementary material. For the algorithms that were run with more than one parameter setting, the best results are reported (for example, TFIDF was run with cosine- and Manhattan-based similarity measures). The best-performing baseline method is node2vec. It is the only one that ranks on average at least one actual reference among the ten most relevant candidate references for the three
Agda libraries. However, it fails to do so for Lean Mathlib4 and this can be only partially explained by the size of the Mathlib4. Note that node2vec is also the only one that explicitly learns from the multi-graph and, apparently, humans writing proofs in Agda, structured the references better than the computers in Lean, where built-in search heuristics (tactics) are used. The multi-graph is partially used by the analogies recommender as well, since the candidate recommendations are evaluated by considering the existing references  $u' \rightarrow v'$  in the library. This might be the reason for its good performance on the Agda stdlib data set.
Table 3: The accuracy (acc) and minimal rank of the true reference for the MLFMF data sets. The best results (bold) are obtained with a combination of a node2vec and a tree-bagging classifier.
method	Agda stdlib	Agda unimath	Agda TypeTopology	Lean Mathlib4
acc	minRank	acc	minRank	acc	minRank	acc	minRank
Dummy	0.51	218	0.53	2134	0.50	4556	0.51	26065
BoW	0.50	1608	0.50	1571	0.50	4496	0.50	15458
TFIDF	0.51	144	0.52	112	0.51	552	0.51	443
fastText	0.51	132	0.52	394	0.50	1292	NA	NA
analogies	0.52	37	0.51	158	NA	NA	NA	NA
node2vec	0.96	4.37	0.96	3.24	0.98	5.81	0.95	195
Surprisingly, TFIDF embeddings perform no worse (or even better) than FastText embeddings. The reason for this might be that many words, such as group, ring, etc. have different meanings in mathematics than in general texts. Note that we tried to run additional experiments with the combination of node2vec and TFIDF/fastText embeddings, but accuracy and minRank were both worse, as compared to the node2vec results.
In sum, the baseline results show that the information on the structure of the multi-graph is crucial for obtaining classifiers with performance beyond the default performance of the dummy baseline. The recommendation performance, measured as mean minimal rank, is valuable enough (less than five recommendations to be checked to find the right one) for two Agda libraries. Developing sound recommendation systems for the other two libraries remains a challenge to be addressed by machine learning methods beyond the baselines considered here.
We introduced MLFMF, a suite of four data sets corresponding to three libraries in Agda and one library in Lean proof assistants. It includes almost 250 000 entries, i.e., definitions, axioms, and theorems with accompanying proofs. References between entries are included in a multi-graph, where nodes are entries, edges represent references among the entries, and each entry is represented with a direct acyclic graph reflecting the structure of the entry source code encoding. Such a structure provides machine learning researchers with an opportunity to address the task of recommending relevant entries for the goal at hand as a standard edge prediction task. Such a representation of the entries allows for use of graph-based methods that can exploit the structural and semantic information stored in the multi-graphs. The report on the results of the baseline methods establishes a benchmark for comparative evaluation of future developments of machine learning for mathematical formalization that goes beyond a single proof assistant.
A notable limitation of our data sets is the lack of information on the developmental evaluation of the libraries. If the latter could be followed, the more realistic test nodes  $V_{\mathrm{test}}$  could be defined as the latest  $|V_{\mathrm{test}}|$  nodes encoded in the library. However, even with version control information (available since the libraries are stored in GitHub repositories but currently not included), determining the entries' chronological order might be computationally expensive. An approximation of the chronological order might be obtained by computing a topological ordering on the function nodes and selecting nodes from the tail of the ordered list. However, existing definitions in a library might be rewritten, so the accuracy of such an approximation is questionable.
Finally, we plan to include other, more recent libraries in our data set collection. The newly incorporated libraries might include references to earlier, standard libraries, providing further opportunities for real-world testing scenarios.
This material is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-21-1-0024. The authors also acknowledge the financial support of the Slovenian Research Agency via the research core funding No. P2-0103 and No. P1-0294.
Agda. The Agda proof assistant. https://wiki.portal.chalmers.se/agda/.
Kshitij Bansal, Sarah M. Loos, Markus N. Rabe, Christian Szegedy, and Stewart Wilcox. Holist: An environment for machine learning of higher order logic theorem proving. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 454-463. PMLR, 2019. URL http://proceedings.mlr.press/v97/bansal19a.html.
Kshitij Bansal, Christian Szegedy, Markus N. Rabe, Sarah M. Loos, and Viktor Toman. Learning to reason in large theories without imitation, 2020. URL https://doi.org/10.48550/arXiv.1905.10501.
de NG Bruijn. The mathematical language automath, its usage and some of its extensions. In Symposium on Automatic Demonstration, Versailles, 1968, volume 125 of Lecture Notes in Mathematics, pages 29-61, 1970.
Alonso Church. A formulation of the simple theory of types. Journal of Symbolic Logic, 5(2):56-68, 1940. doi: 10.2307/2266170.
The Mathlib community. The Lean mathematical library, 2019.
Coq. The Coq proof assistant. https://coq.inria.fr/.
Th. Coquand and G. Huet. The calculus of constructions. Information and Computation, 76, 1988.
Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. The Lean theorem prover (system description). In 25th International Conference on Automated Deduction (CADE 25), August 2015. doi: 10.1007/978-3-319-21401-6_26.
Aditya Grover and Jure Leskovec. Node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 855-864, New York, NY, USA, 2016. Association for Computing Machinery. doi: 10.1145/2939672.2939754.
Jesse Michael Han, Jason Rute, Yuhuai Wu, Edward W. Ayers, and Stanislas Polu. Proof artifact co-training for theorem proving with language models. In *The Tenth International Conference on Learning Representations*, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=rpxJc9j04U.
John Harrison. The HOL Light theorem prover. Available at https://www.cl.cam.ac.uk/~jrh13/hol-light/.
John Harrison. Hol light: An overview. In Stefan Berghofer, Tobias Nipkow, Christian Urban, and Makarius Wenzel, editors, Theorem Proving in Higher Order Logics, pages 60-66, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg.
HOL. HOL Interactive Theorem Prover. Available at https://hol-theorem-prover.org.
Geoffrey Irving, Christian Szegedy, Alexander A. Alemi, Niklas Eén, François Chollet, and Josef Urban. DeepMath-deep sequence models for premise selection. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2235-2243, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/f197002b9a0853eca5e046d9ca4663d5-AAbstract.htm1.