SlowGuess/ACL_2020_OCR huggingface dataset:
The dataset is imbalanced by the nature of the natural premise selection problem. In order to solve the natural premise selection task, any approach would have to be able to handle a large number of negative examples. There are 10k different possible premises, and some conjectures are only connected to one premise, creating a large number of negative pairs in our dataset, requiring the definition of a cap for the number of negative samples. In order to provide a more constrained setting, we define a subset of the PS-ProofWiki, named PS-ProofWikiTRIG targeting trigonometric functions.
The proposed approach outperforms the BERT-based model by  $41\%$  in terms of F1-score, as shown in Table 4. We hypothesise that the encoding of the structural patterns of the dependency relations in addition to the content-based similarity better captures the semantic nature of the proof (fundamental to interpret a proof by its neighbourhood).
6.4 Scalability & Imbalance Robustness analysis
In order to evaluate the robustness of the proposed approach and the baseline with regard to an increase in imbalance (reflecting a notion of scalability of the quality of the inference within the KB), we compare how the F1-score changes as we add more (random) negative examples to the dataset.
Figure 5a and Figure 5b presents a comparison between BERT and our approach for the PS-ProofWikiTRIG and the PS-ProofWiki datasets, respectively.
The results indicate that the BERT-based classifier performance degrades faster as we increase the number of negative samples in the dataset. For  $n = 30$ , the F1-score reaches a value of almost zero. In contrast, the proposed model presents a significantly slower decline (25%), showing better scalability properties in the context of the premise selection problem.
Finally we experiment on how BERT and the proposed model compares when we consider transitivity between premises (n-hop relations), using PS-ProofWikiTRIG and 10 negative examples for each positive example. We report the results in Table 5, where we can see that the proposed model obtains better overall performance as the number of
7371
Table 4: Precision (P), recall (R), and F1-score (F1) for the BERT baseline and the proposed approach, with 30 negative examples for each positive case (values are multiplied by 100).
BERT	Proposed Model
P	R	F1	P	R	F1
PS-ProofWikiTRIG	39.9	22.9	29.1	34.0	50.0	40.5 (+ 39%)
PS-ProofWiki	47.1	26.7	34.1	48.5	47.7	48.1 (+ 41%)
(a) Evaluating on PS-ProofWikiTRIG
(b) Evaluating on PS-ProofWiki
Figure 5: Comparison of the proposed model and BERT, showing how both models perform (in terms of F1-score) when adding more negative examples to the training and test set.
hops is increased. These results reinforce the architectural design supported by graph-based models.
6.5 Qualitative analysis
From the results obtained from our model we observed that the model struggles to encode statements which are centered around pure equational (formulae) content. Embeddings for mathematical symbols should take into consideration more specific semantics of operators: such semantics is not obtained using PV-DM (Doc2Vec) or BERT. This provides evidence on the need for more principled structural embeddings for mathematical formulas, which could most certainly improve the prediction of future work in the natural premise selection task.
Table 5: Comparison of BERT and the proposed model for different levels of transitivity between premises (values are multiplied by 100).
BERT	Proposed Model
P	R	F1	P	R	F1
2-hop	47.5	78.9	59.3	54.8	68.7	61.0 (+3%)
3-hop	41.0	45.1	49.2	58.8	63.3	61.2 (+24%)
Even though BERT is not trained in a mathematical corpus, it still obtains relevant results, hinting that training BERT on a mathematical corpus could achieve better results. However, this task is outside the scope of this work and will be left for future work.
The proposed DGCNN-based model is capable of finding structural patterns between the statements and to reinforce content-based semantic evidence. We observed that statements that are similar in content, commonly have a significant intersection of premises, as a result of the graph embedding, the DGCNN-model is able to better discriminate more fine-grained semantic cues better.
In this work, we introduced an approach for natural language premise selection (finding relevant theorems, axioms and definitions) in large natural language mathematical texts. The proposed approach, which uses Deep Graph Convolutional Neural Networks (DGCNNs) combines both structural and content elements of mathematical statements for addressing the premise selection problem as a link prediction classification problem. Results show that the approach outperforms a BERT-based baseline by  $41\%$  in F1-score. Moreover, the proposed model shows significantly lower F1-score degradation concerning class imbalance, a fundamental desirable scalability property for the problem of premise selection.
Our approach is also able to obtain better performance when we consider the transitivity of premises. The qualitative analysis indicates that
7372
there is the demand to design principled embeddings for better capturing the semantics of proofs which are denser in mathematical formulae. As future work, we will explore different heuristics for navigating in the premises graph, as researched before for textual entailment (Silva et al., 2019, 2018) and selective reasoning (Freitas et al., 2014).
The authors would like to thank the anonymous reviewers for the constructive feedback, we also would like to thank Mokanarangan Thayaparan and Marco Valentino for the helpful discussions.
Jesse Alama, Tom Heskes, Daniel Kuhlwein, Evgeni Tsivtsivadze, and Josef Urban. 2014. Premise selection for mathematics by corpus analysis and kernel methods. Journal of Automated Reasoning, 52(2):191-213.
Marcos Cramer, Bernhard Fisseni, Peter Koepke, Daniel Kuhlwein, Bernhard Schroder, and Jip Veldman. 2009. The naproche project controlled natural language proof checking of mathematical texts. In International Workshop on Controlled Natural Language, pages 170-186. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Michael Farber and Cezary Kaliszyk. 2015. Random forests for premise selection. In Frontiers of Combining Systems, pages 325-340, Cham. Springer International Publishing.
Deborah Ferreira and Andre Freitas. 2020. Natural language premise selection: Finding supporting statements for mathematical texts. 12th Language Resources and Evaluation Conference (LREC), Marseille, France.
Andre Freitas, Joao Carlos Pereira da Silva, Edward Curry, and Paul Buitelaar. 2014. A distributional semantics approach for selective reasoning on commonsense graph knowledge bases. In International Conference on Applications of Natural Language to Data Bases/Information Systems, pages 21-32. Springer.
M Ganesalingam and William Timothy Gowers. 2017. A fully automatic theorem prover with human-style output. Journal of Automated Reasoning, 58(2):253-291.
Mohan Ganesalingam. 2013. The language of mathematics. In *The Language of Mathematics*, pages 17-38. Springer.
Thibault Gauthier and Cezary Kaliszyk. 2015. Premise selection and external provers for hol4. In Proceedings of the 2015 Conference on Certified Programs and Proofs, CPP '15, pages 49-57, New York, NY, USA. ACM.
Andr Greiner-Petter, Terry Ruas, Moritz Schubotz, Akiko Aizawa, William Grosky, and Bela Gipp. 2019. Why Machines Cannot Learn Mathematics, Yet. (July).
Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 887-896.
Geoffrey Irving, Christian Szegedy, Alexander A Alemi, Niklas Eén, François Chollet, and Josef Urban. 2016. Deepmath-deep sequence models for premise selection. In Advances in Neural Information Processing Systems, pages 2235-2243.
Kriste Krstovski and David M Blei. 2018. Equation embeddings. arXiv preprint arXiv:1803.09123.
Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International Conference on Machine Learning, pages 1188-1196.
Behrooz Mansouri, Shaurya Rohatgi, Douglas W Oard, Jian Wu, C Lee Giles, and Richard Zanibbi. 2019. Tangent-cft: An embedding model for mathematical formulas.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning convolutional neural networks for graphs. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pages 2014-2023. JMLR.org.
Vivian Dos Santos Silva, Siegfried Handschuh, and André Freitas. 2018. Recognizing and justifying text entailment through distributional navigation on definition graphs. In AAAI.
Vivian S Silva, André Freitas, and Siegfried Handschuh. 2019. Exploring knowledge graphs in an interpretable composite approach for text entailment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7023-7030.
Daniel Solow. 2002. How to read and do proofs an introduction to mathematical thought processes.
Marco Valentino, Mokanarangan Thayaparan, and Andr Freitas. 2020. Unification-based reconstruction of explanations for science questions.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do nlp models know numbers? probing numeracy in embeddings. arXiv preprint arXiv:1909.07940.
Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018. Translating a math word problem to an expression tree. arXiv preprint arXiv:1811.05632.
Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5299-5305. AAAI Press.
Muhan Zhang and Yixin Chen. 2018. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems, pages 5165-5175.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. 2018. An end-to-end deep learning architecture for graph classification. In Thirty-Second AAAI Conference on Artificial Intelligence.
Claus Zinn. 2003. A computational framework for understanding mathematical discourse. *Logic Journal of IGPL*, 11(4):457-484.
Yanyan Zou and Wei Lu. 2019. Text2math: End-to-end parsing text into math expressions. arXiv preprint arXiv:1910.06571.