Disambiguating Symbolic Expressions in Informal Documents

1 INTRODUCTION . Despite huge advancements in machine learning , the task of understanding informal reasoning is still beyond current methods . In fact , it became commonplace that humans annotate informal documents containing reasoning in many domains , e.g . law ( Libal & Steen , 2020 ) . Reasoning is most visible in mathematical documents and software specification and as such in the last decades , the formalization of mathematical knowledge , and the verification of formal proofs , has become increasingly popular . By now , dozens of interactive and automated theorem prover systems are available , each providing libraries with up to hundreds of thousands of formalizations of mathematical definitions , theorems , and their proofs written by human mathematicians ( Harrison et al. , 2014 ) . While formal methods are still primarily used by computer scientists ( e.g . to verify software and hardware , as well as in program synthesis ) , by now they have also drawn the interest of an increasing number of research mathematicians – primarily thanks to famous problems such as Kepler ’ s conjecture ( Hales et al. , 2017 ) or the classification theorem for finite simple groups ( Solomon , 1995 ) , which have successfully been verified using theorem prover systems . However , while some mathematicians have begun actively adapting formal methods for their work , there is a prohibitively large discrepancy between the way new mathematical results are developed , presented , and published in mathematical practice , and the way they are formalized and implemented in formal systems ( Kaliszyk & Rabe , 2020 ) : Most theorem proving systems implement a fixed logical foundation ( such as variants of set theory or various kinds of type theories ) , a surface syntax in which a user declares new definitions and statements in terms of the underlying foundations , and either a tactic language or a language for expressing proof terms ( usually on basis of the Curry-Howardcorrespondence in a typed λ-calculus ) that allow for declaring proofs . Consequently , the process of formalizing new content in a formal system resembles programming much more than it does developing informal proofs . This discrepancy results in severe challenges for traditional mathematicians : Formal systems are difficult to learn and use , even if one is well acquainted with the ( informal ) mathematics involved . They require learning dedicated formal languages resembling programming languages , declaring content on a level of detail that is prohibitive for beginners even for “ obvious ” conclusions , and their libraries are difficult to grasp without already being familiar with the system ’ s language , conventions and functionalities . Due to the required level of detail , knowledge of the existing libraries is crucial when formalizing new content . Furthermore , many “ intuitively valid ” arguments can not be easily expressed in terms of a logical foundation in the first place , and knowing how to deal with those requires familiarity with the logical foundation involved and lots of practice . Consequently , the utility of formalizing mathematical results can be too easily ( and too often is ) dismissed in light of the additional time and work required for non-experts . This is despite the fact that many services available for formal mathematics are already enabled by semi-formal ( or flexiformal ) representations , such as semantic annotations in natural language texts , or formal representations containing opaque informal expressions ( see e.g . Kohlhase ( 2013 ) ; Lange ( 2011a ) ; Iancu ( 2017 ) ; Kohlhase et al . ( 2017a ) ; Corneli & Schubotz ( 2017 ) ; Dehaye et al . ( 2016 ) ) . Therefore , we need to invest into methods for bridging the gap between informal mathematical practice and ( semi- ) formal mathematics . One way to do so is to investigate autoformalization , the task of ( semi-automatically ) converting existing informal mathematical presentations to ( increasingly ) formal representations . Notably , these issues extend beyond pure mathematics to other STEM ( science , technology , engineering and math ) fields , where the formal verification ( or lack thereof ) of results can have direct real-world implications – examples include an infamous and costly error in the floating-point unit of Intel processors ( Harrison , 2003 ) and several human failures to adequately convert between SI and imperial units , most famously in NASA ’ s Mars orbiter ( Grossman ) . In fact , the former has already established formal verification as a vital tool in hardware design ( Harrison , 2003 ) . Two observations motivate the research presented here : 1 . The vast majority of STEM researchers can be assumed to be comfortable with using LATEX ; any integration of formal methods in a LATEX development environment ( e.g . via new packages or IDE integration ) would consequently lower the entry barrier significantly . 2 . The task of going from purely informal mathematical texts to fully formal representations of the contained knowledge is best done via a separation of concerns , by focussing on individual subtasks ( such as disambiguating symbolic expressions , parsing natural language , and translating it to a formal foundation ) using dedicated tools for each . In this paper , we discuss specifically the task of disambiguating symbolic expressions – i.e . associating all symbols in an expression with their precise semantics – in LATEX documents as a machine learning task , using sTEX semantically annotated LATEX ( Kohlhase , 2008 ) . The contributions are threefold : 1 . We discuss the details of disambiguating symbolic expressions in informal STEM documents as a neural machine translation task , 2. we present a new dataset specifically for this task , based on the existing SMGLoM library of sTEX macros ( see Subsection 2.2 ) , and 3. we present a methodology ( using transformer language models ) that allows us to achieve positive results on our dataset . We previously evaluated several baseline NMT models ( such as Luong et al . ( 2017 ) ; Vaswani et al . ( 2017 ) and a plain character-based sequence-to-sequence model ) , which all failed to yield meaningful results due to our dataset being considerably smaller than is required for traditional NMT models.1 2 PRELIMINARIES . By disambiguating , we mean the task of transforming a sequence of symbols ( representing a mathematical formula ) into an abstract syntax tree and associating each leaf in the tree with a unique identifier specifying the precise semantics of the corresponding symbol . While this might superficially seem an easy task , closer consideration shows that even obvious seeming statements such as “ a+ b ” can in fact correspond to a multitude of possible disambiguations : a and b can be variables or previously defined constants , whereas + can represent e.g . addition on multiple different number spaces , generic ring or vector space operations , or string concatenation . In order to adequately disambiguate expressions generically , it is , therefore , necessary to take the context in which the expression occurs into account . 1All code and data relevant to this paper is available at https : //gl.kwarc.info/dmueller/ fifom . In this paper , we consider informal documents in LATEX specifically , which we will disambiguate with the sTEX package , using semantic identifiers provided by the SMGloM library . This eventually enables various formal knowledge management services ( such as type/proof checking ) provided by the MMT system . 2.1 STEX . Kohlhase proposed sTEX ( Kohlhase , 2008 ) , a package for annotating LATEX documents with structural and formal semantics which is today used by multiple groups formalizing mathematics in various systems . In particular , sTEX is based on OMDOC ( Kohlhase , 2006 ) , an extension of OpenMath ( Buswell et al. , 2004 ) which is foundation-agnostic in the sense that it does not favor a specific foundation ( such as type or set theories ) over any other . This approach is consequently best suited for semantifying informal documents , where foundations are often unspecified , left implicit or switched fluently . For example , category-theoretic and set-theoretic formulations are often used interchangeably in algebraic settings , whereas type theories are generally favored for computational aspects and formal systems . Figure 1 shows example sTEX macros and their usage in various stages . Relevant for this paper is primarily the \symdef command , which introduces a new mathematical concept ( e.g . \nattimes in Figure 1 ) . It takes as arguments a macro name ( e.g . nattimes ) , a symbolic notation ( last argument ) and optionally an OMDOC-name ( e.g . multiplication ) , arity ( e.g . [ 1 ] , which may be flexary ) and notational precedence ( e.g . p=600 , for automatic bracketing ) . It generates a unique identifier for the concept being declared ( based on the provided OMDOC-name ) , and a new LATEX macro ( e.g . \nattimes ) for referring to the symbol . Alternative notational variants for symbols can be introduced via \symvariant , which are used as options to the macro ( e.g . \nattimes [ cdot ] ) . In addition to being valid LATEX , compilable via pdflatex , sTEX-documents can be transformed to OMDOC using the LaTeXML-software ( Ginev et al. , 2011 ) , yielding a formally disambiguated representation of the document and in particular the symbolic expressions therein on the basis of the macros provided by \symdefs . LaTeXML also heuristically attempts to disambiguate nonsTEX-symbols , e.g . by considering “ = ” and “ + ” as infix notations for generic equality and addition operators , respectively . 2.2 SMGLOM . The SMGloM ( Kohlhase , 2014 ) , semantic multilingual glossary of mathematics ) is a library of hundreds of sTEX-modules containing mathematical concepts and definitions . It is separated into signature modules ( using the modsig-environment , see Figure 1 ) containing only symbol declarations , and natural language modules ( using the mhmodnl-environment , here exemplary for English ) that serve as dictionary entries for these , in which the semantics of the symbols are described in a semi-formal manner . The second row of Figure 1 shows an SMGLoM entry . 2.3 MMT . sTEX itself is integrated , and shares an underlying OMDOC ontology , with the MMT system ( Rabe & Kohlhase , 2013 ; Horozal et al. , 2012 ; Rabe , 2017 ) – a foundation-independent meta-framework and API for knowledge management services . This integration makes the generic services provided by MMT– e.g . type checking , library management/browsing , translation – available to informal mathematical texts . Using alignments ( Müller , 2019 ; Müller et al. , 2017 ) , OMDOC-expressions can be translated between different libraries , languages and foundations . This allows for e.g . translating ( originally ) sTEX-content to a typed setting in order to e.g . check expressions and run type inference . Additionally , several theorem prover libraries have been translated to OMDOC and integrated in the MMT system , e.g . Kohlhase et al . ( 2017b ) ; Müller et al . ( 2019 ) ( for a detailed overview , see Müller ( 2019 ) and Kohlhase & Rabe ( 2020 ) ) . Extending these integrations to enable exporting from MMT as well ( and in conjunction with natural language processing ) , this could enable verifying informal mathematics imported via sTEX using external state-of-the-art theorem prover systems . 3 STATE OF THE ART . Various papers over the last years have – explicitly or implicitly – attempted to extract formal information from informal documents using machine learning . These fall into two categories : Firstly , there are projects that attempt to fully formalize informal mathematical documents using machine learning techniques , using the surface language of some theorem prover system directly as a target . In Kaliszyk et al . ( 2017a ; 2015 ; 2014 ) , the Flyspeck project ( Hales et al. , 2017 ) – the formalization of Kepler ’ s theorem – was used as a basis for a parallel dataset in order to translate from informal mathematics to HOL Light ( Harrison , 1996 ) syntax . Kaliszyk et al . ( 2017b ) ; Wang et al . ( 2018 ; 2020 ) target the Mizar language ( Mizar ) instead , using the Journal of Formalized Mathematics ( JFM ) as data – an informal representation of the formal Mizar Mathematical Library ( Bancerek et al. , 2018 ) . While these projects achieved impressive results given the ambitious nature of the task , their success rate is naturally limited by the involved models having to solve several tasks at once ( see second observation in Section 1 ) , including ours . Additionally , by going to a fully formal language ( and logical foundation ) immediately , the result does not preserve the narrative presentation of the input document , effectively losing ( for us ) valuable information in the process . Consequently , our task and results obtained on it are not directly comparable to these projects . Secondly , various projects have aimed to solve informally presented mathematical problems of various kinds . These include Arai et al . ( 2014 ) ; Matsuzaki et al . ( 2014 ; 2017 ; 2018 ) on pre-university math problems , Saxton et al . ( 2019 ) and Lample & Charton ( 2019 ) on high-school level equations , Gan & Yu ( 2017 ) and Seo et al . ( 2015 ) on geometric problems , and Huang et al . ( 2018 ) and Wang et al . ( 2017 ) on solving typical high-school word problems . While this naturally entails disambiguating symbolic expressions , all these projects reduce their domain of applicability to specific areas where all occurring formal symbols are syntactically unambiguous – primarily common arithmetic operations , functions , and relations on real numbers – such that disambiguation reduces to simple parsing of a fixed , small set of a priori known symbols .

Summary: In more mathematical fields, theorem provers and similar systems can validate claims made about formal systems.  However, many research contributions come in the form of papers, and thus they are never validated in this way.  Math researchers can express their contributions in a special purpose language to do this, but that places an additional burden on them to learn this skill.