Disambiguating Symbolic Expressions in Informal Documents
Authors: Dennis Müller
Cezary Kaliszyk
Venue: ICLR 2021 Poster (2021)

Abstract: We propose the task of \emph{disambiguating} symbolic expressions in informal STEM documents in the form of \LaTeX files -- that is, determining their precise semantics and abstract syntax tree -- as a neural machine translation task. We discuss the distinct challenges involved and present a dataset with roughly 33,000 entries. We evaluated several baseline models on this dataset, which failed to yield even syntactically valid \LaTeX before overfitting. Consequently, we describe a methodology using a \emph{transformer} language model pre-trained on sources obtained from \url{arxiv.org}, which yields promising results despite the small size of the dataset. We evaluate our model using a plurality of dedicated techniques, taking syntax and semantics of symbolic expressions into account.

Published as a conference paper at ICLR 2021
DISAMBIGUATING SYMBOLIC EXPRESSIONS IN INFORMAL DOCUMENTS
Dennis Müller
Cezary Kaliszyk
Knowledge Representation and Management FAU Frlangen-Nürnberg
Computational Logic University of Innsbruck
Computational Logic
Institute of Computer science
University of Innsbruck
Warsaw University ce cezary wwl.skykkiibkbkk.k ac at
d. wwweeeereeeee.. wwwelerr@kwr...ing info
ABSTRACT
We propose the task of disambiguating symbolic expressions in informal STEM documents in the form of ыТTК files that is, determining their precise semantics and abstract syntax tree as neural machine translation task. We discuss the distinct challenges involved and present dataset with roughly 33,000 entries. We evaluated several baseline models on this dataset, which failed to yield even syntactically valid 15TEX before overfitting. Consequently, we describe a method- ology using a trunsformer language model pre-trained on sources obtained from arxiv org] which yields promising results despite the small size of the dataset. We evaluate our model using plurality of dedicated techniques, taking the syntax and semantics of symbolic expressions into account.
INTRODUCTION
Despite huge advancements in machine learning, the task of understanding informal reasoning is still beyond current methods. In fact, it became commonplace that humans annotate informal documents containing reasoning in many domains, e.g. law (Libal & Steen (020). Reasoning most visible in mathematical documents and software specification and as such in the last decades, the formalization of mathematical knowledge, and the verification of formal proofs, has become increasingly popular. By now, dozens of interactive and automated theorem prover systems are available. each providing librarics with up to hundreds of thousands of formalizations of mathematical definitions, theorems, and their proofs written by human mathematicians (Harrison al. 2014).
While formal methods are still primarily used by computer scientists (e.g. to verify software and hardware, as well as in program synthesis), by they have also drawn the interest of an increasing number of research mathematicians primarily thanks to famous problems such as Kepler's conjecture (Hales et al.) 2017) the classification theorem for finite simple groups (Solomon 1995). which have successfully been verified using theorem prover systems.
However, while some mathematicians have begun actively adapting formal methods for their work, there is prohibitively large discrepancy between the way new mathematical results are developed, presented, and published in mathematical practice, and the way they are formalized and implemented in formal systems (Kaliszyk& & Rabe 2020): Most theorem proving systems implement a fixed logical foundation (such as variants of set theory or various kinds of type theories), a surface syntax in which user declares new definitions and statements in terms of the underlying foundations, and either a tactic language or language for expressing proof terms (usually on basis of the Curry-Howard- correspondence in typed X-caleulus) that allow for declaring proofs. Consequently, the process of formalizing new content in a formal system resembles programming much more than does developing informal proofs.
This discrepancy results in severe challenges for traditional mathematicians: Formal systems are difficult to learn and use, even if one is well acquainted with the (informal) mathematics involved. They require learning dedicated formal languages resembling programming languages, declaring content on level of detail that is prohibitive for beginners even for "obvious" conclusions, and their

---

Published as a conference paper at ICLR 2021
libraries difficult to grasp without already being familiar with the system's language, conventions and functionalities. Due to the required level of detail, knowledge of the existing libraries is crucial when formalizing new content. Furthermore, many "intuitively valid" arguments can not be easily expressed terms of a logical foundation in the first place, and knowing how to deal with those requires familiarity with the logical foundation involved and lots of practice.
Consequently, the utility of formalizing mathematical results can be too easily (and too often is) dismissed in light of the additional time and work required for non-experts. This is despite the fact that many services available for formal mathematics are already enabled by semi-formal (or flexiformal) representations, such as semantic annotations in natural language texts, or formal representations containing opaque informal expressions (see e.g. Kohlhase (2013); Lange (2011a); [anuu (2017);
Kohlhase et al. 7a) Corneli Schubotz 2017): Dehaye et al. (2016). Therefore, we need to invest into methods for bridging the gap between informal mathematical practice and (semi-)formi mathematics. One way to do so is to investigate autoformalization, the task of (emiiautomaticalll)
converting existing informal mathematical presentations to (increasingly) formal representations.
Notably, these issues extend beyond pure mathematics to other STEM (science, technology, engi- neering and math) fields, where the formal verification (or lack thereof) of results can have direct real-world implications - examples include an infamous and costly error in the floating-poit point unit of Intel processors (Harrison) 2003) and several human failures to adequately convert between SI and imperial units, most famously in NASA's Mars orbiter (Grossman). In fact, the former has already established formal verification vital tool in hardware design (Harrison) 2003).
Two observations motivate the research presented here:
1. The vast majority of STEM researchers can be assumed to be comfortable with using BTE:: any integration of formal methods in ВTT development environment (e.g. via new packages or IDE integration) would consequently lower the entry barrier significantly.
2. The task of going from purely informal mathematical texts fully formal representations of the contained knowledge is best done via a separation of concerns, by focussing on individual subtasks (such as disambiguating symbolic expressions, parsing natural language, and translating it to formal foundation) using dedicated tools for each.
In this paper, we discuss specifically the task of disambiguating symbolic expressions- i.e. associating all symbols in an expression with their precise semantics L8TEX documents as a machine learning task, using s TEX semantically annotated L'TEX (Kohlhase) 2008 The contributions are threefold:
We discuss the details of disambiguating symbolic expressions informal STEM documents as a neural machine translation task, 2. we present new dataset specifically for this task, based on the existing SMGLoM library of s sTExX macros (see [Subsection 2.2). and 3. we present a methodology (using transformer language models) that allows to achieve positive results on our dataset. We previously evaluated several baseline NMT models (such as Luong et al. (2017): Vaswani al. (2017) and plain character-based sequence-to-sequence model), which all failed to yield meaningful results due to our dataset being considerably smaller than is required for traditional NMT models
PRELIMINARIES
By disambiguating, we mean the task of transforming a sequence of symbols (representing a ma- thematical formula) into abstract syntax tree and associating each leaf in the tree with a unique identifier specifying the precise semantics of the corresponding symbol.
While this might superficially seem an easy task, closer consideration shows that even obvious seeming statements such as "a+ D" can in fact correspond to multitude of possible disambiguations: and i can be variables or previously defined constants, whereas can represent e.g. addition multiple different number spaces, generic ring or vector space operations, or string concatenation. In order to adequately disambiguate expressions generically, it is, therefore, necessary to take the context in which the expression occurs into account.
All code and data relevant to this paper is available https:///ll http:///gl.wwar.... kwars. i info/ 1er 1er
Fi fom
2

---

Published as a conference paper at ICLR 2021
In this paper, consider informal documents in ITT specifically, which we will disambiguate with the sTEX package, using semantic identifiers provided by the SMGloM library. This eventually enables various formal knowledge management services (such as type/proof checking) provided by the MMT system.
2.1 STEX
Kohlhase proposed TTμę (Kohlhase 2008), a package for annotating KTІX documents with structural and formal semantics which is today used by multiple groups formalizing mathematics in various sys- tems. particular, s TTx is based on OMDoC (Kohlhase (006), an extension of OpenMath (Buswell et al. 2004) which is foundation-agnostii in the sense that it does not favor specific foundation (such as type or set theories) over any other. This approach is consequently best suited for semantifying informal documents, where foundations are often unspecified, left implicit or switched fluently. For example, thtegory- and set-theoretic formulations are often used interchangeably in algebraic settings, whereas type theories are generally favored for computational aspects and formal systems.
Figure shows example s sTEX macros and their usage in various stages. Relevant for this paper is primarily the \symdef command, which introduces new mathematical concept (e.g. Anat imes in Figure I). It takes as arguments a macro name (c.g. mattimes), a symbolic notation (last argument) and optionally
OMDoc-name (e.g. multipl caticoioon,, arity (e.g. [1], which may be flexary) and notational precedence (e.g. p-600, for automatic bracketing). It generates a unique identifier for the concept being declared (based on the provided OMDoC-name), and new ЦЛКК macro (e.g. Anatt imes) for referring to the symbol. Alternative notational variants for symbols can be introduced via esymarriant which are used as options the macro (e.g. Anattines [cdot]).
In addition to being valid 18TEX, compilable via pdf I at latex, s sTpX-docummets can be transformed to OMDoC using the LaTeXML-software (Ginev al., (011), yielding formally disambiguated representation of the document and in particular the symbolic expressions therein on the basis of the macros provided by \symdefs. LaTeXML also heuristically attempts to disambiguate non- TigX-symbols, e.g. by considering "=" and "+" as infix notations for generic equality and addition operators, respectively.
2.2 SMGLOM
The SMGloM (Kohlhase) 2014), semantic multilingual glossary of mathematics) is library of hundreds of s sTeX-module containing mathematical concepts and definitions. It is separated into signature modules (using the modsi mwwsii--nviioonennnee...- g-eviironment, seelFigure 1) containing only symbol declarations, and natural language modules (using the semmoddn-enviionment here exemplary for English) that serve as dictionary entries for these, in which the semantics of the symbols are described in semi-formal manner. The second row of Figure shoows an SMGLoM entry.
2.3 MMT
TEX itself is integrated, and shares an underlying OMDoC ontology, with the MMT system (Rabe) & Kohlhase 2013 Horozal et al. 2012 Rabe 2017 foundation-independeen meta- meta-framewort and API for knowledge management services. This integration makes the generic services provided by MMT- e.g. type checking, library management/browsing translation available to informal mathematical texts. Using alignments (Müller 2019; Müller et al.) (007)). oMDooc-xprrssioos can be translated between different libraries, languages and foundations. This allows for c.g. translating (originally) s TTpX-content to a typed setting in order to e.g. check expressions and run type inference.
Additionally, several theorem prover libraries have been translated OMDoC and integrated in the MMT system, e.g. Kohlhase et al. (2017b): Müller et al. (2019) (for a detailed overview, see Müller (2019) and Kohlhase & Rabe (2020). Extending these integrations to enable exporting from MMT as well (and in conjunction with natural language processing), this could enable verifying informal mathematics imported via s sTTx using external state-of-the-art theorem prover systems.
3

---

Published as a conference paper at ICLR 2021
s s'ix declarations
(signature module)
\syndef [pmpeeeefiaammmmt.....lLL.o....
( (1+)
(+1))
[ayrrariiant//aatt:me]]]]
[/assoc/p==00] (ttp:oo0o//aattttmssssosmo cdot | |11111
\end[medsig)
s sTEE references
(natarith)
(natural language module)
(bggiindefinition
\Defi (multiplicationn) stnattimesOp [cdot] computes the \defi (product) $\nattimes [cdot] (a,b}$ (also written as Shnatrimes(a, or $ Alnattimes [x] (a,b)$ of \trefiis (naturalnumberrs (natural) (number) $as and $b$. It is defined by the equations $'eq( (eggttimmeess [cdot] (x, 01, 0 and &eeq \natt [cdot] (x, \natsucc(y):: \natplus (x, \nattimes [cdot] [x,y))}$. eend ddefinitioon
lend (mhmodn1)
PDF output
Definition. Multiplication computes the product b (also written as ab b) of natural numbers and b. is defined by the equations 0 and x Syy x
(for the natural language module)
OMDoc
<OMB>
<CMS od="smgiommmyyeege
name="equal"/
<OMA>
<OMS cd cd-""amglom:arithheee ca ?hatariich
name= icctiion///
<OMV name="*"/>
<OMI>O</OMI>
</OVA>
<OMI>0</OMI>
</MM>>
Figure 1: An sTEX Example: The OMDoC corresponds the symbolic expression
3 STATE OF THE ART
Various papers over the last years have explicitly or implicitly attempted to extract formal information from informal documents using machine learning. These fall into two categories:
Firstly, there are projects that attempt to fully formalize informal mathematical documents using machine learning techniques, using the surface language of some theorem prover system directly as target. In Kaliszyk et al. 2001a 2015 2014). the Flyspeck project (Hales et al.) 2017) the formalization Kepler's 's theorem was used basis for a parallel dataset order to translate from informal mathematics to HOL Light (Harrison) 1996) syntax. Kaliszyk et al. (2017b); Wang et al.
(2018 2020) target the Mizar language (Mizar) instead, using the Journal of Formalized Mathemat- ics (JFM) as data an informal representation of the formal Mizar Mathematical Library (Bancerek) ct al. 2018).
While these projects achieved impressive results given the ambitious nature of the task, their success rate naturally limited by the involved models having to solve several tasks at once (see second observation in Section I). including ours. Additionally, by going logical foundation) immediately, the result does not preserve the narrative presentation of the input document, effectively losing (for us) valuable information in the process. Consequently, our task and results obtained on are not directly comparable to these projects.
fully formal language (and
Secondly, various projects have aimed solve informally presented mathematical problems of various kinds. These include Arai et al.] (2014); Matsuzaki et al. (2014) 2017: 2018) on pre-univessity math problems, [Saxton al. (209)) and Lample & Charton (209)) high-school level equations,
4

---

Published as a conference paper at ICLR 2021
Gan & Yu 2017) and Seo al. (2015) geometric problems, and Huang et al. (2018) and Wang et al. (2017) on solving typical high-schood school word problems.
While this naturally entails disambiguating symbolic expressions, all these projects reduce their domain of applicability to specific areas where all occurring formal symbols are syntactically unam- biguous primarily common arithmetic operations, functions, and relations on real numbers such that disambiguation reduces to simple parsing of fixed, small set priori known symbols.
TASK DEFINITION
Definition 4.1. (Disamiguation Task) Let L be set of HTEX fragments (i.e. strings), which we assume are syntactically valid BTEX in some suitable document context.
A symbolic expression is (for our purposes, simplified) any substring of some S E L such that is interpreted by the Tex-eeginee math mode e.g., if is delimited by S. $$ or VI \[and \J respectively,
For the purposes of our lask, we call S E C fully disambiguated, if every symbolic expression occurring in only consists of:
1. variable names (e.g. or \mathcal (G), provided they do not represent specific, definite mathematical objects),
2. STXX macros introduced via \symdef declaration in the SMGLoM, or
3. non-semantic commands characters, such additional spaces/tubs/lmebreaks. purely aesthetic spacing or kerning commands, unnecessary parentheses or clarifying comments (e.g. in under- overbraces).
Let Ceggx C. C the subset of fully disambiguated BTEX fragments. Conversely, let LanE C, be the set of BTęX fragments that do not contain any STEX macros
Clearly, for any S E L, there is some (TTX(S) C Lxrgr such that S and any S' E BTTX((S) represent the same symbolic presentation i.e. they generate the same output on pdElatex.
Conversely, assume that for any S E C there is set sTyX(S) C. Lanx such that 1. ETEX(S) ETEX(S')) for all S' E =TEX(S) (i.e. they have the same symbolic presentation) and 2. all S sTEX(S) capture the intended semantics of i.e. the author of S, were they know the SMGLoM library sufficiently well, would agree that S' is a correctly fully disambiguated variant of S.
Our goal is learn function : - L such that any S E L we have f(S) sTEX(S).
Example 4.1. Consider the sentence from the SMGIoM
multiplication $ $ledot computes the product $aledot be (also written Sabs or $a Saitmmes b$) of natural numbers $a$ and $bs.
The last two symbolic expressions ($a$ and $b$) only consist of variable names, and are thus considered fully disambiguated already.
The first one ($ (clotot refers to the multiplication operator on natural numbers, which in sTpX is represented as \nnt t imesOp, the remaining symbolic expressions are all multiplications on natural numbers applied to the variables a and b with different notations, represented in STEX via \nattimes with various options.
We expect the target function f on this input sentence to output
Multiplication $ $inattimesOph computes the product $lnattimes cdoot ba (aiso written as iinatiimesaayy b1$ or Slnattimes (a, b)$) of naturai numbers $a$ and $b$.
Note that Luxg and Lonx disjoint
5

---

Published as a conference paper at ICLR 2021
5 DATASETS
We have two datasets of s cooittc
1. The SMGLoM which introduces precisely those macros that we want be learned by a model. Unfortunately, it provides relatively few symbols and hence can only cover small part of informal documents even in theory. Additionally, apart from some rudimentary concepts such as logical connectives or basic arithmetic functions, the SMGLoM library references the majority of symbols only once (in the corresponding dictionary entry). This is unlike most other formal systems, where all symbols need to be typed or defined for- mally when being declared, which naturally leads to significant number of references to previously declared symbols.
2. The MiIKoMt@reppsiion of lecture notes by Michael Kohlhase (the author of s sTEX) is heavily biased towards subjects in computer science, covering only a small part of SMGLoM- entries, and often introducing local Naymdefs.
Notably, while the translation from source to target language is difficult, the reverse translation (from s TTEX plain (TTE)) is easy: Since sTEXX macros internally expand (ultimately) to the plain notational representation as basic 18TEX, translating from the targer to the source language amounts to merely expanding sTEx macros. This allows for easily generating parallel dataset from set of documents in the target language.
To obtain such a parallel corpus for supervised learning, we take the individual LTTeX- -flles in those repositories and do the following:
1. We separate the documents into small fragments of (on average) 500 character lengths, which we consider to be the sentences in LaTEx. Symbolic expressions occur preferably at the end of a sentence, based on the assumption that preceding text provides a more meaningful context for disambiguation. Sentences that do not contain symbolic expressions are ignored.
2. In each sentence S S.TEX E TE we perform some standardization function which e.g. removes non-semantic macros and ensures that macro arguments are always braced, in order to minimize author bias,
3. We extract all symbolic expressions (mang..)) dissns in S and expand all s sTeX macros in them, resulting in (masgx.))inns (where ns is the number of symbolic expressions in S). Analogously, we expand all sTip macros in S itself, yielding Stap Luppx.
Each entry in our dataset then consists of a 4-tuple (Swrgx Sengx (mamg,.)) (mengx))isng) In
total, we obtain 911 I entries from SMGLoM and 9200 entries from MiKoMH.
Synthesizing Training Data In order to augment our datasets for supervised learning, we opted to exploit the MMT integration to synthesize additional training data.
For that, we aligned SMGLoM symbols with declarations in strongly typed MMT archive; namely the Math-in-the-Middle (Mil) library (Müller 2019). This allows us to randomly generate well- typed (and hence syntactically well-formed) terms in typed setting, translate these along alignments to s aTTxX expressions and subsequently generate surrounding verbalizations.
The generating algorithm takes as input set of symbols Sym (e.g. all MitM-symbols for which an alignment to SMGLoM exists) and a starting symbol s E Sym (e.g. nat t time;; binary multiplication on natural numbers). It returns random well-typed formal expression t which is guaranteed to contain 8. Afterwards, detailed description of the algorithm is given in Appendix A).
verbalized as an s sTEX sentence using natural language fragments (a
The synthesized s TEX sentences are then treated as above to augment our parallel training corpus.
As an evaluation dataset, we developed s TEX documents based selected fragments of introductory sections from mathematics lecture notes; primarily containing basics such as set operations, number
https://// wttp:/uuub..ffooooooooo/ https:/// https:://gl..athhbb wwthhub...tt.......t info/Mi KoMH

---

Published as a conference paper at ICLR 2021
spaces, examples for proofs by induction, basic combinatorics, and definitions of common algebraic structures, containing 161 symbolic expressions in total. Importantly, these documents were written by hand, with a focus featuring multiple symbols with the same symbolic representation; primarily the usual arithmetic operations on different number spaces.
Of the 100 SMGLoM symbols used therein, 92 were aligned with corresponding symbols in the MitM library and used as input symbols for synthesizing sentences; with 250 sentences per starting symbol (as not drown out the non-synthesized sentences), yielding 23,000 additional sentences.
Unlike the training datasets, the evaluation document was translated to plain BTE manually using the PDF a reference, in order to avoid possible spurious patterns in automatically expanded s s'TEX.
S TEX -NNX-TNNGG WITH MACHINE LEARNING AS AN NMT TASK
In the course of our experiments, we considered our disambiguation task as a machine translation (NMT) problem, the models for which have been proven be quite effective even beyond natural language translations (Clark et al. 2020). In fact, the autoformalization projects mentiond in Section 3 which are spiritually closest to our task, all used NMT models with positive results. There are however several aspects that distinguish a 15TEX-to- to- sTEE translation from similar translation tasks which significantly affect the applicability of existing tools and hence our methodology.
First, Unlike the most popular formal systems, there is no large library of formalizations for the translation target. This leaves us with only a small dataset that (for the reasons outlined [Section 5) does not represent well the general distribution we would like to learn.
Second, translation is only relevant for specific fragments of an input text, namely the symbolic expressions; for the surrounding natural language texts, translation should be the identity. Never- theless, surrounding text usually contains critical information for disambiguation; e.g. without the surrounding context, it is impossible to disambiguate expression + b. since the symbol "+" could refer to any of dozens of addition operations.
Finally, depending on perspective, the domain language is proper subset of the target language; or rather (since we want to avoid ambiguous expressions in s TTEx) domain and target language share both a basic grammar well large amount of vocabulary (namely Luspxx O Lxpp) which e.g. subsumes natural English. For the domain language, large datasets are easily obtainable.
Our task could also be considered as text style transfer task e.g. Yang al. (2099) uses pre-trained language models for text style transfer, roughly similar to (but more sophisticated than) our approach. While the datasets used therein are still considerably larger than ours, this might be promising avenue for future improvements over our model.
7 METHODOLOGG
Notably, s TTEx macros reflect the syntax of expression, that on symbolic expressions alone, the representation of the target sequences is naturally analogous to those chosen in string-to-tres translations (Aharoni & Goldberg 2017). Plain LTTXX however naturally amenable tree- structured representation, making tree-to-tree approaches (Chen et al, (018)) not easily applicable to our dataset.
Initial experiments using standard, dedicated NMT models with full sentences as input/output quickly proved be ineffective due to the size of the training corpus, which was too small cause these models to even generate syntactically correct FTTX (e.g. knowing to balance pairs of brackets) before overfitting on the training data. This makes difficult to compare our approach to an informative baseline model.
Trunsformer language models (e.g. Devlin et al. 20018: : Liu et al. (2019); Radford (2018); Radford al. 2019); Clark et al. al. (2020)) allow us to leverage huge available corpora of plain DTTX documents to train a model to "understand" both basic ETEX syntax and mathematical terminology. Using those, consequently do not need to rely our small dataset for this base-level understanding. We can then approach learning s sTEX annotations as a downstream task on a pre-trained transformer model. Consequently, we pre-trained GPT2 (Radford al. 2019) model large portion of available
7

---

Published as a conference paper at ICLR 2021
LTEX sources of scientific papers from the preprint repository arxiv org (6,673,950 entries of length 1,024 tokens). The model was trained from scratch in order to use a dedicated tokenizer trained on LTEX directly (byte-level tokenizer; vocabulary size 32,000) rather than natural language alone.
In order to leverage the pretrained model for both source and target languagd to fine-tune the GPT2-model on inputs of the form
we subsequently opted
Spmp 

mixE
ma 
,
where 
 a single-token separator 0 For example, for Figure the training data contains fragments (normalized) such as:
Multiplicatinn
&'odot& computes the product $a\odot bs (also written as $ab$ or $altimes b$) of natural numbers $a$ and $b$.

 $a ealedot b$ 
 $\nattimes ledot fa, b 

We then use text generation on inputs of the form SIATE <a> M 
 for translating and stop generating after encountering 
>
By using one entry per symbolic expression, we obtain dataset of 121,368 examples. The GPT2- model was finetuned on these for five epochs, resulting in average training loss of 0.04 and yielding promising results on the evaluation set (see below). This approach has the following advantages:
1. It allows for using large datasets of generic 1STEX documents to learn basic syntactic rules and semantics of mathematical expressions beyond our small sTTX datasets.
2. We conjecture that this approach makes the model less sensitive spurious patterns in the synthesized part of our dataset.
3. Adding new symbols to the SMGLoM and aligning them to (new or existent) symbols the MitM library allows for immediately synthesizing training data, obviating the need to first obtain large amounts of data using the new symbol before the model can learn to use it.
4. The mere pretrained GPT2 model can be trained on additional downstream tasks, e.g. introducing macros for referencing mathematical concepts in natural language fragments.
EVALUATION AND RESULTS
The traditional evaluation metrics (loss during evaluation, perplexity, BLEU) are somewhat difficult and/or meaningless to apply in our situation, since 1. the returned tokens and provided label tokens might differ semantically irrelevant ways (e.g. $a+bs vs. $a b&), and 2. loss/perplexity would be evaluated during forward pass in a next token prediction task on token-by-token basis, which would retroactively "correct" errors in prediction that would otherwise yield completely wrong result.
Consequently, opted for plurality of evaluation strategies. Let SF the returned sentence of our model on an input SEEE with the correct label S.tg. Then on our evaluation set we get
Sp E L for 96.9% of inputs
2. SWIX E (TT for 64.0% of inputs,
3. Sp E CeTg for 60.2% of inputs, and
4. SF S.Tx for 47.2% of inputs.
In comparison, using traditional NMT models auch as Luong et al. 20 Vaswani et al, (2117) we effectively obtained 0% success rates for all of the above. Additional evaluation techniques exploiting the MMT integration are described in AAppendix B
Figure shows few examples where our model "failed" in interesting ways. As the first and fourth examples show, the model seems to consistently fail to replace "=" by the intended macro \eq a failure that LaTeXMI. can recover when converting to OMDoc, but also regularly occurs in the training data. Similarly, Nidots often leads to wrong translations: The first example shows that the
$ Initial experiment with the pretrained model as encoder component only showed improvements over randomly initialized mncdeer-decoder-nnneees hut ultimately proved unsuitable still due to the small dataset size. 'inspired
by
http:// 3 ammar r.gg thub. o rustrated-gee 2
#part-33 3--e---b-dd--ee- 1anguage model
8

---

Published as a conference paper at ICLR 2021
Suigi
3 \
Sang:
Vegr/VWatuurllllbmerrrs (0,
Sp:
Sungi
|at (\ferall xiin A. xlin B)
(A \subseteq B) hugaaasteeegbllleeeeiihigaaaaa (N
(x) (8)))
SF:
Abi [on
Suggi
mattisaallff//A/====11////>sssss AM
Sang:
(A)))
Sp:
[afeec/|hoooereeeraaaaaaasttttss (x) (x) A}}
Sungi
L#2+5+445--(5ccc0066///1111 /2-15
Sange
Neql 1etp: \natciv |1 (nattimes(cdod)
15, 15,6)]12),,5)
SF:
3, (Anatplus| (Anattimes [cdctt (2)-15
(5,6), 5, (2)--115
Figure 2: Example Inputs and Outputs from our Evaluation Set
model simply dropped \1dots, using a generic set constructor macro \set rather than \setdots, the one specifically intended for sets ending in ellipses.
In the second example, the model seems to introduce a nonsensical additional argument for the Aforal maero. Notably, the expression Yr E A.P can also be achieved using the dedicated macro Foorllsss (A) (P). Seemingly, the model chose the macro foorl,, and the arguments for the \forals macro, yielding a wrong translation that generates wrong pdf output, while being "semantically almost correct""
In the third example, the model confuses the macro /setst (for set comprehension) with a more complex macro \bsetst (for comprehension with complex pattern the left side). Addition- ally, it confuses \sseteq (for inclusive subsets x S A) with \sset (for generic subsets x c A), duplicating the first argument and moving the intended argument outside the scope of the macro.
Example four interesting in that the model correctly identifies the arithmetic operations as those on the natural numbers, but spuriously inserts additive term \natplus 4, 5;; this is likely an artifact from the left-hand side the equation. Interestingly, these kinds of artifacts occur more than in our evaluation set.
CONCLUSION
We have proposed the task of disambiguating symbolic expressions in informal STEM documents and defined this task formally. This allows for annotating informal documents semantically, and further processing them using tools that support such annotated documents (e.g. MMT). We discussed the specificity of this task and what separates this task from other NMT problems. We developed dataset for this task and presented an approach that yields promising results, especially in light the size of the dataset. In particular, the presented approach points to the efficacy of using transformer models pretrained on generic ESTEX documents.
In the future, we plan to combine the proposed symbolic disambiguation approach with auto- formalization framework. This way we aim achieve better results for end-to-end formalization of informal mathematical documents. Furthermore, more promising results for the currently proposed task could be obtained by reintegrating the proposed models into an encoder-decoder NMT model.
ACKNOWLEDGMENTS
The first author and this work were supported by a postdoc fellowship of the German Academic Exchange Service (DAAD).
The second author is supported by ERC starting grant no. 714034 SMART

---

Published as a conference paper at ICLR 2021
REFERENCES
Roce Aharoni and Yoav Goldberg. Towards string-to- teinngto-trrc tree neural machine translation, 2017.
Noriko H. Arai, Takuya Matsuzaki, Hidenao Iwane, and Hirokazu Anai. Mathematics by machine. In Proceedings of the 39th International Symposium on Symbolic and Algebraic Computation, ISSAC
'14, pp. 1-8, New York, NY, USA, 2014. ACM. ISBN 978 4503 2501-1. doi: 10.1145/2608628. 2627488. URL /tt.a///////..//...../.....///// 2627448
acm. org w
Grzegorz Bancerek, Czeslaw Bylinski, Adam Grabowski, Artur Kornilowicz, Roman Matuszewski, Adam Naumowicz, and Karol Pak. The role of the Mizar Mathematical Library for interactive proof
development in Mizar. Autom. Reasoning, 61(1-4):9-322 2018. doi: 10. 10.1007//10817--017-9940 URL httphhttt/////ttt oor.i..org/// 1007 01111100011100011111111111 004----1-------
S. Buswell, O. Caprotti, D. Carlisle, M. Dewar, M. Gaetano, and M. Kohlhase. The Open Math Standard, Version 2.0. Technical report, The Open Math Society, 2004. See http:://ww
openmath. org www.rrr//taaaadn......o
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation, 2018.
Kevin Clark, Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders discriminators rather than generators. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL https: http::::///peeeeieeew. mnet/forummiin id= 11..-=r1xxx1B
Joe Corneli and Moritz Schubotz. math. math..ikipedia.org A vision for collaborative semi-formal, language independent math(s) encyclopedia. In A/TP 2017. The Second Conference on Artificial Intelligence and Theorem Proving, pp. 28-31. 2017.
Paul-Olivier Dehaye, Mihnea laneu, Michael Kohlhase, Alexander Konovalov, Samuel Lelièvre, Dennis Müller, Markus Pfeiffer, Florian Rabe, Nicolas M. Thiéry, and Tom Wiesing. Interoper- ability in the OpenDreamKit project: The math-in-the-middle approach. In Michael Kohlhase, Moa Johansson, Bruce Miller, Leonardo de Moura, and Frank Tompa (eds.), Intelligent Com- puter Mathematics 2016, number 9791 in LNAL. Springer, 2016. ISBN 378-3-319-084331 www.oo/ommaatrreammm
URL httl:htttss://qthhuu com/ http:///weeeeeeeeeee/////eeeeeee CICM2016/publicee
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
httt:://axxiv.orr/abbo..
.04005
Wenbin Gan and Xinguo Yu. Automatic understanding and formalization of natural language geometry problems using symtax-semantics models. 2017. doi: 10.24507/ijicic.14.00 URL
http::/www. http://wwww.iiilcc.cr///////////
140016.
Herman Geuvers, Mathhew England, Osman Hasan, Florian Rabe, and Ol (eds.). Intelligent Computer Mathematics, number 10383 in LNAI, 2017. Springer. ISBN 978-3-319-6207499 doi:
10.1007//97-33319920 3-117777---1199-7---66
Deyan Ginev, Heinrich Stamerjohanns, Bruce Miller, and Michael Kohlhase. The latexml daemon: Editable math on the collaborative web. In James H. Davenport, William M. Farmer, Josef Urban, and Florian Rabe (eds.), Intelligent Computer Mathematics 18th Symposium, Calculemus 2011, and 10th International Conference, MKM 2011, Bertinoro, Italy, July 18-23, 201 Proceedings, volume 6824 of Lecture Notes in Computer Science, pp. 292 294, Springer, 2011. doi: 10.1007/ 978-3-642-22673-1 25. URL Stte:://ooll.oogg0000000/7733333
25
Lisa Grossman. Metric math mistake muffed mars metcorology mission. com/ /2//201000 2010/11/111 Omars-c? www.mammmtt imate observer-
10

---

Published as a conference paper at ICLR 2021
Thomas C. Hales, Mark Adams, Gertrud Bauer, Dat Tat Dang, John Harrison, Truong Le Hoang, Cezary Kaliszyk, Victor Magron, Sean McLaughlin, Thang Tat Nguyen, Truong Quang Nguyen, Tobias Nipkow, Steven Obua, Joseph Pleso, Jason M. Rute, Alexey Solovyev, An Hoai Thi Ta, Trung Nam Tran, Diep Thi Trieu, Josef Urban, Ky Khac Vu, and Roland Zumkeller. A formal proof of the Kepler conjecture. Forum of Mathematics, Pi, 5, 2017. doi: 10. 10.1017/fmp.201..1
J. Harrison. HOL Light: A Tutorial Introduction. In Proceedings of the First International Conference on Formal Methods in Computer-Aided Design, pp. (65- Springer, 1996.
J. Harrison. Formal verification Intel. In 18th Annual IEEE Symposium of Logic in Computer Science, 2003. Proceedings. pp. 45-54, June 2003. doi: 10. 109/ 200..1204411114
John Harrison, Josef Urban, and Freek Wiedijk. History of interactive theorem proving. In Jörg H. Siekmann (ed.), Computational Logic, volume 9 of Handbook of the History of Logic, pp. 135-214. Elsevier, 2014. doi:
URL
1016/B977 0- 444 4- 12211624-422 50004-6
F. Horozal, M. Kohlhase, and F. Rabe. Extending MKM Formats at the Statement Level. In Campbell, J. Carette, G. Dos Reis, J. Jeuring. Sojka, V. Sorge, and M. Wenzel (eds.), Intelligent Computer Mathematics, pp. 64 Springer, 2012.
Danqing Huang, Jing Liu, Chin- Yew Lin, and Jian Yin. Neural math word problem solver with reinforcement learning. In Proceedings of the 27th International Conference Computational Lin- guistics, pp. 213-223, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL URL httpt::t////tt wwww://wwwwaaa/// ac htw..//wwbboggaatthooog//////3/ 1018
Mihnea lancu. Towards Flexiformal Mathematiccs. PhD thesis, Jacobs University, Bremen, Germany, 2017. URL htttes//.ooan...oe....iieeeeerer
index
docId/721
JFM. Journal of formalized mathematics. http::/wwwwmizzr.crgg
Cezary Kaliszyk and Florian Rabe. survey of languages for formalizing mathematics, 2020. URL
https://arxiv org/ abs 2005. 12876
Cezary Kaliszyk, Josef Urban, Jiff Vyskočil, and Herman Geuvers. Developing corpus- based translation methods between informal and formal mathematics: Project description. In Stephen M. Watt, James H. Davenport, Alan P. Sexton, Petr Sojka, and Josef Urban (eds.), Intelligent Computer Mathematics, pp. 435-439. Cham, 2014. Springer International Publishing. ISBN 978-3-319- 08434-3.
Cezary Kaliszyk, Josef Urban, and Jifí Vyskočil. Learning to parse on aligned corpora (rough diamond). In Christian Urban and Xingyuan Zhang (eds.), Interactive Theorem Proving, pp. 227-233. Cham, 2015. Springer International Publishing. ISBN 978-3-319-22102.1
Cezary Kaliszyk, Josef Urban, and Jifí Vyskočil. Automating formalization by statistical and semantic parsing of mathematics. In Mauricio Ayala-Rincón and César A. Muñoz (eds.). Interactive Theorem Proving, pp- 12-27, Cham, 2017a. 7a. Springer International Publishing. ISBN 978-3-319-66107-0.
Cezary Kaliszyk, Josef Urban, and Jifí Vyskočil. System description: ized Mizar formulas. In Tudor Jebelean, Viorel Negru, Dana Petcu, Daniela Zaharie, Tetsuo Ida, and Stephen M. Watt (eds.). 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2017, pp. 169-172. IEEE Computer Society, 2017b
Statistical parsing of informal-
M. Kohlhase. OMDoc: An Open Markup Format for Mathematical Documents (Version 1.2). Number 4180 in Lecture Notes in Artificial Intelligence. Springer, 2006.
M. Kohlhase. Using LTEX as a Semantic Markup Format. Mathematics in Computer Science, 2(2): 279-304, 2008.
Michael Kohlhase. The flexiformalist manifesto. In Andrei Voronkov, Viorel Negru, Tetsuo Ida, Tudor Jebelean, Dana Petcu, Stephen M. Watt, and Daniela Zaharie (eds.), 14th International Workshop Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2012), pp. 30- 36. Timisoara, Romania, 2013. IEEE Press. ISBN 978- 100----663--02--- URL http: //wwark info//oh hnto:///onlbaaaeeppper////gggg

---

Published as a conference paper at ICLR 2021
Michael Kohlhase. A data model and encoding for semantic, multilingual terminology of mathemat- ics. In Stephan Watt, James Davenport, Alan Sexton, Petr Sojka, and Josef Urban (eds.), Intelligent Computer Mathematics 2014, number 8543 in LNCS. pp. 169-183. Springer. 2014. ISBN 978-3- 319-08433-6. WRRlhttp:/// htt////warc... info/ www.eohheeeeeeeee.. papers cm1 4 smg om pdf
Michael Kohlhase and Florian Rabe, Experiences from exporting major proof assistant libraries. 2020. URL
hwwrraabe/eeeaarcccceeee http://aK_oaaaeee__eeeeaa 2000_220000 pddf
Michael Kohlhase, Thomas Koprucki, Dennis Müller, and Karsten Tabelow. Mathematical models as research data via flexiformal theory graphs. In Geuvers et al (2017). ISBN 978-3-319-6277
9. doi: 10.
URL httt::/kwwatee info
ci cml 7 pdf
Michael Kohlhase, Dennis Müller, Sam Owre, and Florian Rabe. Making PVS accessible to generic services by interpretation in universal format. In Mauricio Ayala-Rincón and César A. Muñoz (eds.), Interactive Theorem Proving, volume 10499 of LNCS. Springer. 2017b. ISBN 978-3-319- 66107-0. BRLhhtp::/// httt:://wwrr:: www.wwww....oooo..o....ooo kohlhase submit/itpt... pvs. pdff
Guillaume Lample and François Charton. Deep learning for symbolic mathematics, 2019.
Christoph Lange. Enabling Collaboration on Semiformal Mathematical Knowledge by Semantic Web Integration. PhD thesis, Jacobs University Bremen, 201 la. URL http:: poctps://svee /ttpa::///vv..et
info/repos/st Latoo//epos//viin//////////////s påf
Also available as a book Lange (2011b).
Christoph Lange. Enabling Collaboration on Semiformal Mathematical Knowledge by Seman- tic Web Integration. Number 11 in Studies on the Semantic Web. AKA Verlag and IOS Press, Heidelberg and Amsterdam, 201 ISBN 978-1-60750-840-33 URL http:// www
semantic-web-studiee. net.
Tomer Libal and Alexander Steen. Towards an executable methodology for the formalization of legal texts. In Mehdi Dastani, Huimin Dong, and Leon van der Torre (eds.), Logic and Argumentation Third International Conference, CLAR 2020, Hangzhou, China, April 6-9, 2020, Proceedings, volume 12061 of Lecture Notes in Computer Science, pp. 151-165. Springer, 2020. doi: 1001/ 978-3-030-44638-3 10. URL /Etps://doi.orr 10 1011000000001111000-0-0--0--10-0 /tt.a//////A///////////////////0
10
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, fbs//197..1192 2019, /ttp/ttp://axx.
/ww.b////999...0 11692
Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. Neural machine translation (seq2seq) tutorial. https://githuu..oom/eee cowps///scotttessooo/o/tttt 2017.
Takuya Matsuzaki, Hidenao Iwane, Hirokazu Anai, and Noriko H. Arai. The most uncreative exami- nee: A first step toward wide coverage natural language math problem solving. In Proceedings of the Twenty-Eightt AAAI Conference on Artificial Intelligence, AAAI'44, pp. 1098 1104 AAAI 2894044
Press. 2014. URL http: / //l. dl. acm. org/ citation. www.:imddmannae.....m
Takuya Matsuzaki, Takumi Ito, Hidenao Iwane, Hirokazu Anai, and Noriko H. Arai. Semantic parsing of pre-university math problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2131-2141, Vancouver, Canada, July 2017, Association for Computational Linguistics. doi: 10. 10.18653/v1/P17-1190 URL
https: www. wwwwww.ww.....cccc org/ http:://www..cccccec..oo//////t.
Takuya Matsuzaki, Hidenao Iwane, Munchiro Kobayashi, Yiyang Zhan, Ryoya Fukasaku, Jumma Kudo, Hirokazu Anai, and Noriko H. Arai. Can an a.i. win medal in the mathematical olympiad? benchmarking mechanized mathematics on mneiveiery problems. Al Commun., 31:251 2018.
Mizar. Mizar. http::/ http:/wwww.iz.......... org. 1973-2006. URL hittt::/www.. mi zar org
Dennis Müller. Mathematical Knowledge Management Across Formal Libraries. PhD thesis. Informatics, FAU Selangen-Nümbeery 10 2019. URL brtps://kwwzc.
dmuellerr Bmueeller/pubs/thes
12

---

Published as a conference paper at ICLR 2021
Dennis Müller, Thibault Gauthier, Cezary Kaliszyk. Michael Kohlhase, and Florian Rabe. Classifica- tion of aligmments between concepts of formal mathematical systems. In Geuvers et al. (011)) ISBN 978-3-319-620744 doi: 10. 01177997788311--20000 koh 1 chase/papers ci cm 1 7-a gnment s. pd
URL
info/
Dennis Müller, Florian Rabe, and Claudio Sacerdoti Coen. The Coq Library as a Theory Graph. accepted at CICM 2019, 2019.
Rabe and M. Kohlhase. A Scalable Module System. Information and Computation, 230(1):1-54 2013.
Florian Rabe. How Identify, Translate, and Combine Logics? Journal of Logic and Computation, 27(6) 2017.
Alec Radford training. 2018.
Improving
language
understanding by generative pre-
URL
wtw.:uaaauuuneeetnnnnnniiiioo by Generative hateerattiiieaaarrrrrrrooo Radford/ cd1 oal 8000ffe006641110332222333000
Smproving-Langueee
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. CoRR, abs/1904.01557, 2019. URL Btl.httpp//arxxv.ooo
0bss/9044 01557
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1466-1476 Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18655/1//D15-1117 URI. https http:ee...rraaggge wtt..//aaaaoooooooll.oo.....
Ron Solomon. On finite simple groups and their classification. Notices of the AMS, pp. 231-239, February 1995.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, 2017. URL
Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR,
Qingxiang Wang, Cezary Kaliszyk, and Josef Urban. First experiments with neural translation of informal to formal mathematics. In Florian Rabe, William M. Farmer, Grant O. Passmore, and Abdou Youssef (eds.), 11th International Conference on Intelligent Computer Mathematics (CICM 2018), volume 11006 of LNCS, pp. 255 270.. Springer, 2018 doi: 10. 10.1007/978-3-311 3-1-7988--31999811--- 22. URL httptttss:////t 0ttp://////o1.rrr/00011000000000 0000007977833
968 -2
Qingxiang Wang, Chad E. Brown, Cezary Kaliszyk, and Josef Urban. Exploration of neural machine translation in autoformalization of mathematics in Mizar. In Jasmin Blanchette and Catalin Hritcu (eds.). Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs. CPP 2020. New Orleans, LA, USA, Jamuary 20-21, 2020, pp. 85 98. ACM, 2020. doi: URL htt.sttp:: / doi org 0. 1 145 337 237333885555 3373827
Yan Wang. Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845 854, Copenhagen, Denmark. September 2017. Association for Computational Linguistics. doi: 10. 1118655///DDD11110
URL wttpt:ttt///t...t. www. a http://wwwww......../aaooooooo//
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P. Xing, and Taylor Berg-Kirkpatrick. Unsupervised text style transfer using language models as discriminators, 2019.
13

---

Published as a conference paper at ICLR 2021
A SYNTHESIZING TRAINING DATA
The generating algorithmm takes as input set of symbols Sym (e.g. all MitM-symbols for which an alignment SMGLoM exists) and starting symbol s Sym (e.g. nat time;; binary multiplication on natural numbers). The algorithm then proceeds as follows:
1. If 8 T has (simple or dependent) function type, we fill in the required arguments. For 8 =natt ines, our type is =Nat-Nat -Nat, hence we need to find two arguments $1, $2 of type Nat. For each s of required type T, proceed follows:
(a) With probability Pear, we introduce new variable v: Ti from list of allowed variable names (which include variants such as a, a', ag etc.) and let 8; := == v.
(b) With probability Pfun. we pick symbol f E Sym with function type with return type T, (e.g. for T. =Nat, we can pick natplus). In that case, we let and set as the result.
recurse,
(c) With probability Pronst type T, (e.g. for T, =Nat we can pick 0) and return 8; == c.
Peor Pfun. pick constant symbol Sym of
In order to avoid stack overflows, we reduce pnun each iteration by a certain factor 1. As not overuse certain symbols, we scale f and Pronst with the number of respectively suitable symbols available; if Sym contains no suitable function or constant symbols, we let Pfun 0 (and/or Pconst 0, respectively).
2. If s T does not have function type (or all its parameters have been filled in 1.), then is Pup.
well-typed and we return s with probability
With probability Pup, we instead pick a new symbol sy E S of some function type such that some i-th parameter type sy T. In that case, we let 8; := s and s == sy and recurse. Again, in order to avoid stack overflows we reduce Pup by some factor with each iteration.
The algorithm also takes subtyping into account, e.g. whenever a term of type Real required, terms of type Int or Nat are used with some probability.
In order to obtain a sentence in the sense of 3 providing context for disambiguation, we first translate t along alignments to SMGLoM (using a random Nsymvarian's collect the set V of all free variables of and verbalize their types. For that, associate each type with set of verbalizations from which we choose randomly to produce a sentence that introduces the variables before using them in the generated expression. |Figure 3) shows a few example verbalizations for variable x of type Nat and generated sentences for the input symbol =real =realuminus: uminus; the negation on real numbers.
The verbalizations are categorized as prefixed (e.g. "a natural number n²") or suffixed (e.g. "n a natural number"), and singular or plural, and picked according the number of variables of the same type and the surrounding sentence, which is also picked at random (e.g. "Assume we have..." uses prefixed, whereas "Let... uses suffixed).
B EVALUATION TACTICS
For every 1TTE input SiSTx, expected label S&TEx and returned sentence SR, we employ the following strategies, the results of which are summarized in Figure 4
islatex We parse Sg into AST. Success implies that Sg is syntactically valid LТEX. This might fail for "minor" reasons such as missing closing bracket. might yield false positives in cases where macros (not explicitly considered by our parser) occurring in Sn have wrong number of arguments.
All subsequent evaluation strategies require islatex to succeed.
stexcheck We heuristically check whether Sa is in La unlike islatex, this requires that all s sTEX macros occurring in Są have the right number of arguments. Success does not tell us that the input has been disambiguated correctly, but does imply that is has been disambiguated at all. False negatives occur if Sn (and thus likely Saape well)
14

---

Published as a conference paper at ICLR 2021
Generated sTk:
PDF output
Verbalizations
V\inset [x] NatuuralNumbers a positive integer $x$
E N
a positive integer 2
integer $ slintmethan(st (038
integer 2 0
a natural number 9x8
a natural number x
Sentences
Assume we have some $linset Naturaalummbrrs] and arbitrary Aiinset (\maathal F} (HNlnceggeruumbrrs It follows that S\realuming nus Arealum (Ninttimes \matthall F,y", ,y' 1116.
Assume we have some Y EN and arbitrary F E Z. It fol- (F x y 1/).
slows that nus
Let $inatmorethan ( Then consic consicebettoo > 0. Then consider $ $\realuminus /realuminus Anatsucol \natsuce n) JS.
S(S(n)).
whenever we have some positive nature Whenever we have some posi- number $tvarepsilons, any integer Veetive natural number s, any in- and a real number $\livar (mathhall C diteger and real number C2. (2)9, then it follows that $\realtime it follows that C2C2(-E C)12), \livar \mathca
(2), Arealplus Arealuminiu \ell), \natsuca( (varepsilon] ))
Figure 3: Example Verbalizations for x :Nat and Generated Sentences
contains complex variable names, or if Sn contains e-g. an equality symbol "=" instead of the corresponding s sTE:X macro, which LaTeXMM. could recover.
evall 1 latex All sTEX macros occurring in SR expanded and SR is normalized as described in Section 5 The result is trring-compared to SYTęX Success thus implies, that the notational presentation PDF output of SKiE and SR will coincide. False negatives can occur due minor differences e.g. in not strictly necessary brackets.
omdoc Sa is translated to OMDoc using LaTeXML and imported to MMT. Success guarantees syntactic formeedeeddess 5R- Since both the LaTeXML-OMDos export and the subsequent MMT-import somewhat brittle, this easily lead to false negatives.
cranslated The import from omdoc is translated to the typed MitM library. This entails that all symbols used in Sg are aligned with MitM symbols and Sr is amenable for formal knowledge management services.
inferred
The translation to MitM obtained from translates is type checked by MMT by having its type inferred. Success guarantees that Sy is well-typed.
Notably, if SR is a mere variable (e.g. the expression Sn$), it does not actually have an inferrable type, but succeeds trivially. This accounts for 60 of the entries in our evaluation set, i.c. 37%.
provided_sted stex Both the expected label Sagx and Sg normalized and string-compared. Success implies that Sy is definitely the correct translation. False negatives can easily occur due non-semantic differences between Sapx and Sn however, such as bracketing, nested applications in Sn (e.g. $ $Anatpl us ('natpluss (a, b), c) vs. $ $Anatpl uus (a, b, aj s),
etc.
stex_as stex_as_emdoc SaTEX translated to OMDoC via LaTeXML and directly compared to the OMDoC-term obtained from omdoc. Like provided stex, success implies that Sş is correct, but more fault-tolerant with respect to the precise syntax of Sn. while being less fault tolerant due to the issues mentioned in omdoc.
The first three evaluations always be applied; from the remaining, all but provided stex require a working installation of LaTeXML and its sTpX-Plugin. The last two require a known correct translation.
15

---

Published as a conference paper at ICLR 2021
Total imputs
161
islatex
96.9%
stexcheck
60.2%
eval_latex
64.0 %
omdoc
76.4%
translated
63.5%
inferred
59.6%
provided_ster
47.2 %
stex as ____oddo
53.4 %
Figure 4: Results on our Evaluation Document
detailed log file on our evaluation document with the individual results for each input and evaluation available in the associated git repository.
16


Reviews:
[{"review_id": "p1YmeznyLg3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new task, disambiguating an informal math expression in LATEX by associating its tokens with concepts in a predefined formal math library and determining its abstract syntax tree. As argued in the paper, I agree that this task could serve as an important step for autoformalization, which is one of the most important problems of formal reasoning. \n\nThe task setup is reasonable. LATEX is commonly acceptable to be the informal language for editing math expressions. STEX and SMGLoM are powerful tools to annotate LATEX expressions with formal concepts. By advancing on this problem, we can greatly reduce the workload of autoformalization. \n\nThe drawback of the current benchmark is the lack of training and evaluation data. I think the lack of training corpora may be addressed by pretraining and building synthetic data. We do need a larger and high-quality evaluation set to validate any actual progress on this problem. The current evaluation set is too small and covers limited math topics. Also, the evaluation protocol is quite unclear. From what I understand, our best evaluation protocol should be checking if S_F belongs to STEX(S_STEX), which is not used in this work. Is there a way to implement this protocol? \n\nThe proposed approach looks fine. It is better to have an ablation study on the corresponding contributions of pretraining and synthetic data.\n\nIn general, I think this paper proposes an important task. By building a larger evaluation set and figuring out a clear evaluation protocol, this could be an important benchmark for the AI/TP community. \n\n=======================================================================\nAfter reading other reviews and authors' responses, I upgrade my score to 6. Despite its relatively small evaluation data, I think the setup of the task of autoformalization could still contribute to the community and inspire more researchers to make efforts in this direction. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "iUH6-XPB-ba", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a dataset for autoformalization (semantic\ndisambiguation) of informal Latex STEM documents. It is based on the\nconsiderable amount of work that has been done in the last decade on\nflexiformal (semi-formal) language formats and tools such as OMDoc,\nOpenMath, sTeX and LaTeXML. The SMGloM glossary and the MiKoMH\nrepository are used as parallel sources, and the MMT system connecting\na number of formal systems and foundations is used for data\naugmentation.\n\nThese are still relatively small datasets, so custom pretraining of\nGPT-2 is done on the full arxiv corpus. The pretrained model is then\nfine-tuned on the smaller training data.  Multiple evaluation metrics\nthat are meaningful in the semantic setting are defined - some of them\nsimilar to those used in Wang et al 18 and Wang et al 20.\n\nThe final success rate of 47.2% of test data predicted correctly looks\nvery good and is comparable with the results of Wang18/Wang20 on the\nsynthetic data obtained by informalizing Mizar.\n\nMy overall impression is that this is an important step in the\nautoformalization program [1]. It has involved a lot of work and brought\nin a range of important tools developed recently.\n\n\nSome detailed remarks:\n\np5: Disamiguation ==> spell check\n\np5: def 4.1 \"We call S ∈ L fully disambiguated\"\n==>\nI would not call the text fully disambiguated without types of\nvariables. In systems with subtypes (e.g., Mizar, possibly also other\nPAs with typeclasses) the meaning and provability of a statement\n(e.g., \"forall x exists y st x = y *_complex y\") will change depending\non whether the quantification is over complex, real, rational, integer\nor natural numbers.\n\n\np7: Question: S_F = S_sTEX means exact string equality or after white space normalization, etc? If so can you say what are exactly the normalizations and what is the success rate before and after them?\n\n- Would larger GPT models help?\n\n- Would unsupervised learning like in Wang 20 be useful in some context here? The unsupervised methods seem to have improved a lot recently.\n\nReferences:\n\n[1] Cezary Kaliszyk, Josef Urban, Jirí Vyskocil, Herman Geuvers:\nDeveloping Corpus-Based Translation Methods between Informal and Formal Mathematics: Project Description. CICM 2014: 435-439\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RaS8YFh7eqh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\n\nIn more mathematical fields, theorem provers and similar systems can validate claims made about formal systems.  However, many research contributions come in the form of papers, and thus they are never validated in this way.  Math researchers can express their contributions in a special purpose language to do this, but that places an additional burden on them to learn this skill.\n\nAn alternative would be to \"translate\" the research into formal languages which could be operated on by automated systems.  This seems to be the goal of this paper, which looks at using translation to disambiguate some expressions from STEM documents written in LaTeX, which maps it into an sTex document.  Previous research in this area used more hand-specified transformations, and was evaluated on different data.  This makes this work closely related to existing work, but not directly comparable.\n\nThe main finding of this work is that pre-trained transformer models outperform more traditional fully-supervised translation systems on this task.  It is difficult to guage precisely to what extent the proposed method solves the task, or to fully grasp what aspect of the translation problem is being solved.\n\n\n\n#### Strong points: \n\nThe proposed approach of using transformers and large in-domain pre-training is similar to a lot of recent work which has shown to work well in practice, and is therefore well-motivated.  The task itself is important and improvements in this area could have broad range of impact that would even be good to the ML field itself, so even a practical improvement using fairly standard ML would be a useful contribution.\n\nThe authors are clearly knowledgable on the topic, and discuss and cite a great deal of the literature and libraries relevant to the problem.  It's a heavy paper -- there's a very extensive set of work that's been studied and referenced.\n\n\n#### Weak Points\n\nIn the start of section 4, a number of systems were listed.  Each of these was an attempt to automate the formalization process, but there was no attempt to compare against these methods.\n\nAs noted towards the end of section 4, the target transformation for much of the document is the identity.  Given that actually a lot of this text should not change, is phrasing it as translation the best choice?  It seems that phrasing it this way sets up the NMT baselines to perform poorly, since a lot of training is necessary to prime the model to learn this identity transformation, and that data is not given to those models.\n\nThe evaluation methodology is confusing.  For instance, it seems some of the data is generated via an automated procedure, both in the supervised learning (Section 5) and in the Synthesizing Training Data section.  It makes it difficult as a reader to understand why this is not a chicken-and-egg type of scenario: if automated methods produce the dataset, which is then used to train the model, then why are those methods not sufficient for the end task?  This may be a problem that arises from introducing so many domain-specific libraries and formalisims, that it leaves the reader with a great deal of difficult to understand precisely what the transformation is accomplishing and from what type of data.\n\nIt was also bizarre in the results section how the baselines were dismissed in writing, their results were never presented.  If the baselines are truly that bad, then do they suffice as baselines?  The authors choose these instead of the existing formalization methods, so why make the contrast with methods that are not in a position to perform the task well?\n\n\n#### Recommendation\n\nBecause the presentation makes it difficult to fully grasp the problem setting, precisely what is being learned, precisely what is failing, it is difficult to recommend the paper for acceptance.  It is actually very understandable that this particular paper has this problem, because the authors are forced to introduce many unfamiliar concepts -- the problem setting, the types of formalisms used, the libraries used in creating the data, etc.  These are all things that are outside the scope of the typical ICLR paper and thus warrant a clear introduction, but space is limited.  I could easily imagine this paper filling up 12-14 pages just with the same content presented here.  But ultimately the paper is not written in a way that can properly convey the scope of the work and narrow in on precisely the targetted problem and why it's difficult and important.\n\nThen the experimental section is quite short and lacks important comparisons.  Given the lack of suitable baselines, I would not be able to recommend accepting the paper without real comparisons to other work in this area.  Again this could be a space concern, but the paper overall spends too much time leading up to methodology/experiments, and then is very light on actual experimental content.  Factor in that the model is used in a very off-the-shelf way, and doesn't treat the problem setting really any different than a standard translation task, it is hard to see real novelty in the modeling contribution either.\n\nOverall I think the work is promising, but it is far too rough in its current state to be considered for acceptance without significant revision.  It would need major restructing and refocusing, more experiments, and more analysis.\n\n\n#### Presentation\n\nI feel like there's a lot of domain specific meanings to terminology that makes it more difficult than necessary to understand by a general ML audience.  Take for instance, formal and informal.  To most language users, a scientific paper is a formal document -- it uses formal language.  So it takes me some time as a reader to get into the actual data section and understand truly what is meant by informal here.  There are many things of this nature that would be better to clarify up-front, so the reader with the typical ML background and biases doesn't carry around incorrect concepts of what the paper is about, for longer than is necessary.\n\n\nThe citation format is incorrect.\n\nSmall typos throughout.\n\n\n#### In considering author response:\n\nThank you to the authors for continuing discussion on the points raised in my review, and for further clarifying the nature of the data as a kind of unidirectional ambiguity problem.   I understand this better now and can see a contribution in releasing this data / data-generating process for other researchers studying autoformalization.  On account of this I'm going to raise my initial scoring.\n\nOn the subject of methodology, I still think there are reasons to reconsider this work.  As discussed, the translation baselines were not great.  I think it's not really fair to compare those models without pre-training on data that was too small to learn basic tree properties.  It is possible that translation models that perform string-to-tree translation would perform better here(1), though results from natural language translation would hint towards the pre-trained models still performing better.  Translation models used in the domain of programs seem more suitable as well, and there's a good number of these, and there is a natural desire to generate strings that reflect a properly nested tree (2).  There is also work on mapping strings to knowledgebase queries that seems similar in input/output (IIRC, Luke Zettlemoyer's had a number of important papers in this line).\n\nBut at best these would still be comparisons of mostly off-the-shelf translation models, which doesn't leave the reader with much of a takeaway.\n\nSo I'm left feeling that if the authors want a useful quantitative comparison, these methods should be explored.  Pre-trained model beats model trained on only in-domain data is not to me a story significant enough to warrant inclusion in the conference, even if it contributes a new dataset (as the modeling is presented as a contribution here).  Even off-the-shelf methods can of course be part of an important contribution when the authors show that they have pushed the field further with an important result (say, GPT) but I do not feel the evaluation in this case supports that conclusion.\n\nIt seems more natural given that none of these methods are likely to out-perform a vanilla pre-trained translation model, that the problem description and qualitative evaluation are of the utmost importance.  I would really recommend expanding this beyond half a page, to give the reader a better idea of what problems are solved and what are remaining.  It also seems that some of the errors pointed out (like those involving ellipses) would likely be remedied by additional synthetic data.  As I'm the most dissenting reviewer, I would still hope the authors attempt to improve the results section with the additional page upon acceptance.\n\n1.\nTowards String-To-Tree Neural Machine Translation\nRoee Aharoni, Yoav Goldberg\n\n2.\nTree-to-tree Neural Networks for Program\nTranslation\nXinyun Chen, Chang Liu, Dawn Song\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KNDWBvF4u8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses a variant of the style transfer problem - that is, transferring formal latex expressions to less formal descriptions that can be followed by mathematician.  \n\nThis is a paper that formulates a new task, provides a dataset for the task and tests initial approaches for solving it. It is important to take this paper type into account when reviewing it  - I am not expecting a very creative solution, or very strong results at this stage. What is important to me when reviewing such a paper is to see an accurate and interesting problem definition, an appropriate dataset, modeling and experiments that demonstrate the challenge of problem and of the evaluation (if evaluation is indeed challenging) and proper awareness of previous work.\n\nIt will be most straight forward for me to review the paper by listing its pros and cons.\n\nStrong points:\n\n1. The problem exposition is through and clear, and the introduction surely provides good motivation for the problem.\n\n2. It is clear that the authors are expert on the subject matter. That is, they are deeply familiar with the problem and with directly related previous word (that is, previous work that addressed this very problem or close variants).\n\n3. The authors propose a new dataset that is likely to be useful for the community of researchers that work on this problem.\n\n4. The paper proposes an algorithmic approach for the problem, tests it in experiments with the new dataset and the authors are aware of potential challenges in the evaluation and try to address them.\n\nWeak points:\n\n1. The problem definition was not clear to me. I surely understand the general idea but I am missing a concrete example that demonstrates what exactly an algorithm for the problem gets as input and what is its output.\n\n2. I had a similar problem with the description of the dataset. Yes, there is a formal description (just as there is a formal description of the task), but the lack of examples leaves the description at a very abstract level - I could not understand what exactly should be expected in the dataset.\n\nI should note that 1+2 makes it harder to evaluate the results and to evaluate the appropriateness of the evaluation.\n\n3. The authors does not show awareness of work in semantic parsing and in style transfer.  These works are very important both for the algorithmic approach and for understanding the challenges of evaluation (e.g. plurality).  For example, there is lots of semantic parsing research on transferring text into SQL queries (the opposite direction of the current problem) or on solving textual mathematical problems. Can any ideas be borrowed from this literature or is it only vanilla MT that can be applied ? As said above, I am aware this is a paper that introduces a task, but part of this introduction should be, I believe, the connection to relevant ideas and approaches. \n\nOverall, despite the challenges, I think the paper can contribute an interesting and practical new task to the research community - both at the task definition level and in providing an actual dataset. I recommend that the authors try to solve the above issues in the final version, but I am leaning towards acceptance.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
This paper tackles the task of translating informal LaTeX
math into a formal representation annotated with abstract concepts (sTeX /
SMGloM).  The authors build a synthetic training data generation mechanism,
and construct an evaluation dataset by hand. The problem is tackled as machine
translation, and vanilla systems fail, while GPT-2 pretrained on LaTeX documents
performs well. The reviewers recognize the importance of this work, in an area
where data is not plentiful and benchmarking is difficult.  The authors do a
good job in presenting a difficult topic rather clearly, but I would encourage
the authors to continue improving the presentation, possibly with clearer examples
or figures.  The particular "copying bias" useful in this task, pointed out by a
reviewer, is indeed interesting and I encourage the authors to consider that
discussion and the thoughtful reviews deeply. Overall, this is a significant
contribution to the field and I recommend acceptance.
Decision: Accept (Poster)