MioGatto: A Math Identifier-oriented Grounding
Annotation Tool
Takuto Asakura1 , Yusuke Miyao1 , Akiko Aizawa1,2 and Michael Kohlhase3
1

The University of Tokyo, Tokyo, Japan
National Institute of Informatics, Tokyo, Japan3
FAU Erlangen-Nürnberg, Erlangen, Germany
2

Abstract

We present a new annotation tool, called MioGatto, to efficiently build large corpora for grounding mathformulae. While in documents in science, technology, engineering, and mathematics, math identifiers
can be used in multiple meanings in a single document, corpora with annotated coreference relations
between identifiers are crucial for the grounding task. Using MioGatto, annotators can produce a list ofmath concepts for each document, associate one of the math concepts with each occurrence of mathidentifiers, and annotate the text span that is the source for grounding. In general, manual annotation of
coreference relations is a very tough task, but this tool is specialized for building grounding corpora
and can annotate them more efficiently than existing general-purpose annotation tools. The tool can beobtained from https://github.com/wtsnjp/MioGatto.

1. Introduction
Recently, the authors have proposed a mathematical language processing (MLP) task called
grounding of formulae [1], which has both aspects of math description alignment [2] andcoreference analysis. In order to create a resource that can be used as training and evaluation
data for this grounding task, we need an annotation tool that can annotate (1) a description
label to each math identifier in formulae, and retain (2) information about coreference relations
between math identifiers. In addition, (3) spans of text that serve as sources of grounding,i.e., natural language phrases that can be regarded as mathematical definitions and declarations,need to be annotated (Figure 1). Not surprisingly, there is no existing tool that can efficiently
perform all such annotations simultaneously.
In order to efficiently create linguistic resources for the grounding tasks, we developed anovel annotation tool that has all the necessary functions. The tool is named the Math Identifieroriented Grounding Annotation Tool (MioGatto). The core functionality of MioGatto is toannotate each math identifier with a math concept and to annotate sources of grounding, wherea math concept is a description of an identifier with some extra information such as arity andmath type. It has a web-based graphical user interface (GUI) that allows users to efficiently
annotate math concepts and source of grounding with visual and intuitive operations (Figure 2;
details will be presented in Section 3). This GUI can be seen as a tool for visualizing the annotateddata, not just for information assignment. For example, in MioGatto, if an annotator mouses
13th MathUI Workshop 2021, July 26–31, 2021, Timisoara, Romania (online)© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR

Workshop
Proceedings

http://ceur-ws.org
ISSN 1613-0073

CEUR Workshop Proceedings (CEUR-WS.org)
In general, the integral of a real-valued function 𝑓 (𝑥)with respect to a real variable 𝑥 on an interval [𝑎, 𝑏] iswritten as
∫ 𝑏𝑎𝑓 (𝑥) 𝑑𝑥.

(Definition of integral in Wikipedia)

Math Concepts
Description :a real-valued function

Description :a real variable

Math Type :function

Math Type :real number

Arity :
1

Arity :
0

Figure 1: Math concepts and sources of grounding. The example sentence is taken from Wikipedia1 .

over a math identifier that is annotated with a math concept, the corresponding description
pops up. In order to visualize a coreference relation, math identifiers referring to the same
math concept are decorated with the same color. Sources of grounding are also highlighted
in the same color as the corresponding math concept. It has been shown that this kind ofcomplementary information to formulae can indeed help a reader understand the content ofscientific papers [3]. During the development of the annotation tool, we will continuously
discuss how to visualize annotated data. The findings will guide the development of a GUI thatwill be useful in supporting readers of the science, technology, engineering, and mathematics(STEM) literature in the future.
Linguistic resources containing math formulae annotated with rich information are the cornerstone for developing various MLP techniques, such as mathematical objects of interest (MOI)
analysis [4], math information retrieval (MathIR), and formula search. In natural language
processing (NLP), language resources that contain plenty of texts annotated with rich additional
information are important for learning and evaluating statistical models. Specifically, information such as parts-of-speech, dependency structures, and coreference relations is requiredat the word, phrase, and sentence levels. Similar information including part-of-math tags [5],1https://en.wikipedia.org/wiki/Integral

math types [6], dependency structures, and coreference relations at symbol and formula levels
is useful for MLP. A key contribution of our tool is to build such a large corpus that will help
advance the MLP technology.
Generally speaking, a large amount of time and economic cost is required to add suchannotation information manually. Especially when annotating formulae, one needs to ask
experts from various STEM disciplines who do not necessarily have a linguistic background.
Therefore, in order to build a large amount of annotated formulae with high accuracy, it is crucial
to have an efficient annotation tool that is easy enough to be used. Within the field of NLP,
annotation tools have been developed that are easy and efficient to use for a range of purposes
and tasks, e.g., brat [7] and WebAnno [8]. Some of these tools claim to be general-purpose, andif they are general enough, they can be applied to annotate math formulae as well. There arealso a small number of tools that are specialized for annotating STEM documents, e.g., KAT [9].
Nevertheless, compared to the wide variety of annotation tools for creating corpora for NLP, thechoice of tools that can add linguistic annotations to math formulae is very limited. Especially,
it is impractical to annotate coreference relations on a large scale without using a tool withdedicated features. MioGatto is a tool that addresses this problem.

2. Related Work
There is commercial software available for the general public, which provide basic annotation
functionality, such as adding free-text annotations or highlighting text spans in documents,
e.g., Adobe Acrobat2 for PDF documents and hypothes.is3 for web pages. However, they are
not designed to create linguistic resources, and do not support the annotations necessary for
language processing, such as dependencies and coreference relations between words. Therefore,
many specialized tools have been developed to efficiently create annotated corpora [11, 12, 7,
13, 14, 15], and these tools are preferred to be used to build language resources. These tools
typically support the annotation of two main types of information: text spans and relations
between regular text spans. The ability to label text spans with part-of-speech tags and whether
they are proper expressions or not is also common. Most of these annotation tools accept XMLor other text data as input, but there are also tools, e.g., PDFAnno [14], that can annotate PDFs
directly. Such a tool is useful for scientific papers, where it is difficult to distinguish tables,
equations, footnotes, etc., from the main text.
Since building an annotated corpus is a time-consuming and laborious task, the efficiency isimportant, and some tools are designed to build large corpora at a high speed. For example,
WebAnno [13] has extensive functionalities for project and user management, which makes it
easy for multiple annotators to work together. There is also research that attempts to makethe annotation more efficient by focusing on specific types of annotations. SACR [15] is a
specialized tool for annotating coreference relations, which compares multiple annotation UIs
and adopts the most efficient one in its design.
In general, annotation tools for creating linguistic resources naturally annotate words, phrases,
and sentences, and do not have special support for math formulae. In some cases, features
2
3

https://acrobat.adobe.com
https://web.hypothes.is

for text-span annotation can be used to annotate math formulae. However, structures suchas superscripts, subscripts, and operators in formulae do not exist in natural language, andthe functions that assume such structures can be used for more efficient annotation. A small
number of tools were developed that specialize in annotating STEM literature with mathformulae. KAT [9] is a web-based annotation tool that is specialized for annotating STEM
documents. This tool allows annotators to effectively add attributes for the OMDoc format [16]to the STEM documents. Its annotation output is expressed in RDF, and thus can be used
in a universal way. AnnoMathTeX [17, 18] is another annotation tool that specializes inannotating math identifiers. The tool takes either a Wikitext or a LATEX document as input

Figure 2: The screen of MioGatto. This is a captured image of annotating an arXiv paper in the fieldof machine learning [10]. The basic annotation operations, such as selecting the math concept thateach identifier refers to, are performed in a sidebar on the right. Each occurrence of an identifier iscolored according to the annotated math concept. In other words, identifiers with the same color have a
coreference relation. Grounding sources are also highlighted in colors that correspond to math concepts
associated with them. When the mouse is over an annotated occurrence, the tooltip with the descriptionof the corresponding math concept is shown.

and annotates formulae in LATEX syntax rather than the rendered result. Notably, the tool
has the ability to recommend candidate math concepts to annotate for each identifier basedon four resources (arXiv, Wikipedia, Wikidata, and the surrounding text). MioGatto treats
all annotations as local annotations, assuming that the meanings of math identifiers change
frequently within a document, while AnnoMathTeX treats them as document-global annotations
unless a ‘local’ option is specified. This allows efficient annotation of documents whose meaningsof identifiers do not change frequently. All these tools have been developed with different
tasks and philosophies in mind. Since manual annotation is an arduous task, it is desirable touse a dedicated tool for efficient corpus building, and thus we needed one that is aimed at theformulae grounding task.

3. MioGatto: the Annotation Tool
Math Identifier-oriented Grounding Annotation Tool (Figure 2) is a tool specialized for annotating math identifiers. It is open source software and distributed under the terms of the MIT
license. It was developed to construct a dataset for solving the grounding task for math formulae.
It also has the ability to annotate text spans to aid in automating grounding of formulae, but
unlike KAT, it does not aim to annotate the structure of all elements in a STEM document.
The goal of the grounding task is to disambiguate the meaning of math identifiers in a document, as an identifier can have multiple meanings in a document and its scope is ambiguous [1].
The existence of ambiguity in the meaning of an identifier in a document means that two
occurrences of an identifier in a document may or may not refer to the same math concept.
Therefore, in the training and evaluation data for the grounding task, the coreference relation of
all occurrences of identifiers must be made explicit. An annotation tool such as those that give afree-text description to each identifier is not appropriate for this purpose. Since a math concept
can be represented by many different natural language texts, extracting coreference relations
from such annotated data would require solving the difficult task of determining whether two
descriptions represent the same math concept. It is not efficient for a human annotator tocarefully annotate every occurrence of an identifier referring to the same math concept withexactly the same description.
Instead of giving a free-text description directly to each occurrence of an identifier, MioGatto
associates each occurrence with an item in a pre-defined list of math concepts. Therefore,
it is easy to see that occurrences of identifiers associated with the same math concept havea coreference relationship. We call the pre-defined list of math concepts the math concept
dictionary. The dictionary is not a global ontology, but a document-specific one. Apart from thedescription of a math concept, each dictionary item can have several additional attributes, suchas arity, math type, and notation usage patterns (i.e., information whether the identifier is usedwith other tokens such as superscripts or independently). Moreover, annotators can register textspans that are useful in identifying math concepts, which are referred to by math identifiers assources of grounding. Most sources of grounding collected in this way correspond to definitions
and declarations. In the future, we intend to use the sources of grounding to automaticallyextract them and dynamically generate a math concept dictionary for each document.

Figure 4: The button to add a source of grounding.

XHTML

Annotation data
MioGatto
ServerJSON

Client(Web Browser)
Annotator

Figure 3: A dialog to add a math concept. Figure 5: The architecture of MioGatto.

3.1. User Interface and Annotation Procedure
Any annotation supported by MioGatto can be done by performing intuitive operations on a
web browser, without having any expertise in constructing language resources. Figure 2 showsa basic screen on MioGatto. The left side is the body of the academic paper to be annotated, andthe right side is the sidebar for the MioGatto operation. Annotators can select the identifiers
and text spans they want to annotate, while reading the article shown on the left. Annotators
can then add the necessary information to the document by manipulating the boxes in the
sidebar and the dialogs that appear as appropriate.
The annotator must first select one occurrence of a math identifier for each annotation. On
the occurrence that is selected, a pointer is shown in a document shown on the left side, and the“Concept” box on the right sidebar shows the information and buttons necessary to annotate
the occurrence (in Figure 2, the occurrence of identifier 𝒟 in the first line of Subsection III-A
is selected). In this state, one can either select the concept to which the occurrence refers
from the list of math concepts displayed in the “Concept” box (if any in the dictionary), or
create a new math concept in the dictionary. When an annotator chooses to add a new mathconcept, the dialog pops up with a web form to enter the required information (Figure 3). In
this form, the annotator will be asked to input information such as a free-text description
and arity. Once an occurrence of a math identifier is annotated with a math concept, theconcept’s information, most notably the description, is displayed as a tooltip when an annotator
mouse over an occurrence. In addition, the annotated occurrences are colored according to thecorresponding math concept, so that the coreference relation is visible to annotators.
MioGatto can also be used to annotate sources of grounding, text spans that are the basis
for the grounding. After selecting a math identifier that a math concept has already beenannotated, dragging the appropriate text span to select it will display a button for adding thesource (Figure 4). If the annotator clicked the button, the text span will be associated with themath concept corresponding to the selected occurrence of the identifier. Most of the sourcesannotated in this way correspond to definitions or declarations in mathematical terms. Within
papers in mathematics, the sources are often fixed phrases, such as “Let 𝑥 be something”, whereas
in the engineering literature, they are often simply apposition nouns. In the latter case, there isno one-to-one relation between math concepts and the sources of grounding, since the sources
corresponding to the same math concept appear many times within the same document. Hencethe annotation scheme allows annotators to annotate an arbitrary number of sources for a mathconcept. Similar to the occurrences of math identifiers, the text spans annotated as sources ofgrounding are highlighted in the color corresponding to the math concept associated with them.

3.2. Architecture and Implementation
MioGatto is a web-based annotation tool, and its entire implementation makes use of a variety
of web standard technologies. The input of MioGatto is XHTML documents converted by
LATEXML [19] from LATEX sources. To be more specific, the input must have the same additional
information as the XHTML contained in the arXMLiv dataset [20, 21, 22]. In XHTML generated
in this way, math formulae are written in MathML format [23], which stores more structural
information inside formulae than mere image data. In order to make use of MathML, a browserthat supports MathML rendering needs to be used. Firefox4 supports MathML among the majorbrowsers today. Each math identifier, i.e., <mi> element, has a unique ID in the input XHTML,and a MioGatto annotation is associated with the ID of the math identifier. The annotation
information is saved and output in JSON format. For the detailed specification of the output
JSON, please refer to the bundled documentation5 .
MioGatto employs a simple server-client model in terms of implementation. Figure 5 shows
the architecture of MioGatto in brief. The server, implemented in Python, loads the input
XHTML and stores the annotation data in JSON format. It also performs a simple preprocessing
to display the input and validate the annotation data before storing them. In contrast, the client,
implemented in TypeScript, is only responsible for handling UI. Such an architecture naturally
scales up in the future, where a single central server will manage the annotation data and many
annotators will annotate concurrently via the Internet.

4. Conclusion & Future WorkIn this paper, we presented MioGatto, a dedicated tool for building datasets for the groundingtask. For each occurrence of an identifier, a math concept can be annotated, and the textual
spans of the sources of grounding can also be associated with the math concept. Compared toother tools dedicated to MLP, MioGatto is unique in its ability to associate math concepts withadditional information such as arity and sources of grounding. This tool is also distinctive inthat it assumes that the meaning of an identifier switches frequently; we have used an earlyversion of this tool to annotate 937 math identifier occurrences for a scientific paper, and havefound that semantic transitions do indeed occur frequently [1]. All the existing data we built
are available from the SIGMathLing repository6 .
4https://www.mozilla.org/firefox/
https://github.com/wtsnjp/MioGatto/wiki
6
https://sigmathling.kwarc.info/resources/grounding-dataset/5

We are now using MioGatto to annotate STEM documents with annotators from a range of
disciplines, including information science, algebra, logic, and physics. Once we have a sufficientamount of identifier annotations with clear coreference relations, we begin to automate theprocess of the grounding task. We will continue to improve MioGatto so that it can be used byexperts across a variety of domains to build the annotated dataset more efficiently. MioGatto
will have a review mode so that discrepancies between annotators can be clearly shown with
the GUI. It would also be valuable if comments can be added to the annotations, so that multiple
annotators can discuss which annotation is better. Output format standardization should also be
considered. We also obtain specific feedback from the annotators and verify that the annotated
information helps the reader to read academic papers. In addition, we will explore how todisplay such additional information more effectively.
Acknowledgements
This work has been supported by JST, ACT-X Grant Number JPMJAX2002, Japan. We appreciate
Mr. Taiga Ishii for his bug reports and feedbacks on the tool. We would like to thank Mr. André
Greiner-Petter and Mr. Jan Frederik Schaefer for fruitful discussions.

References[1] T. Asakura, A. Greiner-Petter, A. Aizawa, Y. Miyao, Towards grounding of formulae, in:Proceedings of the First Workshop on Scholarly Document Processing, 2020, pp. 138–147.
doi:10.18653/v1/2020.sdp-1.16.
[2] M. Alexeeva, R. Sharp, M. A. Valenzuela-Escárcega, J. Kadowaki, A. Pyarelal, C. Morrison,
MathAlign: Linking formula identifiers to their contextual natural language descriptions,in: Proceedings of the 12th Language Resources and Evaluation Conference (LREC 2020),
2020, pp. 2204–2212. URL: https://aclanthology.org/2020.lrec-1.269.
[3] A. Head, K. Lo, D. Kang, R. Fok, S. Skjonsberg, D. S. Weld, M. A. Hearst, Augmenting
Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols,
in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI 2021), 2021, pp. 1–18. doi:10.1145/3411764.3445648.
[4] A. Greiner-Petter, M. Schubotz, F. Müller, C. Breitinger, H. S. Cohl, A. Aizawa, B. Gipp,
Discovering mathematical objects of interest—a study of mathematical notations, in:Proceedings of The Web Conference 2020 (WWW 2020), 2020, pp. 1445–1456. doi:10.
1145/3366423.3380218.
[5] A. Youssef, Part-of-math tagging and applications, in: Proceedings of 10th International Conference on Intelligent Computer Mathematics (CICM 2017), 2017. doi:10.1007/
978-3-319-62075-6_25.
[6] Y. Stathopoulos, S. Baker, M. Rei, S. Teufel, Variable typing: Assigning meaning to variables
in mathematical text, in: Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies(NAACL HLT 2018), 2018, pp. 303–312. doi:10.17863/CAM.30845.
[7] P. Stenetorp, S. Pyysalo, G. Topić, T. Ohta, S. Ananiadou, J. Tsujii, brat: a web-based
tool for NLP-assisted text annotation, in: Proceedings of the Demonstrations at the 13th

Conference of the European Chapter of the Association for Computational Linguistics(EACL 2012), 2012, pp. 102–107. URL: https://aclanthology.org/E12-2021.
[8] R. Eckart de Castilho, É. Mújdricza-Maydt, S. M. Yimam, S. Hartmann, I. Gurevych, A. Frank,
C. Biemann, A web-based tool for the integrated annotation of semantic and syntactic
structures, in: Proceedings of the Workshop on Language Technology Resources andTools for Digital Humanities (LT4DH), 2016, pp. 76–84. URL: https://www.aclweb.org/
anthology/W16-4011.
[9] D. Ginev, S. Lal, M. Kohlhase, T. Wiesing, KAT: an annotation tool for STEM
documents,
in: Mathematical user interfaces workshop at CICM, 2015. URL:
http://www.cermat.org/events/MathUI/15/proceedings/Lal-Kohlhase-Ginev_
KAT_annotations_MathUI_15.pdf.

[10] O. Simeone, A very brief introduction to machine learning with applications to communication systems, IEEE Transactions on Cognitive Communications and Networking (2018).
doi:10.1109/TCCN.2018.2881442.
[11] C. Müller, M. Strube, Multi-level annotation of linguistic data with MMAX2, in: Corpus
Technology and Language Pedagogy: New Resources, New Tools, New Methods, 2006, pp.
197–214.
[12] K. Bontcheva, H. Cunningham, I. Roberts, V. Tablan, et al., Web-based collaborative corpus
annotation: Requirements and a framework implementation, New Challenges for NLP
Frameworks (2010) 20–27.
[13] S. M. Yimam, I. Gurevych, R. E. de Castilho, C. Biemann, WebAnno: A flexible, web-based
and visually supported system for distributed annotations, in: Proceedings of the 51st
Annual Meeting of the Association for Computational Linguistics: System Demonstrations,2013, pp. 1–6.
[14] H. Shindo, Y. Munesada, Y. Matsumoto, PDFAnno: a web-based linguistic annotation
tool for pdf documents, in: Proceedings of the Eleventh International Conference on
Language Resources and Evaluation (LREC 2018), 2018, pp. 1082–1086. URL: https://aclanthology.org/L18-1175.
[15] B. Oberle, SACR: A drag-and-drop based tool for coreference annotation, in: Proceedingsof the Eleventh International Conference on Language Resources and Evaluation (LREC2018), 2018. URL: https://aclanthology.org/L18-1059.
[16] M. Kohlhase, OMDoc—An Open Markup Format for Mathematical Documents [version1.2], 2006.
[17] P. Scharpf, I. Mackerracher, M. Schubotz, J. Beel, C. Breitinger, B. Gipp, Annomathtex—aformula identifier annotation recommender system for stem documents, in: Proceedingsof the 13th ACM Conference on Recommender Systems, 2019, pp. 532–533. doi:10.1145/
3298689.3347042.
[18] P. Scharpf, M. Schubotz, B. Gipp, Fast linking of mathematical wikidata entities in wikipedia
articles using annotation recommendation, in: Companion Proceedings of the Web
Conference 2021, 2021, pp. 602–609. doi:10.1145/3442442.3452348.
[19] B. Miller, LATEXML The Manual—A LATEX to XML/HTML/MathML Converter, Version 0.8.3,
2018. URL: https://dlmf.nist.gov/LaTeXML/.
[20] H. Stamerjohanns, M. Kohlhase, D. Ginev, C. David, B. Miller, Transforming large collections of scientific publications to xml, Mathematics in Computer Science (2010).

doi:10.1007/s11786-010-0024-7.
[21] D. Ginev, H. Stamerjohanns, B. R. Miller, M. Kohlhase, The LATEXML daemon: Editable
math on the collaborative web, in: Intelligent Computer Mathematics, 2011. doi:10.1007/
978-3-642-22673-1_25.
[22] D. Ginev, arxmliv:08.2018 dataset, an html5 conversion of arxiv.org, 2018. URL: https://sigmathling.kwarc.info/resources/arxmliv/, sIGMathLing.
[23] R. Ausbrooks, S. Buswell, D. Carlisle, G. Chavchanidze, S. Dalmas, S. Devitt, A. Diaz,
S. Dooley, R. Hunter, P. Ion, M. Kohlhase, Mathematical Markup Language (MathML) 3.0
Specification, 2014. URL: https://www.w3.org/TR/MathML3/.