What Is Needed for Intra-document Disambiguation of Math Identiﬁers?
Takuto Asakura and Yusuke Miyao, University of Tokyo @ LREC-COLING2024, Turin, Italy
Long-term Goal: P2C Conversion

Usage of Multi-Layer Perceptron (MLP)

STEM Documents (Natural Language + Formulae)
Paper, Textbook, etc. Calculating the mean of n2 for n = 1,2, . . . ,10Conversion
Computational Form (Formal Language)
Executable code,mean([n^2 for nFirst-order logic, etc.
in range(1, 11)])▶ The label set (candidate concepts) vary from paper to paper→ It is not a simple multiclass classiﬁcation▶ The use of MLP is somewhat unique
1. Make the pairs of (occurrence, concept) for each occurrence
2. Train MLP to predict the likelihood of the correct pair
→ This method can be used for unknown label sets

Technologies for the conversion▶ NL: POS tagging, semantic parsing, text classiﬁcation, etc.
▶ Formulae: Token-level analysis, parsing, etc.
▶ Integration of NL texts and formula analyses
Grounding of Formulae
1. List up math concepts used in a document cf. Deﬁnition extraction
2. Assign a math concept to each math token occurrence
3. Associate math concepts with external knowledge cf. MathIR(1) List up math concepts

The result of running the machinelearning algorithm can be expressed asa function y(x) which takes a new digit
image x as input and that generates an
output vector y, encoded in the same
way as the target vectors. The precise
form of the function y(x) is determined
during the training phase. (p. 2, PRML)

External knowledge

Math concepts• function y(·)• output vector yConcept 1

Concept 2(2) Assign concepts(3) Associate withexternal knowledge

Math Concept Assignment
1Feature Engineeringc: Context Embeddings▶ Natural language text surrounding the target occurrence
E.g. feature vector $v’_{x}$ extracted from▶ Vector embeddings with Sentence Transformer [Reimers+, 2019]▶ Used MiniLM as a pretrained model (because it performed the best)▶ Impacts of window size and formula representation are little
a: Aﬃx Types▶ Local formula structure E.g. Use of sub-/super-script▶ We built a rule-based detector → Accuracy 90.56%
p: Position Data
▶ Cascade eﬀect scope and distance from the initial position→ eﬀective even if it soley used (identical to cascade baseline)
Model Comparison

Our Scope▶ Problem to solve: Intra-document ambulation▶ Target: Math identiﬁers (most frequent token type)Research Questions▶ What is the important feature for the disambiguation?
▶ Are those features depends on domain of the papers?

We trained our model with various combinations of the features▶ Model variations: 23 − 1 = 7 models▶ Used features are represented with a letter
E.g. c+ / a+ / p−: the model using context and aﬃx types

Task Overview
Input ▶ Structured document representation (XHTML)▶ The initial occurrence location associated with eachmath concept for identiﬁer = about 10% of the labels
Output Math concepts assigned to every occurrence identiﬁers= remaining 90% of the labels
0.85340.73300.8312

0.8

Score

0.6

0.5894

0.63290.00990.00000.6431

0.95150.79390.4
0.2

manhuodsca
ca

Use of Dataset▶ Split the dataset according to the ﬁeld of the paper▶ NLP subset: used for both development and evaluation▶ Others subset: used only for evaluation
Subsets of the Dataset
Subset #papers #words #idf_types #occrs #concepts
NLP
Others20 97,045
20 140,017

789 9,278
953 18,377

1,518
2,085

Total

40 237,062

1,742 27,655

3,603

Cross-domain Comparison

humanoubadeel

our model

rm

ine
sel

ba

mode

selbamdocascade baselineine

mode baselineine

random baseline

Accuracy
Kappa

sel0.0

ranTask Diﬃculty▶ Cascade baseline= assuming no scope switches
occurs except initial pos→ Kappa 0.6431▶ Human annotators→ Kappa 0.7939

1.0

https://sigmathling.kwarc.info/resources/grounding-dataset/

We trained our models with NLP subset, and evaluated with others