Mathematical Expression Retrieval in PDFs
from the Web Using Mathematical Term Queries
Kuniko Yamada1 and Harumi Murakami1
Osaka City University, 3-3-138, Sugimoto, Sumiyoshi, Osaka 558-8585 Japand18ud512@eb.osaka-cu.ac.jp

Abstract. Since mathematical expressions on the web are not annotated with natural language, searching for expressions by conventional
search engines is diﬃcult. Our method performs web searches using amathematical term as a query and extracts expressions related to it fromthe obtained PDF ﬁles. We convert the PDF to TeX, create images fromthe mathematical descriptions in TeX and obtain image feature quantities. The expressions are discriminated by a support vector machine(SVM) using the feature quantities. Our experimental results show thateliminating slide-derived PDF ﬁles eﬀectively improves F-measure andthe mean reciprocal rank (MRR) is best when using both PDFs andHTML.
Keywords: Mathematical expression retrieval · PDF ﬁle · Web search.

1Introduction

There is much useful mathematical information on the web, especially in PDF
ﬁles that contain reliable information. However, it is diﬃcult to search for mathematical expressions in PDF ﬁles eﬃciently. Our method performs an ordinary
text search using a mathematical term as a query and presents mathematicalexpressions related to an input query from the PDF ﬁles. After converting thePDF to TeX, we create images from the mathematical descriptions in TeX andobtain the image feature quantities. Our method measures the relevance between a query and a mathematical expression from the following viewpoints:
the expression is in a separate line; the query is in the neighborhood and it
has appropriate image feature quantities when converting it to an image; and itappears in the ﬁrst part of the PDF ﬁle.

2Approach

2.1

Mathematical expression

Mathematical documents have features which diﬀer from ordinary documents.
The writing style for mathematical expressions in a document is: (a) variables
and signs, or an additional expression are on a line and are expressed with math2K. Yamada et al.

notations instead of characters because the mathematical font is special; (b) an
important theorem or formula is on a separate line, even though it is part of asentence; and (c) when a theorem is derived or a calculation example is shown, its
expression is very long. In such a case, the expression is usually not important.
2.2

MethodOur method is illustrated in Fig. 1. This approach does not depend on thelanguage; however, we conducted our original study in Japanese, so we translated
the data into English for ease of explanation. In our previous study [1], we
proposed a method which retrieves mathematical expression images on the web
using mathematical terms as queries. Here, we propose a method which retrieves
mathematical expressions in PDF ﬁles. In Fig. 1, highlighted parts ((2) and (3))
are the new proposal and the others are adjusted to suit the TeX documents.

㻔㻞䠅 㻯㼛㼚㼢㼑㼞㼠㻌㻼㻰㻲㻌㼠㼛㻌㼀㼑㼄 㼍㼚㼐㻌㻔㻝㻕㻌㻰㼛㻌㼍㻌㼠㼑㼤㼠㻌㼟㼑㼍㼞㼏㼔㻌㼣㼕㼠㼔㻌㼍㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㼑㼤㼠㼞㼍㼏㼠㻌㼙㼍㼠㼔㼑㼙㼍㼠㼕㼏㼍㼘㻌㼗㼑㼥㼣㼛㼞㼐㻌䠄㻹㼍㼠㼔㼑㼙㼍㼠㼕㼏㼍㼘㻌㼠㼑㼞㼙㻕㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㻌㼒㼞㼛㼙㻌㼀㼑㼄 㼒㼕㼘㼑㼟㼍㼚㼐㻌㼛㼎㼠㼍㼕㼚㻌㻼㻰㻲㻌㼒㼕㼘㼑㼟㻔㻥㻕 㻼㼞㼑㼟㼑㼚㼠㻌㻌㻌㻌㻌㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㼣㼕㼠㼔㻌㼟㼡㼞㼞㼛㼡㼚㼐㼕㼚㼓㻌㼕㼚㼒㼛㼞㼙㼍㼠㼕㼛㼚㻔㻣㻕 㻳㼕㼢㼑㻌㼙㼍㼞㼗㼟㻌㼠㼛㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㼒㼞㼛㼙㻌㼞㼑㼟㼡㼘㼠㼟㻌㻔㻠㻕㻘 㻔㻡㻕㻘㻌㼍㼚㼐㻌㻔㻢㻕㻗㻔㻤㻕 㻭㼐㼐 㼎㼛㼚㼡㼟㻌㼜㼛㼕㼚㼠㼟㻌㼠㼛㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㻔㻟䠅 㻯㼍㼘㼏㼡㼘㼍㼠㼑㻌㼠㼔㼑㻌㼘㼑㼚㼓㼠㼔㻌㼛㼒㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㻌㼍㼚㼐㻌㼏㼛㼚㼢㼑㼞㼠㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㻌㼣㼕㼠㼔㻌㼍㻌㼘㼑㼚㼓㼠㼔㻌㼛㼒㻌㼟㼕㼤㻌㼛㼞㻌㼙㼛㼞㼑㻌㼕㼚㼠㼛㻌㼕㼙㼍㼓㼑㼟㻔㻠㻕 㻰㼑㼠㼑㼞㼙㼕㼚㼑㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㻌㼕㼚㻌㼍㻌㼟㼑㼜㼍㼞㼍㼠㼑㼘㼕㼚㼑㻔㻡㻕 㻰㼑㼠㼑㼞㼙㼕㼚㼑㻌㼏㼍㼚㼐㼕㼐㼍㼠㼑㻌㼕㼙㼍㼓㼑㼟㻌㼎㼥㻌㻿㼂㻹㻌㼡㼟㼕㼚㼓㻌㼠㼔㼑㻌㼒㼛㼘㼘㼛㼣㼕㼚㼓㻌㼒㼑㼍㼠㼡㼞㼑㼟㻭㻦 㻲㼕㼘㼑㻌㼟㼕㼦㼑㻮㻦 㻺㼡㼙㼎㼑㼞㻌㼛㼒㻌㼣㼕㼐㼠㼔㻌㼜㼕㼤㼑㼘㼟㻯㻦㻌㻺㼡㼙㼎㼑㼞㻌㼛㼒㻌㼔㼑㼕㼓㼔㼠㻌㼜㼕㼤㼑㼘㼟㻰㼑㼚㼟㼕㼠㼥 䞉䞉䞉 㻭㻛䠄㻮㻖㻯㻕㻭㼟㼜㼑㼏㼠㻌㼞㼍㼠㼕㼛 䞉䞉䞉 㻮㻛㻯㻔㻢㻕 㻰㼑㼠㼑㼞㼙㼕㼚㼑㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㻌㼔㼍㼢㼕㼚㼓㻌㼍㻌㼗㼑㼥㼣㼛㼞㼐

Fig. 1. Overview(1) We perform a web search using mathematical terms as a query and capture the top 100. From the search result page returned by the search engine, weobtain PDF ﬁles and the boldface parts of the snippets.
(2) Preliminary experiment results showed that PDF ﬁles derived from slides
had poor accuracy. In order to eliminate these, we obtain the aspect ratio of thePDF ﬁles from “MediaBox” and “Rotate” in the PDF page object, and remove
landscape-oriented ﬁles beforehand. Then, we covert the PDF ﬁles to TeX ﬁlesusing InftyReader (http://www.inftyproject.org/en/index.html).
(3) In order to eliminate variables and fragments of expressions, we deﬁne
and calculate the length of a mathematical expression according to the following
procedure. (a) TeX commands are extracted from dataset D1 (described later)

Mathematical Expression Retrieval in PDFs

3and the command is given a 1 if it is a constituent element of an expression;otherwise, it is given a 0. For example, commands with 1 are ”\frac”, ”\partial”,and ”\alpha”, and commands with 0 are ”\mathrm”and ”\overline”. (b) TheseTeX commands are checked to determine whether they match part of the expression. The number of commands with 1 is length1 and all matched commands aredeleted. The number of the characters in the rest of the expression is length2.
The length of the expression is then the sum of length1 and length2. Since thelength of y = f (x) or E = mc2 (E=mc^2 in TeX notation) is 6, we deﬁne 6 ormore as a mathematical expression.
(4) When a mathematical expression part in TeX is in the display math mode,the expression is given a 1. When an expression part is in the inline mode, theexpression is given a 0.
(5) Mathematical expressions converted to images are classiﬁed by SVM.
We convert an expression to a PNG format image and use the SVM classiﬁer
developed by LIBSVM [2] from dataset D2 (described later). We previously
developed this classiﬁer in [1]. The feature quantities are shown in Fig. 1(5). If
an output of the SVM is positive, the expression is given a 1 and, if not, a 0.
(6) We search for the keyword (query) as a window size that is set from -200
to +200 characters surrounding a mathematical expression in TeX. If it exists,
the expression is given a 1; if not, a 0. However, when there is no expression witha keyword, we search again for an alternative keyword which is the boldface partof the snippet.
(7) Each expression is given a score using (4) to (6). At this point, threepoints is a full score.
(8) The ﬁrst order of expressions is returned by the search engine and the appearance order in the PDF ﬁle. When expressions are ranked in descending orderof their score, incorrect expressions are still highly ranked because their order
is aﬀected by the ranking of the original web pages. As important information
often appears in the ﬁrst part of a document, we propose a bonus point system
to solve this problem. This system gives an additional point to an expression
when it ﬁrst obtains the full score in the appearance order in the PDF. With
this system, the possibility increases that the correct expression in each PDF
rises to a higher order. From the above, score(ik ) is given to each expression ikby Eq. (1) in [1].
score(ik ) = xline + xkey + xsvm + xbo ,(1)

when xline = 1, an expression is in a separate line; when xkey = 1, an expression
has a keyword; when xsvm = 1, an expression is discriminated to be positive by
the SVM; and when xbo = 1, an expression has a bonus point.
(9) Our system ranks expressions according to score(ik ), and obtains thetop ten. Finally, the top ten expressions with their surrounding information
are displayed. In order to show collateral conditions and commentary on theexpressions, 200 characters before and after the expression are also displayed.
Figure 2 shows a screen display example. Here, the top one with “kernel trick”
as the keyword was retrieved correctly. This expression is extracted from the4

K. Yamada et al.

end of the second page of the original ﬁve-page PDF document. This PDF ranks
12th in the original ranking returned by the search engine.

㻷㼑㼥㼣㼛㼞㼐 㼍㼟㻌㼝㼡㼑㼞㼥㻻㼞㼕㼓㼕㼚㼍㼘㻌㻼㻰㻲㻿㼑㼍㼞㼏㼔㻌㼞㼍㼚㼗㻷㼑㼥㼣㼛㼞㼐㼟㻌㼕㼚㻌㻼㻰㻲㻾㼑㼠㼞㼕㼑㼢㼑㼐㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㻱㼤㼠㼞㼍㼏㼠㼕㼛㼚
Some right parentheses changed to brackets as a result of conversion mistakes.

Fig. 2. Display example of keyword: kernel trick

3Experiments

3.1

DatasetIn [1], we randomly selected keywords from the index of “Pattern Recognition
and Machine Learning [3] (Japanese version)” by Bishop and conducted a web
search using keywords as queries. We obtained the top 100 results for each keyword. We created dataset D1 using keywords 1 to 30, D2 using keywords 31 to60, and D3 usingis keywords 61 to 70 for evaluation. Table 1 shows the contentsof D3 . The expressions in D3 were judged manually.
Table 1. Contents of dataset D3
PDFHTMLExpression
FileImageFile Other
Correct Total
Correct Total
Total
398 29,708 297 (included 60 slide-derived PDFs)
181 6,030 554 149
Keywords: positive deﬁnite matrix, multi modality, equality constraint, Newton-Raphson method,root-mean-square error, Lagrange multiplier, sum rule of probability, kernel trick,uniform sampling, kinetic energy
Other: includes broken links, other ﬁle formats, PDF that cannot be converted to TeX, etc

Mathematical Expression Retrieval in PDFs

3.2

5Experiment1

The aim of this experiment was to examine the eﬀect of eliminating slide-derived
PDF ﬁles. Using F-measure, MRR (Eq. (2)), and mean average precision (MAP),
we evaluated the output results of the top ten. MAP was obtained by calculating
the macro mean of the Average Precision (AP) (Eq. (2)).
∑2 · Precision · Recall
1∑ 1
1, MRR =, AP =I(s)Prec(s),Precision + Recallnrk′min(n, c) sn

F − measure =(2)k=1where Precision = nr , Recall = rc , r is the number of correct expressions of topn, c is the total number of the correct expressions, rk′ is the rank of the correct
expression at the top of the k-th keyword, I(s) is the ﬂag indicating whether theexpression at s-th is correct or not, and Prec(s) is precision at s-th.
In Condition A, our method is applied to all PDF ﬁles in D3 and, in Condition
B, it is applied to the PDF ﬁles which were not derived from slides in D3 . The
results show that the F-measure and MRR improved by eliminating slide-derived
PDF ﬁles, as illustrated in Fig. 3 and Table 2.
㻜㻚㻟㻡㻜㻚㻟㻜㻜㻚㻞㻡㻜㻚㻞㻜㻜㻚㻝㻡㻜㻚㻝㻜㻜㻚㻜㻡㻜㻚㻜㻡㻜㻚㻜㻠㻮㻭㻜㻚㻜㻜㼀㼛㼜㻌㻝 㼀㼛㼜㻌㻞 㼀㼛㼜㻌㻟 㼀㼛㼜㻌㻠 㼀㼛㼜㻌㻡 㼀㼛㼜㻌㻢 㼀㼛㼜㻌㻣 㼀㼛㼜㻌㻤 㼀㼛㼜㻌㻥 㼀㼛㼜㻝㻜

Table 2. Results of Experiment1 (MRR, MAP)AB
MRR 0.50 0.56
MAP 0.17 0.17

Fig. 3. Results of Experiment1 (F-measure)

3.3

Experiment2

From the results of Experiment 1, we used the PDF ﬁles with the slide-derived
PDFs eliminated. The aim of this experiment was to compare three conditions:Condition B uses PDF (same as in Experiment 1); C uses HTML (same as in[1]); and D uses both PDF and HTML, and these are ranked again after restoringto the original ranking by the search engine. As shown in Fig. 4 and Table 3,almost all values demonstrate that C is better than B and D; however D is thebest in MRR.
3.4Discussion

PDF ﬁles have many correct expressions (see Table 1) and clear sources. 99.4%
of D3 was obtained from organizations such as universities, academic societies,

6K. Yamada et al.

㻜㻚㻢㻜㻜㻚㻡㻜
㻜㻚㻠㻜㻜㻚㻟㻜
㻜㻚㻝㻝㻯㻰㻜㻚㻞㻜㻜㻚㻝㻜㻮㻜㻚㻜㻡㻜㻚㻜㻜㼀㼛㼜㻝㼀㼛㼜㻞㼀㼛㼜㻟㼀㼛㼜㻠㼀㼛㼜㻡㼀㼛㼜㻢㼀㼛㼜㻣㼀㼛㼜㻤㼀㼛㼜㻥㼀㼛㼜㻝㻜Table 3. Results of Experiment2 (MRR, MAP)BCD
MRR 0.56 0.75 0.77
MAP 0.17 0.34 0.29

Fig. 4. Results of Experiment2 (F-measure)research institutes, and enterprises. To use these PDF ﬁles more eﬀectively, ourfuture tasks are to: (1) increase evaluation data to verify experimental results inmore detail; (2) develop a new classiﬁer to improve accuracy; and (3) perform
preprocessing, other than the length, to improve accuracy.

4

Related work

Mathematical expression retrieval usually means retrieval of similar mathematical expressions and a search scope within a particular database. For example,
Zanibbi et al. conduct similar mathematical expression retrieval which searches
in the dataset developed from arXiv using mathematical expressions written inLaTeX and those converted into images [4]. We were unable to ﬁnd any researchusing a dataset from PDFs on the web.

5Conclusions

We proposed a method to perform web searches using a mathematical term as
a query and extracted mathematical expressions related to it from the obtained
PDF. Our experiments demonstrate the usefulness of using PDFs other thanslide-derived PDFs as well as using both PDF and HTML in MRR.

References
1. Yamada, K., Ueda, H., Murakami, H., and Oka, I.: Mathematical expression image
retrieval on web using mathematical terms as queries. Transactions of the JapaneseSociety for Artiﬁcial Intelligence 33(4), A-H91 1–13 (2018)
2. Chang, C.-C. and Lin, C.-J.: LIBSVM: A library for support vector machines. ACM
Transactions on Intelligent Systems and Technology, 2:27:1–27:27 (2011)
3. Bishop, C. M.: Pattern Recognition and Machine Learning. Springer-Verlag, NewYork (2006)
4. Zanibbi, R. and Yuan, B.: Keyword and image-based retrieval for mathematicalexpressions. In: Proceedings of Document Recognition and Retrieval XVIII, pp.
011–019 (2011)