Presenting Mathematical Expression Images
on Web to Support Mathematics Understanding
Kuniko Yamada1 , Hiroshi Ueda2 , Harumi Murakami1 , and Ikuo Oka1
1Osaka City University,
3-3-138, Sugimoto, Sumiyoshi, Osaka 558-8585 Japank16uz90z10@st.osaka-cu.ac.jp
2
Stroly Inc.
7-5-1, Seikadai, Seika, Soraku, Kyoto 619-0294 Japan

Abstract. People cannot use a text search to find mathematical expressions because expressions cannot be replaced with words. Our researchuses an ordinary text search and presents appropriate mathematical expression images (hereinafter called math-images) for input keywords.
First we classify a set of the top ranking images from all the images
in HTML files by scoring them. We focus on three viewpoints that are
unique to mathematical expression images and mark the images by using
these viewpoints. Then by adding bonus points to these marked images,
the best three images are chosen from the set and presented with an explanation of the keyword and the surrounding information in the HTMLfiles. We conducted two experiments to optimize the parameters of theexpression giving the mark and to evaluate the eﬀect of the bonus points.
The rate of the average correct images of the best three was 79.5%.
Keywords: mathematical expression image, mathematics understanding, mathematical information retrieval, web search, SVM, Wikipedia.

1Introduction

For such fields as science, technology, or economics, searching for mathematicalexpressions is necessary because these concepts are explained using expressions.
However, mathematical information retrieval is diﬃcult. Since expressions are
not annotated by names, we cannot search for them by names; we cannot use atext search.
Our research uses an ordinary text search given a mathematical term as akeyword and presents appropriate math-images for an input keyword. We focus
on the viewpoints where important math-images are on a separate line in adocument, and many of these images have unique feature quantities. Adding
another viewpoint where these important images often have a keyword aroundthem to the viewpoints, we obtain a set of the top ranking math-images. Focusing
on one more viewpoint where important information often appears in the firstpart of a document, we obtain the best three math-images from a set of thetop rankings. Our experiment result showed that at least two of the best three2K. Yamada, H. Ueda, H. Murakami, and I. Oka

math-images were correct images. To support mathematics understanding, thebest three images are shown on a screen display with keyword explanations andthe surrounding information in HTML files.

2Approach

2.1

Mathematical Expression

General ways of writing mathematical expressions on the web use pdfs, images
embedded in a HTML file, or MathML based on XML. The following is thewriting style for mathematical expressions in a document. (a) A variable, a sign,or an additional expression is on a line and is expressed with math notation
instead of characters because the mathematical font is special. (b) An important
theorem or a formula is on a separate line, even though it is in a sentence. (c)Connected by equal signs, a mathematical expression is too long from which atheorem is derived or a calculation example is shown.
2.2

MethodIn the following steps, we explain Fig.1.

㻰㼛㻌㼍㻌㼠㼑㼤㼠㻌㼟㼑㼍㼞㼏㼔㻌㻌㻌㻌㼣㼕㼠㼔㻌㼍㻌㼗㼑㼥㼣㼛㼞㼐䠄㻹㼍㼠㼔㼑㼙㼍㼠㼕㼏㼍㼘㻌㼠㼑㼞㼙㻕㻔㻣㻕㻌㻼㼞㼑㼟㼑㼚㼠㻌㼕㼙㼍㼓㼑㼟㻔㻝䠅 㻻㼎㼠㼍㼕㼚㻌㼕㼙㼍㼓㼑㼟㻌㼍㼚㼐㻌㼠㼔㼑㼕㼞㻌㼜㼞㼛㼜㼑㼞㼠㼕㼑㼟㻲㼕㼘㼑㻌㼟㼕㼦㼑㻺㼡㼙㼎㼑㼞㻌㼛㼒㻌㼣㼕㼐㼠㼔㻌㼜㼕㼤㼑㼘㼟㻺㼡㼙㼎㼑㼞㻌㼛㼒㻌㼔㼑㼕㼓㼔㼠㻌㼜㼕㼤㼑㼘㼟㻔㻡㻕 㻳㼕㼢㼑㻌㼙㼍㼞㼗㼟㻌㼠㼛㻌㼕㼙㼍㼓㼑㼟㼒㼞㼛㼙㻌㼞㼑㼟㼡㼘㼠㼟㻌㻔㻞㻕㻘㻌㻔㻟㻕㻘㻌㼍㼚㼐㻌㻔㻠㻕㻗㻔㻢㻕 㻳㼕㼢㼑㻌㼎㼛㼚㼡㼟㻌㼜㼛㼕㼚㼠㼟㻌㼠㼛㻌㼕㼙㼍㼓㼑㼟㻔㻞㻕 㻰㼑㼠㼑㼞㼙㼕㼚㼑㻌㼕㼙㼍㼓㼑㼟㻌㻌㼕㼚㻌㼍㻌㼟㼑㼜㼍㼞㼍㼠㼑 㼘㼕㼚㼑㻔㻟㻕㻌㻰㼑㼠㼑㼞㼙㼕㼚㼑㻌㼏㼍㼚㼐㼕㼐㼍㼠㼑㻌㼕㼙㼍㼓㼑㼟㻌㼎㼥㻌㻿㼂㻹㻌㼡㼟㼕㼚㼓㻌㼠㼔㼑㻌㼒㼛㼘㼘㼛㼣㼕㼚㼓㻌㼒㼑㼍㼠㼡㼞㼑㼟㻭㻦 㻲㼕㼘㼑㻌㼟㼕㼦㼑㻮㻦 㻺㼡㼙㼎㼑㼞㻌㼛㼒㻌㼣㼕㼐㼠㼔㻌㼜㼕㼤㼑㼘㼟㻯㻦㻌㻺㼡㼙㼎㼑㼞㻌㼛㼒㻌㼔㼑㼕㼓㼔㼠㻌㼜㼕㼤㼑㼘㼟㻰㼑㼚㼟㼕㼠㼥 䞉䞉䞉 㻭㻛䠄㻮㻖㻯㻕㻭㼟㼜㼑㼏㼠㻌㼞㼍㼠㼕㼛 䞉䞉䞉 㻮㻛㻯㻔㻠㻕 㻰㼑㼠㼑㼞㼙㼕㼚㼑㻌㼕㼙㼍㼓㼑㼟㻌㼔㼍㼢㼕㼚㼓㻌㼍㻌㼗㼑㼥㼣㼛㼞㼐Fig. 1. Overview(1) Obtain images: When the HTML files of a mathematical term are given,
we obtain the images and extract the following image information: the file size,
the number of width pixels, and the number of height pixels. When there is notext between the images, they are concatenated and considered as one.
(2) Determine images on a separate line: When an image satisfies thefollowing two conditions, it is defined as being on a separate line: (a) An image
tag to which the image belongs has no text before or after it. (b) The image
tag has tags that denote a new line before and after, such as <br>, <p>, <tr>,</br>, </p>, </tr>.
(3) Determine candidate images by SVM: We determine whether animage is a candidate by Support Vector Machines (SVM). The features given toPresenting Mathematical Expression Images on Web

3

the SVM are file size, the number of width pixels, the number of height pixels,
the density, and the aspect ratio.
(4) Determine images having a keyword: We search for the closest image
to a keyword as the window size that is set from -80 to +80 Japanese characterssurrounding the keyword. If it exists, the image is defined as having a keyword.
However, if an image and a keyword appear in the same sentence, it is preferentially defined as having a keyword.
(5) Give marks to images: Each image is given a mark using (2) to (4)and ranked in descending order of points. Fig. 2 shows an example of the top
ranking images when the keyword is “spherical harmonics”. The first two images
are incorrect and the others are correct.
㻔㻝㻕㻔㻡㻕

㻔㻞㻕­ !2 2½
  V r ¾\ r
®¯ 2m¿l Yll Yl 1fll 0 m lclml Yl 10,¦¦ c Yf T ,I
SmllmwY ·1 w §1 w 2Y
 l l 1 Y¨ sin T¸wT ¹ sin 2 T wI 2sin T wr ©

S1³ S dI ³ d cos T YmT ,ImT ,Il³ S dI ³ dT sin T f T , I YlT , I Yl m T , I'm

S³ S dI ³ dT sin T Y0㻔㻝㻞㻕'l

1

S

1S0

Yl m T , I㻔㻣㻕01m m / 22l  1 l  m ! mPl cosT e imI
4S l  m !

㻔㻤㻕Yl m T , I

Yl m T , I㻔㻠㻕wY ·1 w 2Y1 w § OY¨ sin T¸
wT ¹ sin 2 T wI 2sin T wT ©

Pl m cos T e imI㻔㻥㻕Yl m T , IlmT , I Yl m T , I'Ylm T , I0

­ 2 K lm cos mI Pl m cos T0m°°mmm0cos T® 2 K l sin  mT Pl° K 0 P 0 cos Tm 0°¯ l l2l  1 (l  m )!
4Sl m !

K lm㻔㻝㻝㻕T ,I

1

S



mlm l0³ S dI ³ d cos T f T , I Y
㻔㻢㻕! l  m l  m  1 Yl m 1l Yl m

㻔㻝㻜㻕m 1㻔㻟㻕ll 0 m 1! l  m l  m  1 Ylmf

¦ ¦ A Y T ,Ig T ,IH\ rN lm Pl m cos T ) I'­1 l l ' , m m '®
¯0 otherwise㻯㼛㼞㼞㼑㼏㼠㻌㼕㼙㼍㼓㼑㻵㼚㼏㼛㼞㼞㼑㼏㼠㻌㼕㼙㼍㼓㼑

Fig. 2. Example of top ranking images for keyword: spherical harmonics

㻷㼑㼥㼣㼛㼞㼐

Spherical harmonics

㼃㼕㼗㼕㼜㼑㼐㼕㼍㻌㼑㼚㼠㼞㼥㻻㼡㼠㼘㼕㼚㼑 Spherical harmonics from “ja.wikipedia.org/wiki/Spherical_harmonics”
The spherical harmonics are the angular portion of the solution to Laplace’s equation in spherical coordinates.
-----------------------------------------------------------------------------------------------------------------㻹㼍㼠㼔㼑㼙㼍㼠㼕㼏㼍㼘㻌㼑㼤㼜㼞㼑㼟㼟㼕㼛㼚㼟㻌㼞㼑㼘㼍㼠㼑㼐 to spherical harmonics
From Spherical harmonics - Wikipedia

Spherical harmonics Ylm (ȟ, ȭ) are the functions expressed by the following equation:㻯㼛㼞㼞㼑㼏㼠㻌
㼕㼙㼍㼓㼑㼟

where l is a non-negative integer and m is an integer in -l ӊ m ӊ l .
From Spherical harmonics - szksrv

When the differential equation is given by㻿㼡㼞㼞㼛㼡㼚㼐㼕㼚㼓㻌㼕㼚㼒㼛㼞㼙㼍㼠㼕㼛㼚㻌㼛㼒㻌㼏㼛㼞㼞㼑㼏㼠㻌㼕㼙㼍㼓㼑㼟the solutions are called spherical harmonics
where l0, 1, 2, 3,    .

.
.

Fig. 3. Screen display example of keyword: spherical harmonics

4

K. Yamada, H. Ueda, H. Murakami, and I. Oka(6) Give bonus points to images: In Fig. 2, the first two happen to
be incorrect, because their order is aﬀected by the ranking of the original web
pages. This order is not useful when we choose the best three or five from thetop ranked images. To solve this problem, we propose a bonus point system
where the important information on a keyword often appears in the first part ofa document. This system gives an additional point to an image when it obtains
the highest marks in the order of the appearance in the HTML file. Then thepossibility increases that the actually correct image in each HTML file rises tothe higher position in the set of the top rankings.
(7) Present image: Fig. 3 shows a screen display example that has allcorrect images (The third image is omitted). At the top, the display shows
the outline extracted from the first paragraph of Wikipedia that matches thekeyword. Below it, the display shows the best three images with the information
around them in the HTML files. Adding the surrounding information provides
clearer explanation about the images.

3Experiment

3.1

DatasetWe extracted 16 keywords from the science and engineering web syllabi of Osaka
City University and obtained the top five HTML files for each keyword using
a text search in Experiment 1. We added five keywords in Experiment 2, for a
total of 105 web pages and 3,262 images. These images were judged manuallyand SVM used eight keyword search results as training data. In Experiment 1,
the remaining eight keyword search results were evaluation data. In Experiment2, 13 keyword search results including additional five keyword search results wereevaluation data.
3.2

Experiment 1

To give marks to the images, we developed Expression (1), where points is theacquisition mark, image is the math-image, and αk is the weights. When k = 1,an image in a separate line is correct; when k = 2, an image determined to becorrect by SVM is correct; when k = 3, an image with a keyword is correct. The
aim of this experiment is to optimize the values of αk and δ3 (image) when an
image is incorrect.
points =3∑αk δk (image)

(1)

i=1

When k = 1, 2 {
1 (image : correct)
δk (image) =−1 (image : incorrect)

When k = 3 {1(image : correct)−1 or 0 (image : incorrect)

δk (image) =First, to determine -1 or 0 of δ3 (image), two sets were created by fixing each
αk = 1. One had -1 as its δ3 value and the other had 0, and then the top ranking

Presenting Mathematical Expression Images on Web

5images were compared using precision, recall, and F-measure. From Figs. 4 (1)and (2), we adopted 0. Next, to determine the αk values, four sets were created.
The first had α1 = α2 = α3 = 1 as its condition and the second to the fourth
had conditions where only one value of each αk was 2 and the others were 1.
Then we compared these sets using F-measure. In Figs. 4 (2) to (5), since (2) isthe highest, we adopted α1 = α2 = α3 = 1.

㻝㻜㻜㻚㻜㻑㻥㻝㻚㻝㻑㻥㻜㻚㻜㻑㻤㻜㻚㻜㻑㻣㻝㻚㻜㻑㻣㻜㻚㻜㻑㻣㻝㻚㻜㻑㻢㻞㻚㻞㻑㻡㻡㻚㻟㻑㻢㻜㻚㻜㻑㻣㻝㻚㻤㻑㻣㻜㻚㻞㻑㻢㻞㻚㻢㻑㻢㻞㻚㻥㻑 㻡㻢㻚㻡㻑㻡㻢㻚㻠㻑㻡㻢㻚㻡㻑㼜㼞㼑㼏㼕㼟㼕㼛㼚㼞㼑㼏㼍㼘㼘㻲㻙㼙㼑㼍㼟㼡㼞㼑㻠㻢㻚㻢㻑㻡㻜㻚㻜㻑㻠㻜㻚㻜㻑㻟㻝㻚㻤㻑㻟㻜㻚㻜㻑㻝㻥㻚㻞㻑㻞㻜㻚㻜㻑㻝㻜㻚㻜㻑㻜㻚㻜㻑㻔㻝㻕䃐㻝㻩㻝㻘㻌䃐㻞㻩㻝㻘㻌䃐㻟㻩㻝㻔㻞㻕䃐㻝㻩㻝㻘㻌䃐㻞㻩㻝㻘㻌䃐㻟㻩㻝䃓㻟㻦㻌㻝㻘㻌㻙㻝䃓㻟㻦㻌㻝㻘㻌㻜㻔㻟㻕䃐㻝㻩㻞㻘㻌䃐㻞㻩㻝㻘㻌䃐㻟㻩㻝㻔㻠㻕䃐㻝㻩㻝㻘㻌䃐㻞㻩㻞㻘㻌䃐㻟㻩㻝㻔㻡㻕䃐㻝㻩㻝㻘㻌䃐㻞㻩㻝㻘㻌䃐㻟㻩㻞(1) and (2): Comparison of applying 0 and -1 to images without a keyword(2) to (5): Comparison when changing weights

Fig. 4. Results of Experiment 13.3

Experiment 2%HVW%HVW㻢㻞㻚㻠㻡㻠㻝㻚㻤㻟㻜㻞㻝㻜

1XPEHURIFRUUHFWLPDJHV

1XPEHURIFRUUHFWDJHV㻢㻟㻚㻤㻡㻞㻚㻤㻠㻟㻞㻝㻜㻝㻞㻟㻠㻡㻢㻣㻤㻥㻷 㼑 㼥 㼣㼛㼞㼐 㼟㻺㼛㼠㻌㼍㼐㼐㼕㼚㼓㻌㼎㼛㼚㼡㼟㻌㼜㼛㼕㼚㼠㼟㻾㼍㼠㼑㻌㼛㼒㻌㼍㼢㼑㼞㼍㼓㼑㻌㼏㼛㼞㼞㼑㼏㼠㻌㼕㼙㼍㼓㼑㼟㻦㻌㻢㻝㻚㻡㻑㻝㻜㻝㻝㻝㻞㻝㻟㻝㻠㻭㼐㼐㼕㼚㼓㻌㼎㼛㼚㼡㼟㻌㼜㼛㼕㼚㼠㼟㻾㼍㼠㼑㻌㼛㼒㻌㼍㼢㼑㼞㼍㼓㼑㻌㼏㼛㼞㼞㼑㼏㼠㻌㼕㼙㼍㼓㼑㼟㻦㻌㻣㻥㻚㻡㻑

㻝

㻞㻟㻠㻡

㻢㻣㻤
㻥㻷 㼑 㼥 㼣㼛㼞㼐 㼟㻺㼛㼠㻌㼍㼐㼐㼕㼚㼓㻌㼎㼛㼚㼡㼟㻌㼜㼛㼕㼚㼠㼟㻾㼍㼠㼑㻌㼛㼒㻌㼍㼢㼑㼞㼍㼓㼑㻌㼏㼛㼞㼞㼑㼏㼠㻌㼕㼙㼍㼓㼑㼟㻦㻌㻡㻡㻚㻜㻑㻔㼍㻕Keyword List:
1. Ampere’s rule
2. Euler’s formula
3. Gauss’ law
6. Cauchy’s integral theorem 7. Schrodinger equation
8. Navier-Stokes equation
11. Gamma function
12. Green’s theorem
13. Taylor-Hamilton theorem㻝㻜㻝㻝㻝㻞㻝㻟㻝㻠㻭㼐㼐㼕㼚㼓㻌㼎㼛㼚㼡㼟㻌㼜㼛㼕㼚㼠㼟㻾㼍㼠㼑㻌㼛㼒㻌㼍㼢㼑㼞㼍㼓㼑㻌㼏㼛㼞㼞㼑㼏㼠㻌㼕㼙㼍㼓㼑㼟㻦㻌㻣㻡㻚㻜㻑㻔㼎㻕

4. Dirac delta function
9. Euler-Maclaurin formula
14. Average

5. Spherical harmonics
10. Divergence theorem

Fig. 5. Comparison of adding and not adding bonus points

6

K. Yamada, H. Ueda, H. Murakami, and I. Oka∑3Expression (1) was fixed to points = i=1 δk (image) and δ3 (image) = 0 when
the image is incorrect. Using this expression, we examined the validity of ourbonus point system. In Fig. 5, adding bonus points was basically better than
not adding them. Especially in Fig. 5 (a), the number of correct images of thethird keyword, “Gauss’ law”, rose to 2 from 0. The rates of the average correct
images showed that the bonus point system was eﬀective, and since the rate of
the average correct images of the best three exceeded the rate of the best five,we adopted the best three to which bonus points were added.

4

Related work

Even though various methods have addressed mathematical information retrieval[1], they are searching for math notations with similar structure or semantics for agiven expression. Among those methods, content-based image retrieval (CBIR)
inputs a digital image to search for similar images [2], and instead of using
images, others use MathML [3] or LaTeX [4].
This paper’s approach is quite diﬀerent. Focusing on the unique features ofmathematical expressions and math-images led us to use text information. Even
people who are unfamiliar with handling MathML or LaTeX can use our method.

5Conclusions

We proposed a method to present math-images with the information aroundthem in the HTML files. Our method uses an ordinary text search that inputs
a mathematical term as a keyword. To obtain the best three math-images, wedeveloped Expression (1) and optimized its parameters. Our results showed thatat least two of the best three were correct, and the rate of the average correct
images of the best three was 79.5%. Then using these three images, we showeda screen display with an explanation of the keyword and information about theimages to support mathematics understanding.

References
1. Zanibbi, R., Blostein D.: Recognition and Retrieval of Mathematical Expressions.
International Journal on Document Analysis and Recognition 15(4), 331-357 (2012)
2. Shirmenbaatar, M., Koga, H., Watanabe, T.: Content-Based Mathematical Formula
Image Retrieval System. In: The 4th forum on Data engineering and Information
Management (DEIM2012) (2012)
3. Yokoi, K., Nghiem, M.-Q., Matsubayashi, Y., Aizawa, A.: Contextual Analysis of
Mathematical Expressions for Advanced Mathematical Search. Research journal on
Computer science and computer engineering with applications 43, 81-86, Polibits(2011)
4. Zanibbi, R., Yuan, B.: Keyword and image-based retrieval for mathematical expressions. In: Proceedings of Document Recognition and Retrieval XVIII, SPIE, pp.
011-019, (2011)