A Ground-Truthed Mathematical Character and Symbol Image Database
Masakazu Suzuki∗ , Seiichi Uchida∗∗ and Akihiro Nomura∗∗∗∗ Faculty of Mathematics,∗∗ Faculty of Information Science and Electrical Engineering,∗ ∗ ∗ Graduate School of Mathematics,
Kyushu University, 6-10-1 Hakozaki, Higashi-ku, Fukuoka-shi, 812-8581 Japansuzuki@math.kyushu-u.ac.jp
Abstract
This paper describes the specifications for our
ground-truthed mathematical character and symbol
image database, called InftyCDB-1. The ground-truthof each character is composed of type, font, quality(touched/broken) and link (relative position), etc. The
database includes all the characters and symbols of 467
pages of 30 articles on mathematics, and is organized
so that it can be used as word image database or as
mathematical formula image database. InftyCDB-1 isa public database that is freely usable for research anddevelopment purposes.

1 Introduction
In this paper, we report the specifications for a groundtruthed mathematical character and symbol image database,called InftyCDB-1. The database is comprised of 467 pagesof 30 articles on pure mathematics. InftyCDB-1 can beused, for example, in the following purposes:• development and evaluation of character and scientificsymbol recognition,• development and evaluation of mathematical formula
recognition, and• analysis of words in mathematical documents.
InftyCDB-1 is freely usable for those research and development purposes.
All the character/symbol images in the 467 page images
are included in the database with their ground-truth. Thus,
users can get training data or test data for character/symbol
recognition from the database. The ground-truth each character/symbol is composed of type, font (italic/upright, boldor not), quality (touched/broken) and link (relative position), and so on. Thus, for example, the database can beused for evaluations of font distinction ability in character
recognition. Note that for all special (i.e., very rare) math-ematical symbols, their own ground-truths have been carefully attached.
The image data are stored separated into word or mathformula units. Thus, InftyCDB-1 can be used as a word image database or as a mathematical formula image database.
The image data are arranged in alphabetic order independent of the content of papers. No whole page image is included in the database to avoid copyright problems.
Hereafter, the term character means not only ordinary
characters (e.g., “A”), but also math symbols (e.g., “+”), unless otherwise noted. The term category means the finest
level of character classification and the term type meansa set of categories having a similar property. For example, “A”, “B” and “C” are three categories belonging to thesame type (Roman). In contrast, “A”(Roman), “A”(italic),“A”(calligraph), “A”(blackboard bold), “A” (German), and“A” (script) are six categories belonging to different types.
Each character belongs to either the text region or the mathregion. The math region includes not only numbered equations but also in-line math formulae. Note that many in-line
math formulae are composed of a single character, such as“x” in the sentence “The variable x denotes . . .”.

2 Outline of database
2.1 Data collection
The documents contained in the database are 30 English
articles on pure mathematics (published 1970 ∼ 2000). The
numbers of pages, characters, words and math expressions
in the database are 467, 688,570, 108,914 and 21,056, respectively. For a quantitative analysis of the database, see[1]1 . This database is larger than past databases for research on math-OCR (e.g., about 15,000 characters in [2],about 10,000 characters in [3]). Note that matrices, tables,and figures are excluded from the database.
1 There are slight differences between the table1 below and the table in[1] because some errors were found after submission of the paper [1] andcorrected

Table 1. Characters in the database.
type

font

category
examplesˆ˜¯¨`ˇ←↔← #predefined
text region
math region
categories#cat.
#char (%) #cat.
#char (%) #cat.
accent13
1
2 ( <0.01)
72,700 ( 1.72)
7
arrow
16
1
3 ( <0.01)
71,103 ( 0.70)
7
big symbol
180
0 ( 0.00)
11
2,458 ( 1.57)
11
blackboard bold
ABCDEF
26
0
0 ( 0.00)
9427 ( 0.27)
9calligraphic
ABCDE F
26
0
0 ( 0.00)
19592 ( 0.38)
19
GermanUpright
ABCabc5200 ( 0.00)
25
1,041 ( 0.66)
25
Bold
ABCabc5200 ( 0.00)
0
0 ( 0.00)
0
Greek
UprightΓΔΘ
110
0 ( 0.00)
10
2,148 ( 1.37)
10Italicαβγ29
519 ( <0.01)
23
10,618 ( 6.76)
23
Bold
ΓΔΘ
11
0
0 ( 0.00)
1
3 ( <0.01)
1Italic Boldαβγ
290
0 ( 0.00)
531 ( 0.02)
5extended Latin
Upright
18230
392 ( 0.07)
2
3 ( <0.01)
30Ä Æ èItalic182
955 ( 0.01)
210 ( 0.01)
10Ä Æè
Ä Æ èBold182
46 ( <0.01)
0
0 ( 0.00)
4Ä Æè
Italic Bold
182
0
0 ( 0.00)
0
0 ( 0.00)
0
numeric
Upright012
10
10
12,018 ( 2.26)
10
15,294 ( 9.74)
10Italic012
1010140 ( 0.03)
4
118 ( 0.08)
10
Bold
012
10
10
923 ( 0.17)
426 ( 0.02)
10Italic Bold
012
10
0
0(
0)
0
0 ( 0.00)
0
operator+ − ×/ < &926154 ( 0.03)
49
20,359 ( 12.96)
50c ∞∀∃∂
othersUpright§@42
10
2,903 ( 0.55)
15
1,797 ( 1.14)
20
c
Bold
§@
16
3
42 ( 0.01)
0
0 ( 0.00)
3
parenthesis
Upright(){}[]
20
7
8,082 ( 1.52)
1230,334 ( 19.31)
12Bold( ) { } []20
2
112 ( 0.02)
0
0 ( 0.00)
2
point
Upright,.‘’
17
11
21,599 ( 4.06)
11
8,443 ( 5.41)
14
Bold
,.‘’
17
6
469 ( 0.09)
0
0 ( 0.00)
6
Roman
Upright
ABCabc61
57
414,825 ( 78.05)
55
8,259 ( 5.26)
57
ItalicABCabc61
55
63,590 ( 11.96)
53
49,072 ( 31.24)
56
Bold
ABCabc61
56
6,178 ( 1.16)
13538 ( 0.34)
56
Italic Bold
ABCabc610
0 ( 0.00)
19
1,508 ( 0.96)
19
scriptABC
52
0
0 ( 0.00)
7
176 ( 0.11)
7
total
1,571
294
531,512 (100.00) 373
157,058 (100.00) 487
Notes: (1) Each “Roman” and “italic” type includes nine double letters (i.e., ligatures), such as “fi”.

total#char (%)
2,702 ( 0.39)
1,106 ( 0.16)
2,458 ( 0.36)427 ( 0.06)
592 ( 0.09)
1,041 ( 0.15)
0 ( 0.00)
2,148 ( 0.31)
10,637 ( 1.54)
3 ( <0.01)
31 ( <0.01)395 ( 0.06)
65 ( 0.01)
6 ( <0.01)
0 ( 0.00)
27,312 ( 3.97)
258 ( 0.04)
949 ( 0.14)
0 ( 0.00)
20,513 ( 2.98)
4,700 ( 0.68)
42 ( 0.01)
38,416 ( 5.58)
112 ( 0.02)
30,042 ( 4.36)
469 ( 0.07)
423,084 ( 61.44)
112,662 ( 16.36)
6,716 ( 0.98)
1,508 ( 0.22)
176 ( 0.03)
688,570 (100.00)

All pages were scanned in 600 dpi and binarized automatically by the same commercial scanner (RICOH Imagio
Neo 450). The quality of the resulting page images varies
with the quality of original print and/or copy. Several page
images are noisy and include a lot of abnormal characters,such as touching or broken characters.

2.2 Ground-truth
The ground-truth for each character was attached manually by seven students in, or a graduate from, a university math department. The ground-truth of each character iscomposed of the following attributes:• type, category and font
• text or math region• normal or abnormal character• size (height and width)• link• location in the word or formula image• path (folder name + file name) to the image file.
The fifth attribute, link, represents the positional relationship to the preceding character, called parent character,Figure 1. Link example of a math expression.

and was attached to describe the structure of a math formula(as a tree). There are six kinds of links: horizontal, rightsuperscript, right-subscript, left-superscript, left-subscript,
upper, and lower. Figure 1 shows the link structure of a formula xt = xt−k2 + p. Note that the parent character of“+” is x and neither 2 nor k. The sixth attribute, location,is the rectangular coordinates (left,top,right,bottom) of thecharacter image in the corresponding word/formula image.
Each character data is connected to the word/formula image
data by the seventh attribute, path, and the sixth attribute,
location.

Figure 2. Word/formula segmentation.

Figure 3. SyllableAfter attribute for hyphenation.

2.3 Word segmentation
The segmentation of words is basically done by spacing.
On the other hand, a set of consecutive math expressions in
a line are unified into one formula regardless of the spacing,e.g., before and after a relative operators such as equal sign,
etc. A word or a formula continued to next line is separated
at the end of a line. Points (e.g. “,”, “.”, etc) are includedin the word/formula just before the points. Opening parenthesis are included in the next word and closing parenthesis
are included in the preceding words like points. Quotation
marks are treated in a similar way.
In Figure 2, the arrows show the segmentation points andthe sentence is separated into 11 words/formulae in this example.

3 Structure of database
The database InftyCDB-1 is composed of two parts: (i)text data and (ii) image data, related to each other. Text
data is a Microsoft Access or CSV-format, the user’s choice,while image data are systematically named PNG files.

3.1 Text data
For each character, the 29 attributes listed in the Table 2
are attached:
The attribute (5) is code defined in our laboratory to distinguish character/symbol categories in the math-OCR software called InftyReader[5]. The attribute(6) is a string toread the character: e.g., “int” for “ ”, “Omega” for “Ω”.
The attribute (7) is “text” for text region character, and“math” for math region character. The attribute (8) is “True”(resp. “False”) if the character is on the baseline (resp. in
sub/super-script area). The attribute (9) (resp. (10) is “True”
if the character is italic (resp. bold) font and “False” otherwise. The attribute (11) is “touched” for a touched character, “separate” for a broken character, “touch and sep” fortouched and broken character, and “normal” otherwise. By
using the attributes (14), (15), a user can reproduce the mathtree structure for each formula.
As for the rule to define the path to the image file in theattribute (16), see 3.2 below.
Attributes (21) – (29) are the same for all the charactersin a word/formula. Attribute (21) is the ID number attachedto each word/formula. Attributes (22), (23) and (24) are
string data that represent the corresponding word/formulain MathML format, LaTeX format, and IML format, respectively. IML format is the XML format used in the software developed in our laboratory, math-OCR “InftyReader”
and an authoring tool “InftyEditor” for mathematical documents. According to those ground-truthed string data, users
can evaluate the structural analysis procedures used in theirown math-OCRs.
The attribute (29) is “True” for words at the beginningof line continued by hyphenation from the word at the endof previous line, and “False” for other cases. In Figure 3,
the latter part “monic” of “harmonic” has the SyllableAfter
attribute “True”.
dhtν,z0 |t=0
A sample of the text data for a formula “
dt” and a word “and” is shown in Table 3. As noted in 2.2,
the link represents the positional relationship to the parent
character. Thus, for example, the “HORIZONTAL” link of“|” (vert, CharID=24) means that it and its parent character “h” (CharID=18) are on the same level. Then, using
the “HORIZONTAL” links from “h” to the fractional line
(CharID=14 and the first character), it is found that the “|”is on the baseline.

3.2 Image data
To reduce the number of image files, the images of asame word in a same article are stored in one image file.
However, italic words and upright words are stored in different image files, and Roman capital/small letter are distinguished so that, e.g., “And” “and” are stored in different
image files. Formula images in an article are also grouped
in a same way, when the expressions are identical. Image
file names are defined as follows:• word · · · “string”( FontFlag) “number”.png• formula · · · “number of characters in the formula”“first three characters in the formula” ( FontFlag)“number”.png
For example, the file names of the word “(and)” andthe formula “α ≤ 1” are “LeftPar-and-RightPar 0.png”
and “3 alpha-le-1 0.png”, respectively. Figure 4 shows
two image files in the database having the path names“ImagesActaM 1970 37 63Aapproaches I 0.png”
and“ImagesActaM 1970 37 63MATH1-97 DOmega-BigLeftPar 0.png”, respectively.

Table 2. List of attributes.
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
(15)
(16)
(17)(18)(19)(20)
(21)
(22)
(23)
(24)
(25)(26)(27)(28)(29)

Attribute
CharID
JornalID
SheetID
Type
Code
EntityRegionBaseline
ItalicFlag
BoldFlag
Quality
Width
HeightParentCharID
Link
ImageName
Rect
WordID
WordMathML
WordTeX
WordIML
WordRect
SyllableAfter

Explanation
ID number of each characterID number of the article
Page number
Type name (see Table 1)
Category code (OCR code)
Category nameDistinction of text/math area
Distinction of baseline/subscript
Italic flag
Bold flag
Normal/touched/separate/touch and sep
Width of the character
Height of the character
ID of the parent character
Type of link, which represents the positional relation to the parent character
Path (folder name + file name) to the image file including the character
Coordinates of the character in the image file (left,top,right,bottom)
ID number of the word/formula including the character
MathML string
LaTeX string
IML string
Coordinates of the word/formula in the image file (left,top,right,bottom)
Flag of word continued from the previous line by hyphenation

Figure 4. Example of image files

InftyCDB-1. The database consists of two parts: text dataand image data that are related to each other.
The ground-truth of each character is composed of type,
font, quality (touched/broken) and link information to represent the tree structure of math formula, etc. The databaseincludes all the 688,570 characters (and symbols) of 467
pages of 30 English articles on mathematics (published1970∼ 2000). Characters are grouped into words/formulaein the database. Total number of words and formulae inthe database are 108,914 and 21,056, respectively. The
database is freely usable for research, development andevaluation of math-OCRs.

4 Distribution

References

The database InftyCDB-1 is made and will be maintained in M.Suzuki laboratory, Faculty of Mathematics,
Kyushu University. It is freely available for researchand development purposes after user registration. (URL:
http://www.inftyproject.org/).
The text data part is in CSV (16.8MB) or Microsoft Access format (26.7MB) as the user chooses, and the image
data is in PNG format (202MB). The database is delivered
in CD-ROM.

[1] S. Uchida, A. Nomura, M. Suzuki “Quantitative Analysis of Mathematical Documents,” Int. J. Doc. Anal.
Recog., to appear.

5 Conclusion[4] A. Nomura, K. Michishita, S. Uchida, and M. Suzuki,“Detection and segmentation of touching characters inmathematical expressions,” Proc. ICDAR, 1:126-130,
2003.

In this paper, we described our ground-truthed mathematical character and symbol image database, called[2] H.-J. Lee and J.-S. Wang, “Design of a mathematical expression understanding system,” Pattern Recognition Letters, 18(3):289–298, 1997.
[3] M. Okamoto, H. Imai, and K. Takagi, “Performance
evaluation of a robust method for mathematical expression recognition,” Proc. ICDAR, 121-128, 2001.

Table 3. Example of the text data in the database for a formula “
(1)
14
15
16
17
18
19
20
21
22
23
24
25
26
27
..
.
804
805
806
(14)
-1
14
14
15
14
18
19
20
21
22
18
24
25
26
..
.
-1
804
805

(2)
5
5
5
5
5
5
5
5
5
5
5
5
5
5
..
.
2
2
2

(3)
10
10
10
10
10
10
10
10
10
10
10
10
10
10
..
.
13
13
13
(15)

(4)
BigSymbol
Roman
Roman
Roman
Roman
Roman
Greek
Point
Roman
Numeric
Parenthesis
Roman
Operator
Numeric
..
.
Roman
Roman
Roman
(16)
TOP
∗1
UNDER
∗1
UPPER
∗1
HORIZONTAL ∗1
HORIZONTAL ∗1
RSUB∗1HORIZONTAL ∗1
HORIZONTAL ∗1
HORIZONTAL ∗1
RSUB∗1HORIZONTAL ∗1
RSUB∗1HORIZONTAL ∗1
HORIZONTAL ∗1
..
..
.
.
TOP
∗5
HORIZONTAL ∗5
HORIZONTAL ∗5

(5)
(6)
(7)
(8)
(9)
33D1
fractionalLine
math
TRUE
FALSE
0164
d
math
TRUE
FALSE
0164
d
math
TRUE
FALSE
0174
t
math
TRUE
FALSE
0168
h
math
TRUE
FALSE
0174
t
math
FALSE FALSE
426D
nu
math
FALSE TRUE
142C
comma
math
FALSE FALSE
017A
z
math
FALSE FALSE
0130
zero
math
FALSE FALSE
197C
vert
math
TRUE
FALSE
0174
t
math
FALSE FALSE
1D3D
equal
math
FALSE FALSE
0130
zero
math
FALSE FALSE
..
..
..
..
..
.
.
.
.
.
0161
a
TRUE
text
FALSE
016E
n
TRUE
text
FALSE
0164
d
TRUE
text
FALSE
(17)
(18)
(19)
(20)
(21)
(22)
(23)
0
73
82
85
28005695
∗2
∗3
12
101
50
161
28005695
∗2
∗3
24
0
62
59
28005695
∗2
∗3
54
109
78
162
28005695
∗2
∗3
95
35
134
95
28005695
∗2
∗3
135
78
162
130
28005695
∗2
∗3
173
96
215
133
28005695
∗2
∗3
234
119
251
145
28005695
∗2
∗3
262
92
300
131
28005695
∗2
∗3
306
116
340
165
28005695
∗2
∗3
363
18
374
205
28005695
∗2
∗3
386
130
412
182
28005695
∗2
∗3
423
154
466
176
28005695
∗2
∗3
487
135
522
183
28005695
∗2
∗3
..
..
..
..
.. ..
..
.
.
.
.
. .
.
0
5850
37
5888
1000299
and
and
40
5851
82
5887
1000299
and
and
87
5831
129
5887
1000299andanddhtν,z0 |t=0 ” and a word “and”.
dt
(10)
(11)
(12)
(13)
FALSE normal
82
12
FALSE normal
38
60
FALSE normal
38
59
FALSE normal
24
53
FALSE normal
39
60
FALSE normal
27
52
FALSE normal
42
37
FALSE normal
17
26
FALSE normal
38
39
FALSE normal
34
49
FALSE normal
11
187
FALSE normal
26
52
FALSE touched
43
22
FALSE normal
35
48
..
..
..
..
.
.
.
.
FALSE normal
37
38
FALSE separate
42
36
FALSE normal
42
56
(24)
(25)
(26)
(27)
(28)
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
∗4
0
0
522
205
..
..
..
..
..
.
.
.
.
.
and
0 5831
129
5888
and
0 5831
129
5888
and
0 5831
129
5888

···

(29)
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
FALSE
..
.
FALSE
FALSE
FALSE

(∗1) AnnMS 1971 157 173MATH10-1914 fractionalLine-dd 0.png, (∗5) ActaM 1970 37 63Aand 1.png
dhtν,z0 |t=0 in MathML, in LaTeX and in IML respectively.
(∗2),(∗3), (∗4): Expression of the formuladt[5] M. Suzuki, F. Tamari, R. Fukuda, S. Uchida, T. Kanahori “Infty- an integrated OCR system for mathematical documents,” ACM Symposium on Document Engineering, 95-104, 2003
[6] J. Ha, R. M. Haralick, and I. T. Phillips, “Understanding mathematical expressions from document
images,” Proc. ICDAR, 956-959, 1995.
[7] Y. Eto and M. Suzuki, “Mathematical formula recognition using virtual link network,” Proc. ICDAR, 762–
767, 2001.
List of articles in the database• Acta Math., 124(1-2), 37-63, 1970. • ibid., 181(2), 283-305,
1998. • Ann. Sci. Ecole Norm. Sup., 4d sér, t.3, 273-284, 1970.

• ibid., t.30, 367-384, 1997. • Ann. Inst. Fourier, 20(1), 493-498,
1970. • ibid., 49(2), 375-404, 1999. • Ann. Math., 91, 550-569,
1970. • Ann. Math. Studies, 66, 157–173, 1971. • Arkiv für
Matematik, 9(1), 141-163 1971. • ibid., 35(1), 185-199, 1997.
• Bull. Amer. Math. Soc., 77(1), 157-159 1971. • ibid., 77(1),160-163 1971. • ibid., 80(6), 1219-1222, 1974. • ibid., 35(2),123-143, 1998. • Bull. Soc. Math. France, 98, 165-192, 1970.
• ibid., 126, 245-271, 1998. • Invent. Math., 9, 121-134, 1970.
• ibid., 138, 163-181, 1999. • J. Math. Soc. Japan, 27(2), 281288, 1975. • ibid., 27(2), 289-293, 1975. • ibid., 27(2), 497-506,
1975. • J. Math. Kyoto Univ., 11(1), 181-194, 1971. • ibid.,
11(1), 373-375, 1971. • ibid., 11(2), 377-379, 1971. • Kyushu
J. Math., 53, 17-36, 1999. • Math. Ann., 225(3), 275-292, 1977.
• ibid., 315, 175-196, 1999. • Tohoku Math. J., 25, 317-331,
1973. • ibid., 25, 333-338, 1973. • ibid., 42, 163-193, 1990.