INFTY — An integrated OCR system formathematical documents
Masakazu Suzuki 1) , Fumikazu Tamari2) , Ryoji Fukuda3) ,Seiichi Uchida4) , Toshihiro Kanahori 5)1) Faculty of Mathematics, Kyushu University, Japan
2) Department of Information Education, Fukuoka University of Education, Japan
3) Department of Human Welfare Engineering, Oita University, Japan
4) Faculty of Information Science and Electrical Engineering, Kyushu University, Japan
5) Research Center on Educational Media, Tsukuba College of Technology, Japansuzuki@math.kyushu-u.ac.jp
ABSTRACTAn integrated OCR system for mathematical documents,
called INFTY, is presented. INFTY consists of four procedures, i.e., layout analysis, character recognition, structure analysis of mathematical expressions, and manual error correction. In those procedures, several novel techniques
are utilized for better recognition performance. Experimental results on about 500 pages of mathematical documents
showed high character recognition rates on both mathematical expressions and ordinary texts, and suﬃcient performance on the structure analysis of the mathematical expressions.

Keywords
Mathematical OCR, character and symbol recognition, structure analysis of mathematical expressions

1.

INTRODUCTION

Optical character reader (OCR) systems which can recognize not only ordinary texts but also mathematical expressions have been investigated [1]. The development of such
OCR provides the following merits.
• Storage size reduction: The OCR result of a document
requires far less storage size than its original scanned
image.
• Search services: Various search services (e.g., keyword
search, deﬁnition search, and theorem search) are available across the mathematical documents.
• Format conversion: The OCR result can be provided
in various document formats (e.g., XML, LaTeX, Mathematica notebook, and braille).

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
DocEng ’03, Grenoble, France
Copyright 2003 ACM X-XXXXX-XX-X/XX/XX ...$5.00.

Especially, the OCR for mathematical documents is indispensable on digitizing numerous historical mathematical documents for digital library [2, 3].
In this paper, an integrated OCR system for mathematical
documents, called INFTY, is presented. Figure 1 shows asnapshot of INFTY on a PC. INFTY reads scanned page images of a mathematical document and provides their character recognition results. Since INFTY analyzes the structuresof mathematical expressions in the document, INFTY canproduce its recognition result in the LaTeX format (Fig. 2)
and other math-description formats . Figure 3 shows thediagram of INFTY, which consists of four procedures, i.e.,
1 layout analysis, 
2 character recognition, 
3 structure
analysis of mathematical expressions, and 
4 manual error
correction.
Novel and distinctive features of INFTY are summarized
as follows.
• The character recognition procedure of INFTY consists of two independent and complementary recognition engines; one is a commercial OCR engine not specialized for mathematical documents and the other isa character recognition engine originally developed formathematical symbols.
• The separation of ordinary text parts and mathematical expression parts is performed in the character recognition procedure while utilizing recognition results.
• The structure analysis procedure is based on an optimization framework and therefore stable against toboth recognition errors and ambiguity in the mathematical expressions.
• A clustering technique is incorporated for higher accuracy and eﬃciency.
The rest of this paper is organized as follows. In Section
2,3,4, and 5, the details of above four procedures (Fig. 3–1 ∼4 ) are described, respectively. In those description, themerits of above features are emphasized. Then, in Section
6, the performance of INFTY is evaluated qualitatively andquantitatively through experimental results on about 500
pages of mathematical documents.

recognition result frame

scanned image frame

thumbnails frame

Figure 1: Snapshot of INFTY.

the functions $\sigma_a(r)={\displaystyle \int}_{ ||z-a||\leq r} \sigma$ and $\nu_a(r)={\displaystyle \int}_{||z-a||\leq r}\nu_a$. Bothare positive increasing functions of $r$. Then
Figure 2: (Upper) Input to INFTY. (Lower) Output
of INFTY in LaTeX format.

2.

LAYOUT ANALYSIS

In the layout analysis procedure (Fig. 3–1 ), which is
the ﬁrst procedure of INFTY, several preprocessing operations, such as binarization, noise removal, and deskewing,
are performed on the page images (scanned in 600dpi) of amathematical document.
After all connected components are extracted from thepreprocessed page image, the page image are separated intoﬁgure / table areas and non-ﬁgure areas. One of the main
criteria used in this separation is the size of the connectedcomponents. For example, the area with large connectedcomponents will be judged as a ﬁgure / table area. Notethat big symbols, such as root symbols, big parentheses,
etc., are ignored in this separation process by some special
treatments.
The non-ﬁgure area is further decomposed into text lines.
On non-mathematical documents, each text line is simply
extracted by searching for the periodical local minima on
the horizontal projection histogram of the page image. On
mathematical documents, however, this strategy is not expedient; the heights of mathematical expressions are very variable and therefore the horizontal projection histogram areoften irregular around the mathematical expressions. Our
strategy is similar to Kacem et al.[4], where connected components in a certain neighborhood are concatenated to build
a text line.

3.

CHARACTER RECOGNITION

The character recognition procedure (Fig. 3–2 ), which isthe second procedure in INFTY, plays two important roles.
The ﬁrst role is the separation of each text line into mathea
matical expressions (e.g., “α2 ”, “P (a) = −∞ p(x)dx”) andordinary texts (e.g., “Theorem”, “deﬁned”). The secondrole is the character recognition for both ordinary texts (e.g.,recognition results. The second sub-procedure is incorporated to improve the recognition accuracy by reducing misrecognitions due to slight shape diﬀerence. In the following,
the details of each sub-procedure are described.

scanned mathematical document image

1

Section 2
preprocessing

3.1
extraction of connected components
connected componentsof each page
separation of figure/table areas
non-figure areas(=ordinary text
+ math expression)

figure/table areas

extraction of lines in non-figure areas2

connected components
on each line
Section 3
Section 3.1
initial character recognition and math-text separation
ordinary texts
with recog. result

math-symbols
with recog. results

Section 3.2
automatic correction of recognition results using clustering
ordinary
textswith recog.
resultmath-symbols
with recog. results

3

Section 4
structure analysis of math expressions
non-math
structures+

math. symbols
with recog.
results

structure ofmath.
expression

4

Section 5
manual error correction

recog resultsof charactersin ordinary
text

XML

recog resultsof math. symbols

structure ofmath.
expression

LaTeX

etc..

brailleFigure 3: Diagram of INFTY. INFTY consists offour procedures (surrounded by dashed lines), i.e.,
1 layout analysis, 
2 character recognition, 
3 structure analysis of mathematical expressions, and 
4manual error correction.

“a”,
 “A”) and mathematical expressions (e.g., “a”, “A”,
“ ”, “α”, “(”).
As shown in Fig. 3–
2 , the character recognition procedure for those two roles consists of two sub-procedures,i.e., (i) initial character recognition and math-text separation and (ii) automatic correction of recognition results using
clustering. The ﬁrst sub-procedure is incorporated to provide math-text separation results as well as initial character

Initial character recognition and math-text
separation

Figure 4 illustrates the detail of the sub-procedure for
initial character recognition and math-text separation. This
sub-procedure has two features. One feature is that thissub-procedure is that the math-text separation is performed
while utilizing the result of character recognition. That is,the character recognition and the math-text separation are
performed simultaneously and cooperatively. The other feature is that two complementary recognition engines, a commercial OCR engine for ordinary texts and an original recognition engine for mathematical expressions, are used in a
two-step manner.
(a) Recognition by commercial OCR engine
The connected components on a text line is ﬁrstly subjected to a commercial OCR engine. When the text line
only contains ordinary texts, this OCR engine will produce
its good recognition result utilizing a rich lexicon. However,when the text line contains mathematical expressions, theOCR engine will fail (due to italic fonts, sub-/super-scripts,
mathematical symbols, etc.) and may produce some meaningless string (e.g., “y?zs” in Fig. 4) as the recognition result. In INFTY, this failure is exploited for initial math-text
separation. Namely, the connected components recognizedas such a meaningless string are selected as the connectedcomponents in the mathematical expressions.
(b) Verification based on position and size
Some mathematical expressions might be wrongly recognized as ordinary words (e.g., “x2 ”→“at” in Fig. 4) by thecommercial OCR (often due to the eﬀect of the lexicon) andthen they will not be detected in the above math-text separation procedure. Thus, after the initial recognition by thecommercial OCR, the connected components separated into
the ordinary text part should be veriﬁed.
In the veriﬁcation, the consistency of the position and thesize of the connected components of each word is checked.
For example, in the misrecognition “x2 ”→ “at”, the size of“a” (“x”, actually) is far diﬀerent from the size of “t” (superscript “2”, actually). In addition, the position of “a”
and the position of “t” also shows some inconsistency. According to those checks, “x2 ” is detected as a mathematicalexpression.
(c) Character recognition for mathematical expressions byoriginal recognition engine
The connected components selected into the mathematical expressions are then subjected to a recognition engine
originally developed for the characters and symbols in general mathematical expressions (i.e., English and Greek alphabets, numerals, operators, parentheses, etc.,). The number of categories distinguishable by this recognition engine
is about 564. In the mathematical expressions, font-types(e.g., roman, italic, calligraphic, and German) are often very
important. For example, “i” and “i” are used in diﬀerent
meanings. Thus, this engine should be designed to distinguish the font-types and therefore “i” and “i” (as well asconnected components on each line

Section 3.1

Section 3.1 (a)recognition by commercial OCR engine
ordinary
words

"The"

others(candidates of math)"value""at""y?zs""to""equals"

Section 3.1 (b)verification based on position and size
certaindoubtful+?

Section 3.1 (c)character recogition for mathematical expressions
by original recogition engine"The""equals""x2""value""to""y2 + z2"

ordinary texts
with recog. res.

math. expressions
with recog. res.

Figure 4: Initial character recognition and math-text separation described in Section 3.1.

“C” and “C” and “B” and “ ”) are considered as characters belonging to diﬀerent categories.
In this original recognition engine, a three-step coarse-toﬁne classiﬁcation strategy is employed for computational efﬁciency. At the ﬁrst step, a coarse classiﬁcation is performed
using low-dimensional features (aspect ratio and crossing
features). The features are extracted from every connectedcomponent (i.e., character) and then simply compared tothose of reference patterns prepared in the engine. At thesecond step, 36-dimensional directional features are used
to select 5 (or more) category candidates. At the ﬁnal
step, those candidates are ranked according to three different measurements (i.e., distances based on the above 36dimensional directional features, 64-dimensional peripheralfeatures, and 64-dimensional mesh features) and then those
three ranking results are uniﬁed by a voting method.
The recognition result of a character is not ﬁxed to one
category at this stage. Namely, each connected component
still has several category candidates. The recognition result will be ﬁnally ﬁxed in the following structure analysis
procedure (as discussed in Section 4).

3.2

Automatic correction of recognition resultsby clustering

In INFTY, a clustering technique is utilized in order toreduce misrecognitions. The clustering technique divides allconnected components (i.e., all characters and symbols inboth the ordinary text part and the mathematical expression parts) into several independent sets, called clusters, according to their shapes, and elects one representative, called
centroid, for each cluster. After the clustering, the connectedcomponents belonging to the same cluster are forced to havethe same recognition results by majority voting within thecluster. As this result, the connected components initiallymisrecognized due to slight shape diﬀerences will be recognized into their correct category. On the other hand, if
a cluster contains connected components of diﬀerent categories, new misrecognitions are induced by the voting. In order to avoid this side-eﬀect, the clustering should be “mild”
so that each cluster only contains the connected componentsof the same category.
The clustering technique employed in INFTY is based ona sequential appending and splitting strategy. After the
initial recognition, each connected component is appended
to its nearest cluster one after another. During this appending procedure, the cluster whose variance exceeds somethreshold is split into two independent clusters. This simple
clustering technique is far faster than conventional iterative
clustering techniques, such as k-means method. Since eachcluster should contain the connected components belonging
to the same category as noted above, the threshold should
be set to a small value.
Note that the clustering algorithm plays another important role in INFTY, i.e., the reduction of manual error correction operations. This role will be discussed in Section 5.

4.

STRUCTURE ANALYSIS OF MATHEMATICAL EXPRESSIONSIn this section, the procedure for the structure analysis ofmathematical expressions (Fig. 3–3 ), the third procedure
in INFTY, is described. The connected components classiﬁed as the characters and the symbols in mathematical expressions by the preceding character recognition procedure
are subjected to this procedure.
This procedure has three roles in INFTY. The ﬁrst role
is to represent the structure of each mathematical expression by a tree for converting the mathematical document

center-band(a)(7,11,Horizontal,100)
7

3

(b)

4

1

9
6

sub-/super-scripts(10,11,RSupScript,10)

8

5

2

normalizedbounding box

11

Figure 7: Center-band of line.
10

recognition result to be corrected(highlighted by mouse dragging)(c)Figure 5: (a) A mathematical expression, (b) its digraph representation, and (c) the search graph forfinding the minimum-cost spanning tree of the digraph (b).

c / h1h1cnormalizedbounding box(a)no linksuper-scripth20horizontal
sub-script1

h2/ h1(b)

Figure 6: (a) Three values h1 , h2 , and c and (b) labelling conditions based on those values.

into XML, LaTeX, and other math-description formats. The
second role is to ﬁx the character recognition result of themathematical expressions. The third role is to detect theordinary texts mis-classiﬁed as some mathematical expressions.
In order to simplify the structure analysis, each fraction isdecomposed into its numerator and denominator in advance.
This decomposition can be done rather easily since fractionlines in the mathematical expressions can be detected fromits location, shape, and size. The mathematical expression√
in a root sign (“ ”) is also extracted from the sign before
applying the structure analysis.

4.1

Network representation of mathematicalexpression

The problem of the structure analysis is represented asa minimum-cost spanning-tree problem on a weighted digraph, or network. In Fig. 5(a) and (b), a mathematicalexpression and its digraph representation are shown, respectively.
Each node of the digraph corresponds to a category candidates of a character in the subjected mathematical expression. (As noted in Section 3.1(c), each character still hascorrection
candidates

original
image

pop-up window on
manual specification

Figure 8: Snapshot of manual error correction on
INFTY. The misrecognition “L” → “L” is to be corrected.

several candidates as its recognition result.) For example,
node 1 and node 2 of the digraph of Fig. 5(b) correspondto the two category candidates of the character “x”. A link,
or a directed path, is prepared between two nodes if theircorresponding characters are adjacent and satisfy two conditions, whose details will be discussed later. Each link l isrepresented by a four-tuple, (parent, child, label, cost), whereparent and child are the parent and the child nodes of thelink l, respectively. Generally, the character corresponding
to parent lies in the left side of child. For example, at thelink between “x” and “2” of “x2 ”, the node of “x” will be
the parent node 1 . The element label represents the positional relation between parent and child. There are 9 typesof the label. For example, “Horizontal” is the label indicating that child and parent are adjacent horizontally, and“RSubScript” is the label indicating that child is a right subscript of parent. The element cost is a value proportional toan uncertainty level of the link l.
As noted above, every link should satisfy two conditions.
The ﬁrst condition restricts the relative position of the two
characters corresponding to parent and child nodes. This
relative position is evaluated using three values, h1 , h2 , and1
Left subscripts, such as “m” of “m Cn ” are ﬂipped as rightsubscripts in advance to building the digraph.

Table 1: Detail of database for experimental evaluation. (1)#categories examples#components in database (26 articles, 476 pages)
normal touchingbrokentouching
total& brokenaccent14 ˆ ˜ ¯ ¨ ` ˇ
2,483
44
2
25
2,554
arrow
16 ←↔←
926
4
0
0
930
 
bigsymbol
14
1,230
2
0
0
1,232
blackboard bold
52

414
8
0
0
422
calligraphic
52

{
547
0
0
0
547
german
52
955
0
89
0
1,044
greek
40
γδθαβγ12,400134
66
34
12,634
italic52
abcabc108,3142,172
1,044
433 111,963
numeric
1001234526,85610871
18
27,053operator
83 + − ×/ <19,877137
73
27
20,114others44 #%∞∀∃†
9,287
279
191
27
9,784
parenthesis18( ){ }[ ]36,93277762
43
37,814point
13, . ‘’
26,99114547
35
27,587roman
52
abcabc435,5978,599
1,138
268 445,602
script
52
3
0
0
0
3
total
564
682,812
12,2783,283
910 699,283(97.64%)(1.76%) (0.47%)(0.13%)(100%)
type

Table 2: Detail of database for experimental evaluation. (2)part#components in database (26 articles, 476 pages)
normal touching broken
touching
total& brokenordinary text 534,750
10,0181,717
655 547,140
math. exp.
148,0622,260
1,566
255 152,143
total
682,812
12,2783,283
910 699,283

c (Fig. 6 (a)). The values h1 and h2 are the heights of normalized bounding boxes of those two characters respectively,
where the normalized bounding box is a rectangle covering acharacter expanded with imaginary ascender and descender.
The value c is the vertical diﬀerence between the two normalized bounding boxes. A link is prepared if (h2 /h1 , c/h1 )falls on one of three regions of Fig. 6(b) (shown as ellipsoids) predetermined by observing actual mathematical expressions. The label of the link is also decided at this time.
For more details of this condition, see [5].
The second condition restricts the label of each link using
the absolute vertical position of the corresponding characters. In this condition, a center-band (Fig. 7) is utilized.
The center-band is a horizontally elongated region containing neither ascender nor descender. If the center-band iscovered by the normalized bounding box of a character (e.g.,“f ”, “(”, “x” in Fig. 7), the character is considered as neither a subscript nor a superscript. Thus the link whose
element child is such character is not allowed to have thelabel “RSubScript”.

4.2

Algorithm for optimal structure analysis
resultThe structure analysis problem is now can be consideredas the problem of searching for the minimum-cost spanningtree on the weighted digraph prepared in the previous section. Our “spanning-tree” is somewhat peculiar one because

it should consist of only nodes which do not correspond to
the same character. For example, any spanning-tree of the
digraph of Fig. 5(b) will not include node 9 (“z”) and node
10 (“2”) simultaneously because those nodes correspond to
the same character “z” in Fig. 5(a). The tree by the thickened links in Fig. 6(b) is an example of the spanning-tree.
Again, our structure analysis problem can be consideredas the problem to ﬁnd the minimum-cost spanning-tree fromall possible spanning-trees. While any spanning-tree corresponds to a consistent structure of the mathematical expressions, the minimum-cost one may be the most reliable structure. Unlike the minimum spanning-tree problem, where thenumber of links is to be minimized, there is no well-known algorithm for our somewhat peculiar problem. Thus, we havedeveloped a breadth-ﬁrst search algorithm (with a pruning
technique). In the algorithm, the search graph of Fig. 5(c)is built from the network of Fig. 5(b) and then its right-toleft path corresponding to the minimum-cost spanning-tree
is searched for. For the details of the algorithm, see [5].
When the minimum-cost spanning-tree of the weighted digraph is obtained, the structure of the subjected mathematical expression is represented as a (spanning-) tree and thecharacter recognition result is ﬁxed. Thus, the ﬁrst and thesecond role noted at the beginning of this section are fulﬁlled
by this procedure. On the other hand, if the spanning-tree
is not obtained, the mathematical expression is rejected as
an ordinary text. Thus, the third role can be fulﬁlled along

(a)

(b)

Table 4: Relation between the abnormal character ratios and the recognition rates (of all normal
and abnormal characters) in mathematical expression part.
abn. char. ratio
in math part(%)# articles
ave. recog. rate
in math part(%)

<1

1∼3

3∼10

>10

total

6
97.70

9
96.60

9
93.00

2
82.42

26
95.18

(c)
Figure 9: Three types of abnormal characters.
(a) Touching characters, (b) broken characters, and(c) touching and broken characters in mathematicalexpressions. Broken points are indicated by arrows.

Table 5: Character recognition rate of each character/symbol type. In this evaluation, abnormal
characters (i.e., touching/broken characters) were
excluded.
type

Table 3: Character recognition results.
partrecognition rate (%)
best
worst
average overarticle article
26 articles
ordinary text
99.81
98.51
99.44
math. exp.
98.19
79.31
95.18
total(text+math)
99.68
93.79
98.51

with this procedure.

5.

MANUAL ERROR CORRECTION

The manual error correction procedure (Fig. 3–4 ) is thelast and optional procedure of INFTY. The errors of thecharacter recognition and the mathematical structure analysis can be manually corrected using a graphical user interface. Figure 8 is a snapshot where a character recognition
error (“L”→ “L”) is likely to be corrected. This correction
is done by simply selecting the category out of correction
candidates (e.g., three candidates “L”, “R”, and “L” areshown in Fig. 8) while watching the document image. If
the correct category is not be found in those candidate, thecorrect result can be manually speciﬁed by keyboard input(or table pick-up) on a pop-up window.
The distinctive feature of this error correction procedureis that the result of the clustering performed in the character recognition procedure is exploited to reduce the manualerror correcting operations. Speciﬁcally, if the recognition
result of a connected component belonging to a cluster ismanually altered to its correct category, the recognition results of the other connected components belonging to thesame cluster are also altered automatically to the same category. Thus, similar misrecognitions can be corrected by
one operation, the correction operations can be reduced signiﬁcantly in total.

6.

EXPERIMENTAL EVALUATION

6.1

Database

A recognition experiment for the quantitative and thequalitative evaluations of INFTY was performed on 26 sci-

accentarrow
bigsymbol
blackboard boldcalligraphic
germangreek
italicnumeric
operatorothersparenthesis
point
roman
scripttotal#(normal)characters
2,483
926
1,230
414
547
955
12,400108,31426,856
19,877
9,287
36,932
26,991
435,5973
682,812recog
rate(%)
96.46
98.60
75.69
78.99
78.98
66.18
97.05
97.81
97.80
97.10
92.38
99.51
98.92
99.84
0.00
99.01

entiﬁc articles; 25 English papers from 15 pure mathematical journals (e.g., Bulletin of American Math. Soc., Bulletin de la Soc. Math. France, Mathematische Annalen,
and Kyushu J. Math.) issued in 1960–1990 and one bookon physics. The number of total pages was 476 (443 pages
from the mathematical journals + 33 pages from the bookon physics). The number of the subjected connected components was about 700,000 (547,140 from ordinary texts andabout 152,143 from mathematical expressions). The articles were scanned at 600dpi. For each character and mathsymbol, its ground truth (correct category) was attachedmanually.
The details of the all connected components are shown in
Table 1 and Table 2. The database contains various kindof characters peculiar to mathematical documents, such as
Greek alphabets (α, β, . . . ), calligraphic typeface (A, B, . . . ),and German typeface ( , , . . . ), in
addition to mathematical symbols, such as big symbols ( , ) and mathematicaloperators (+, ×).
As shown in Table 1, there are many abnormal characters, i.e., (a)touching characters, (b)broken characters, and(c) touching and broken characters, in the database (Fig. 9).
Most of the abnormal characters in the ordinary text part



6.3

p−1



J≈1
pE Pi

Kt∗

I∗GΥ


U

,

Figure 10: Misrecognitions in mathematical expressions.

Figure 11: Scanned image of mathematical expressions analyzed perfectly.

are correctly recognized by the commercial OCR engine, because the lexicon employed in the engine will help to correct
the misrecognitions of such abnormal characters. On theother hand, most of the abnormal characters in the mathematical expression part are misrecognized by the present INFTY. Thus, the detection and the separation/concatenationof the touching/broken characters are left as urgent futuretasks. Among the tasks, the separation of the touching characters in the mathematical expressions is troublesome. Thisis because those touching characters might be touching diagonally or vertically (Fig. 9(a)) and therefore can not be
separated into single characters by the conventional segmentation technique (such as projection based techniques [6, 7])
where the characters are assumed to be touching horizontally. One idea for this task can be found in [8].
In the following experiments, only matrices were manuallydetected and excluded as ﬁgure/table areas. The character
recognition and structure analysis of the matrices are very
challenging problems themselves and their implementationis left as a future work. Recently, Kanahori and Suzuki[9]have proposed a structure analysis technique of various matrices.

6.2

Reference patterns

Reference patterns used in the character recognition of themathematical expression part (Section 3.1(c)) were manually prepared. The reference patterns were collected fromhard-copies of LaTeX documents and scanned page imagesof mathematical books independent of the above database.
The number of the reference patterns of each category isoften 2 or more in order to manage the variations in character sizes (there are sub-/super-scripts in the mathematicalexpressions) and shapes.
Reference patterns used in the character recognition of
the ordinary text part (Section 3.1(a)) are unknown because
they are concealed in the commercial OCR engine.

Accuracy of character recognition

As shown in Table 3, the character recognition rate averaged over the 26 articles was 98.51% (without manual error
correction) and the recognition rates on the ordinary text
part and the mathematical expression part were 99.44% and95.18%, respectively. Those evaluations were very strict; thecharacters recognized in wrong font-types (e.g., “L”→“L”
and “L”→“L”) were counted for misrecognitions as wellas the characters recognized to their hardly-distinguishable
characters (e.g., “o”→“O” and “l”→“1”). In spite of thestrict evaluations, the recognition rate on the ordinary text
part was considerably high and enough for practical uses.
On the other hand, the recognition rate on the mathematical expression part should be improved in our future work,
although the rate of 95.18% seems to be considerably higher
than or comparable to recent mathematical OCR systems(such as [10, 11]).
The degradation of the recognition rate on the mathematical expressions was mainly due to (1) abnormal characters.
As noted in 6.1, there are many abnormal in our database(over 2% of all characters) and most of the abnormal characters in the mathematical expression part are to be misrecognized. Table 4 shows the relation between the ratio of
the abnormal characters over all the characters in the mathematical expression part and the character recognition rateof the part. Clearly, the recognition rates of the articles withmany abnormal characters are low. Thus, the separation/
concatenation of the touching/broken characters will be very
eﬀective to attain higher recognition rates especially in themathematical expression part. It is worth to note that if allabnormal characters are excluded from the evaluation, theoverall character recognition rate increases from 98.51% to99.00%.
In addition to the abnormal characters, (2) heavy size
variations (sub-/sup-scripts, bigsymbols), (3) large categories(English and Greek alphabets, operators, parentheses, etc.),(4) font variations (italic, calligraphic, etc.), and (5) the existence of similar characters (e.g., r, γ, and Υ) also induce
the misrecognitions in the mathematical expressions. Figure 10 shows several misrecognitions due to these reasons.
Among those 5 misrecognitions, the last two are due to double sub-/super-scripts (i.e., a sub-script of a sub-script anda super-script of a super-script), which are one of the mostserious causes of the misrecognitions.
Table 5 shows the character recognition rate of each character/symbol type. In this evaluation, all the abnormal
characters were excluded. The result shows that charactersof some special font-types (often complex and infrequent asshown in Table 1) are hard to be recognized.
The clustering technique of Section 3.2 was performed on
every article for the automatic correction of the initial recognition results given by the procedure of Section 3.1. From
our rough observation, it was shown that misrecognitions
were reduced to 2/3 by the use of the clustering technique.

6.4

Accuracy of structure analysis of mathematical expressions

Among 12,423 mathematical expressions 2 in 476 pages,
11,194 expressions were perfectly analyzed, where the term
2
Mathematical expressions with a single character, such as“x”, were disregarded in this evaluation. In addition, onedimensional mathematical expressions, such as “a = b” and“x × y + z”, were also disregarded here.

that the operations could be reduced to about 1/3 by theautomatic error correction using the result of the clustering.

(a)

(b)

ΘDη (c)
cn
P −1

7.
αn−1
0
(n − 1)!

Figure 12: Failure results of math-structure analysis.

I97I]

UNIVERSITY

Figure 13: Ordinary texts mis-detected as mathematical expressions.

“perfect” means that both the character recognition result
and the structure analysis result are correct. Thus, the perfect analysis rate was 89.6%. Figure 11 shows two perfectly
analyzed results.
The failure results were mainly due to the misrecognitions of their component characters. In fact, there were9602 mathematical expressions without misrecognitions andtheir perfect analysis rate was 97.9% (i.e., far higher than
89.6%). The double sub-/super-scripts also induce failure
results. Figure 12(a) shows a failure result due to the double sub-/super-scripts. The double sub-script “η” is wrongly
analyzed as one of sub-scripts of “Θ”. Namely, the slight
diﬀerence between the baselines of “D” and “η” was notdistinguished during the structure analysis. Another typical failure result is shown in Figure 12(b). In this result,the misrecognition “C”→“c” badly aﬀects the analysis of
the structure of “CP n−1 ”. Thus, the reduction of the misrecognition is an essential task for higher accuracy.
There were several ordinary texts mis-detected as mathematical expressions. Figure 13 shows two examples. In
the left example, a ﬂuctuation of its baseline causes the misdetections. In the right example, “small capitals” (“NIVERSITY”) were detected as sub-scripts of “U” in the veriﬁcation process in the character recognition procedure. The
number of such mis-detections were 1,249 in all the ordinary text parts (which contains 547,140 characters) and often found in the running headers and the footers.

6.5

Computation time

INFTY requires about 10 s/page to perform all procedures (except for the manual error correction procedure) on
a desk top PC (Pentium III, 1GHz) and about 20 s/page on
a notebook PC (Pentium III, 700MHz). This computation
time is very comparable to the computation times of (commercial) OCR softwares which can not handle mathematicalexpressions.

6.6

CONCLUSION

An integrated OCR for mathematical documents, called
INFTY, was presented. In INFTY, several novel techniques,
such as simultaneous character recognition and math-text
separation based on two complementary recognition engines,
are utilized for better recognition performance. From experimental results on 476 pages of mathematical documents it
was shown that suﬃcient character recognition rates (99.44%
on ordinary texts, 95.18% on mathematical expressions, and98.51% in total). It was also shown that 89.6% mathematical expressions are perfectly analyzed, i.e., no error in bothcharacter recognition and structure analysis.

Manual error correction

Manual error correction was performed on several articles
for evaluating the eﬀect of the clustering to reduce the number of error correcting operations by users. Its result showed

8.

ACKNOWLEDGMENTS

The authors greatly thank alumnae and alumni of Suzuki
Laboratory of Kyushu University for their generous eﬀorts
on the development of INFTY. The authors also thankToshiba Cooperation for their continuous support to our research project.
A beta version of INFTY can be downloaded freely. For
further information, please visit [12].

9.

REFERENCES[1] K.-F. Chan and D.-Y. Yeung, “Mathematicalexpression recognition : a survey,” Int. J. Doc. Anal.
Recog., 3(1):3–15, 2000.
[2] G. O. Michler, “Report on the retrodigitization project“Archiv der Mathematik”,” Archiv der Mathematik,
77:116–128, 2001.
[3] K. Dennis, G. O. Michler, G. Schneider, and M. Suzuki,“Automatic reference linking in distributed digital
libraries,” Proc. Workshop on Document Image
Analysis and Retrieval (DIAR-03), to appear, 2003.
[4] A. Kacem, A. Belaı̈d, and M. B. Ahmed,“EXTRAFOR: automatic EXTRAction ofmathematical FORmulas,” Proc. ICDAR, 527–530,
1999.
[5] Y. Eto and M. Suzuki, “Mathematical formula
recognition using virtual link network,” Proc. ICDAR,
762–767, 2001.
[6] R. G. Casey and E. Lecolinet, “A survey of methods
and strategies in character segmentation,” IEEE Trans.
Pat. Anal. Mach. Intell., 18(7):690–706, 1996.
[7] Y. Lu, “Machine printed character segmentation – anoverview,” Pattern Recognition, 28(1):67–80, 1995.
[8] A. Nomura, K. Michishita, S. Uchida, and M. Suzuki,“Detection and segmentation of touching characters inmathematical expressions,” Proc. ICDAR, to appear,
2003.
[9] T. Kanahori and M. Suzuki, “A recognition method of
matrices by using variable block pattern elements
generating rectangular area,” Graphics Recognition.
Algorithms and Applications (Lecture Notes in
Computer Science, 2390), Springer-Verlag, 2001.
[10] J. -Y. Toumit, S. Gracia-Salicetti, and H. Emptoz, “Ahierarchical and recursive model of mathematicalexpressions for automatic reading of mathematical
documents,” Proc. ICDAR, 119–112, 1999.

[11] H.-J. Lee and J.-S. Wang, “Design of a mathematicalexpression understanding system,” Pattern Recognition
Letters, 18(3):289–298, 1997.
[12] http://infty.math.kyushu-u.ac.jp