This article appeared in a journal published by Elsevier. The attachedcopy is furnished to the author for internal non-commercial researchand education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of thearticle (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies areencouraged to visit:
http://www.elsevier.com/copyrightAuthor's personal copy

Available online at www.sciencedirect.com

Pattern Recognition Letters 29 (2008) 1326–1332
www.elsevier.com/locate/patrec

Mathematical symbol recognition with support vector machines
Christopher Malon a,*,1, Seiichi Uchida b, Masakazu Suzuki abaFaculty of Mathematics, Kyushu University, Hakozaki 6-10-1, Higashi-ku, Fukuoka 812–8581, Japan
Faculty of Information Science and Electrical Engineering, Kyushu University, Motooka 744, Nishi-ku, Fukuoka 819-0395, Japan

Received 23 October 2006; received in revised form 25 January 2008
Available online 10 March 2008
Communicated by T. Tan.

Abstract
Single-character recognition of mathematical symbols poses challenges from its two-dimensional pattern, the variety of similar symbols that must be recognized distinctly, the imbalance and paucity of training data available, and the impossibility of ﬁnal veriﬁcation
through spell check. We investigate the use of support vector machines to improve the classiﬁcation of InftyReader, a free system for
the OCR of mathematical documents. First, we compare the performance of SVM kernels and feature deﬁnitions on pairs of letters that
InftyReader usually confuses. Second, we describe a successful approach to multi-class classiﬁcation with SVM, utilizing the ranking ofalternatives within InftyReader’s confusion clusters. The inclusion of our technique in InftyReader reduces its misrecognition rate by 41%.
Ó 2008 Elsevier B.V. All rights reserved.
Keywords: Support vector machine; OCR; Mathematical document; Mathematical symbol recognition

1. Introduction
Mathematics is the universal language of the scientiﬁcliterature, but a computer may ﬁnd it easier to read thehuman language in which surrounding text is written.
The failure of conventional OCR systems to treat mathematics has several consequences: Readers of mathematical documents cannot automatically search for earlier occurences of a variable or operator, in tracing the notation and deﬁnitions used by ajournal article.
 The appearance of mathematics on the same line as text
often confounds OCR treatment of surrounding words.
 Equations can only be represented as graphics bysemantic transformation systems, such as those converting digital documents into braille for accessibility byblind readers (Suzuki et al., 2004).
*

Corresponding author. Tel.: +1 609 951 2594; fax: +1 609 951 2482.
E-mail addresses: malon@nec-labs.com (C. Malon), uchida@is.
kyushu-u.ac.jp (S. Uchida), suzuki@math.kyushu-u.ac.jp (M. Suzuki).
1
Present address: NEC Laboratories America, 4 Independence Way,
Princeton, NJ 08540, USA.
0167-8655/$ - see front matter Ó 2008 Elsevier B.V. All rights reserved.
doi:10.1016/j.patrec.2008.02.005

Mathematical OCR was investigated as early as 1968
(Anderson, 1968); a survey of its diﬃculties and previous
approaches may be found in (Chan and Yeung, 2000). A
modern system competing with ours has achieved a93.77% single-character recognition rate (Garain et al.,2004). OCR of mathematics diﬀers markedly from typicaltext recognition because its single-character recognition
phase must be followed by a structural analysis phase, inwhich symbol relationships involving superscripts, subscripts, fractions, and matrices must be recovered. The
two-dimensional arrangement aﬀects not only structural
analysis but single-character recognition itself, because typical assumptions about bounding boxes and baselines are
violated. Even in relatively simple equations such as/jC ðzÞ ¼ expðzN / Þ
the subscript-positioned capital blackboard bold C, whosebase is nearly aligned with that of the vertical bar, might bemistaken for a lower-case letter.
In this paper, we focus on the single-character recognition phase that precedes structural analysis. We haveaddressed structural analysis in (Eto et al., 2001; Murakami et al., 2002; Kanahori et al., 2003; Suzuki et al., 2003).

Author's personal copy

C. Malon et al. / Pattern Recognition Letters 29 (2008) 1326–1332

The single-character OCR of mathematics poses challenges
that, if not unique, place it alongside the most diﬃcult
human languages to recognize. The recognition problem
consists of about 1000 classes, many with little existing
ground truth data. Certain distinct letters, such as Latinv and Greek m, are in close resemblance. Most unusually,
we desire the distinction of styles.
In typical mathematical usage, diﬀerent styles of thesame letters will have completely diﬀerent meanings. The
problem is most severe not in engineering, but in puremathematics. For example, within a single article in p-adic
representation theory, the bold letter G often will representa group over an algebraically closed ﬁeld, the plain italic G
will represent its rational points over a p-adic ﬁeld k, andsans-serif G a reductive quotient over the residual ﬁeld k,with German g used for a Lie algebra. Calligraphic A
may represent a simplicial complex, and italic A a torus.
(see, e.g., DeBacker, 2006). An optical character recognizerthat does not keep these letters distinct would be practicallyuseless in this branch of algebra. However, within a single
style, fonts (Computer Modern, Times, Helvetica, etc.)
should not be distinguished, so that mathematical formulas
can be compared between articles, regardless of the fonts
the publisher has chosen.
OCR problems were considered very early in the development of SVM, with promising results. An experiment by
Cortes and Vapnik (1995) achieved 95.8% accuracy on
handwritten digits in the US Postal Service database. Moreparticularly, in character recognition of human languages
with hundreds of distinct characters, SVM have achieved
promising results, for example, in handwritten Chinese(99.0%, Dong et al., 2005) and printed Ethiopian (91.5%,
Meshesha and Jawahar, 2005). Recently, SVM has been
applied to handwritten mathematics on a blackboard(Tapia et al., 2003), but to our knowledge, OCR of printed
mathematics using SVM has not been investigated before.
This paper describes an experiment using SVM toimprove multi-class classiﬁcation by an existing OCR system. This OCR system is a puriﬁed version of the InftyReader, a freely available OCR engine for mathematics,
described in Suzuki et al. (2003). First, we study the abilityof various kinds of SVM, as binary classiﬁers, to distinguish pairs of letters that confuse InftyReader. Then, weshow how the classiﬁers may be integrated with the system
to improve its multi-class classiﬁcation ability.
2. Ground truth data
The InftyProject deﬁned a set of 1629 mathematicalcharacters to be distinguished, and released several databases of ground truth, containing both single-characterand structural recognition results, starting withInftyCDB-1 (Suzuki et al., 2005), whose composition is
described in (Uchida et al., 2005). Because some mathematical symbols occur very rarely, it is necessary to choose
between extracting each symbol from documents in their
entirety, or seeking out samples of particularly rare charac-

1327

ters to provide more uniform representation. The newest
databases of the InftyProject, InftyCDB-3-A and
InftyCDB-3-B (Suzuki et al., 2007), targeted at single-character recognition experiments, cover both approaches.
InftyCDB-3-B represents 20 articles from advanced mathematics journals at full length; it consists of a tenth of
the samples of InftyCDB-1, chosen by clustering techniques. InftyCDB-3-A (Suzuki, 2006) aims to represent
rare characters by more samples; it includes not only journal articles, but font samples, and multiple scans of lettersat diﬀerent greyscale thresholds. We use InftyCDB-3-A(188,752 characters, representing 384 symbol entities from
326 documents) for training, and InftyCDB-3-B (70,637
characters, representing 275 symbol entities from 20 documents) for testing. No database includes samples of all
1629 symbol entities deﬁned by the Infty Project.
In InftyCDB-3-A and InftyCDB-3-B, a sample ofground truth data for a symbol entity consists of a black
and white bitmap image of that symbol in isolation, framed
inside its bounding box, extracted from a scanned physicaldocument. Thus, the data set does not test the OCR system’s ability to group nearby components together, as in
the two parts of the symbol ‘6’. Because spatial context
is lost, some pairs of symbols, such as hyphens and underscores, must be regarded as the same. InftyReader distinguishes among these characters after an entire line of text
or mathematical expression was read. Also, light or dark
printing can aﬀect whether a character should be regarded
as bold or not; InftyReader makes such decisions after thedensity of all characters on the page is known. Thus, boldcharacters are thus identiﬁed with their nonbold counterparts. German letters, which number too few, and touching
and broken characters, are excluded from our training andtesting data. 2
In Table 1, we present representatives of the 384 symbol
entities appearing in InftyCDB-3-A. Fig. 1 shows the number of training samples available for each of these classes.
Although an average class has between 500 and 1000 training samples, more than 75 classes have fewer than 50 training samples.

3. Confusion matrix
The engine of InftyReader typically makes use of contextual information, but for this experiment, we distill it
to ignore information about a character’s size or surrounding characters. The puriﬁed engine simply classiﬁes images
framed inside bounding boxes. By running the puriﬁed
InftyReader engine on the training data, we produce aninteger-valued confusion matrix, with rows that countground truth and columns that count recognition results.
2

In the earlier ground truth database InftyCDB-1, touching characterscomprise 1.25% of the character samples. We address the segmentation oftouching characters in (Malon et al., 2007), with a strategy that only needs
to query SVM that are trained to recognize correctly segmented input
images.

Author's personal copy

1328

C. Malon et al. / Pattern Recognition Letters 29 (2008) 1326–1332

Table 1
Classes represented in InftyCDB-3-A

80
70

Upright Greek
Calligraphic

Number of classes

Upright Latin
60
50
40
30
20
10

Blackboard
Bold

0

0

500

Punctuation
Brackets

1000
1500
2000
Number of samples

2500

3000

Fig. 1. Histogram of number of training samples, by class.

Accents
120

ArrowsBinaryoperators
Relational
operators

Big symbols
Other symbols

Italic Latin

Italic Greek

Number of classes

100
80
60
40
20
0

0

5

10

15
20
Size of cluster

25

30

Fig. 2. Sizes of confusion clusters.

This ordering will be utilized in our multi-class classiﬁcation strategy later.
4. Pairwise classiﬁcation with SVM

Italic ligatures

4.1. Features for SVM training
Upright
ligatures

Every nonzero oﬀ-diagonal entry of this matrix represents
a confusing pair, for which an SVM should be trained.
There are 771 confusing pairs, counted as unordered pairs.
In the confusion matrix, each row represents Infty’s recognition result and each column represents ground truth.
The set of nonzero entries from each row of the confusion
matrix represents a confusion cluster. The sizes of theseclusters are indicated in Fig. 2. Most clusters consist of
fewer than ﬁve alternatives, and the biggest cluster contains
26 alternatives. As the ﬁgure shows, the confusion matrix is
relatively sparse, and performing multi-class classiﬁcation
only on confusing alternatives, instead of all 384 symbols,
signiﬁcantly reduces complexity. Each cluster can be partially ordered by the likelihoods of each alternative, as indicated by the values of the corresponding matrix entries.

Some of our SVM are trained with directional histograms of the contour, introduced by Kimura et al. (1997)
for Japanese handwriting recognition. For a single mesh,these feature vectors are constructed from directional histograms, measuring the amount of horizontal, vertical,
and diagonal contour, in each position of the mesh. A single pixel contributes to the mesh position in which it lies,
and possibly to several neighboring positions, as determined by mask functions over the bitmap. These maskfunctions sum to one everywhere in the bitmap.
The recognition engine for InftyReader uses diﬀerently
sized meshes, in case a character is especially tall or short.
Depending on a character’s aspect ratio, data from a 3  5,
5  5, or 5  3 mesh may be more signiﬁcant. Our 221dimensional ‘‘directional” feature vectors consist of the
aspect ratio, and three blocks, representing the directional
contour histograms from the three mesh sizes. Dependingon the character’s aspect ratio, data from only one orAuthor's personal copy

C. Malon et al. / Pattern Recognition Letters 29 (2008) 1326–1332

two of the blocks will be regarded as signiﬁcant and used
for training or testing. For example, an input ‘l’ that has
aspect ratio exceeding 1.7 will use only the 5  3 block,and the aspect ratio, for classiﬁcation.3
In Vapnik (1995), Vapnik states the philosophy that, incontrast to classical approaches that work best with‘‘strong features”, ‘‘it is not important what kind of ‘weakfeature’ one uses; it is more important to form ‘smart’ linear combinations”. As an extreme example of this philosophy, Cortes and Vapnik’s original SVM study of the USPS
handwritten digit database (Cortes and Vapnik, 1995) utilizes (smoothed, centered, de-slanted) bitmap images asfeature vectors. Bitmaps as feature vectors, sometimes processed by principal component analysis, linear discriminant
analysis, or nonlinear normalization, also have been thebasis of more modern OCR experiments with SVM (Donget al., 2005; Chang et al., 2004; Meshesha and Jawahar,2005).
To investigate whether the style-but-not-font distinction
aspect of our recognition problem makes bitmap-based
approaches less eﬀective, or rather if directional features
discard too much potentially useful information, we train
another set of SVM with bitmap-like feature data. Becausecharacters appear in bounding boxes of diﬀerent aspectratios, we cannot use raw bitmaps directly. Rather, we
impose a 20 by 20 grid onto each bitmap, and measure
the blackness in each grid position. Taking these measurements together with the arctangent of the aspect ratio, we
obtain 401-dimensional ‘‘density” feature vectors.
4.2. Benchmark: A naive classiﬁer
Ideally, we would compare performance of the SVM
against the pure Infty recognizer itself. However, the pure
Infty recognizer does not solve a binary classiﬁcation problem like the SVM classiﬁers do. We can only say that therate at which it picks a class A over a class B in binary
selection should be greater than the rate at which it selects
A out of all possible classes, and vice versa. These twobounds typically yield an interval too wide to be informative, so we implement a naive binary classiﬁer as a more
precise benchmark.
The naive classiﬁer is constructed by recording the centroids of the sets of feature vectors representing instancesof each symbol in the training data. We use the directionalfeature vectors for this construction. The naive classiﬁer
can perform either multi-class or binary classiﬁcation; inany case, it assigns a test sample to the class with the closest
centroid.

3

The SVM uses the 221-dimensional feature vector, with the insignificant blocks’ coordinates changed to zero, for training and testing. The
naive classiﬁer assigns a testing sample to a class by ﬁnding the nearest
centroid in the feature subspace consisting of blocks that are signiﬁcant forthat sample. Within a single class, not every training sample will have thesame signiﬁcant blocks, so the centroids for each block are computed
separately.

1329

4.3. SVM training and performance
Altogether, we consider ﬁve forms of SVM constructions. On the directional features, we construct SVM withlinear, Gaussian, and cubic polynomial kernels. On thedensity features, we construct SVM with linear and cubic
polynomial kernels. These kernels have the forms:
K linear ð~x;~yÞ ¼ ~x ~yK Gaussian ð~x;~yÞ ¼ e

ð1Þ

ck~
x~
yk2

x;~
yÞ ¼ ðc~x ~y þ 1Þ
K cubic ð~

ð2Þ
3

ð3Þ

Support vector machines are trained to perform binaryclassiﬁcation by solving the following optimization problem. Given training data with feature vectors ~xi assigned
to class yi 2 {  1, 1} for i = 1, . . . , l, the support vector
machines solve
min
~
w;b;~
n

l
X
1
Kð~
w; ~
wÞ þ Cni
2
i¼1

subject toy i ðKð~w; ~xi Þ þ bÞ P 1  ni

ð4Þni P 0where ~
n is an l-dimensional vector, and ~w is a vector in thesame feature space as the ~xi (see, e.g., Hsu et al., 2003). The
solution determines the classiﬁerf ð~xÞ ¼ sgnðKð~w;~
xÞ þ bÞ:We use the LibSVM software Chang and Lin (2001) totrain SVM classiﬁers.
The performance of a binary classiﬁer f that assigns an
input vector ~x of label y to the class f ð~xÞ, may be measured
using the smaller of the recognition rates of the classes:
xÞ ¼ yjy ¼ cÞ
min P ðf ð~c¼1;1

We call this number the min-recall of the classiﬁer.
Before training an SVM, the soft margin C and any
parameters appearing in the kernel K must be chosen inadvance (here, c). For the linear and Gaussian kernel
experiments using directional features, we choose theseparameters by ﬁvefold cross-validation. Each training document is assigned, in its entirety, at random to one of ﬁvesets. For each binary classiﬁcation problem, the cross-validation accuracy for a choice of parameter values is computed by the leave-one-out method. Parameter choices
are inspected within a grid in logarithmic space, and thegrid is expanded until the accuracy stabilizes or begins
decreasing at all boundaries, or until an 8-min timeout.
The parameter choice producing the highest cross-validation accuracy is used once more to train the ﬁnal SVM
for the problem on the entire training set. This procedure
cannot be performed on a binary classiﬁcation problem if
all the training data for either class is concentrated in a single one of the ﬁve sets; for the 17 (of 771) pairs where wehave so little data, we do not construct a binary SVM.
In fact, the parameter choice is rarely important for the
linear SVM; up to the hardest soft margin considered,Author's personal copy

1330

C. Malon et al. / Pattern Recognition Letters 29 (2008) 1326–1332

accuracies typically remain the same, as one would expect
if the data were linearly separable. A softer hard margin
produces a 3% or greater improvement in min-recall onfour pairs, and the constant choice C = .01 produces thebest accuracies on training data overall. For the Gaussian
kernel, as well, there is a parameter setting that yields
cross-validation accuracies on each problem that are nearlyas high as if the assignment is allowed to vary with the
problem.
The binary classiﬁers are then evaluated on the testingdata set. In Tables 2 and 3, we compare their performanceagainst each other and the naive classiﬁer, for contour
directional features and for density features. These tables
give the percentage of confusing pairs on which each classiﬁer surpasses various min-recall thresholds. This evaluation is only carried out for the 528 confusing pairs forwhich both classes have at least 10 samples of testing data.
Using contour directional features, each SVM kernel
substantially exceeds the performance of the naive classiﬁer. The Gaussian kernel falls short of the performanceof the linear kernel. Feature choice matters greatly, perhaps because training samples are so scarce. With density
features, the linear SVM performs slightly worse than withdirectional features, but more complicated kernels perform
far worse, not even matching the naive classiﬁer’s benchmark. These kernels require more training time, so thatfewer kernel parameter choices can be tested within the
8 min training limit. The linear SVM with directional fea-tures is eﬃciently chosen, trained, and utilized, and is aseﬀective as the other classiﬁers, so we will use it as the basis
for the analyses and multi-class experiments in the following sections.
The confusing pairs on which the linear SVM achieves
the greatest improvement in min-recall over the naive classiﬁer are shown in Fig. 3. The most diﬃcult pairs for SVM
are shown in Table 4; as expected, many of these requirelarge fractions of training vectors for support. Remarkably, many distinctions are adequately learned by the linear
SVM without much training data. Even among the 53 confusing pairs with fewer than 25 samples in the smaller class,the average min-recall is 96.2%.

5. Multi-class classiﬁcation
By starting with a fast classiﬁer, we reduce our multiclass classiﬁcation problem from 1629 classes to the sizeof the confusion cluster of an Infty recognition result,which can vary as shown in Fig. 2. Popular methods of
combining binary SVM to perform multi-class classiﬁcation are reviewed in Hsu and Lin (2002), including amethod based on one-versus-all classiﬁers, and two meth-

Table 2
SVM performance on confusing pairs, using contour directional features
Min-recall

Naive
(%)

SVM linear(%)

SVM
Gaussian (%)

SVM
cubic (%)

>0
>.5
>.6
>.7
>.8
>.9
>.95
>.97
>.99
>.995
>.999

100.00
98.67
98.30
97.35
95.27
93.75
90.91
84.28
73.30
66.86
56.82

100.00
99.05
98.48
98.30
97.35
95.83
92.99
90.34
84.28
78.22
69.13

100.00
99.05
98.67
98.30
97.54
95.64
93.18
89.77
82.95
74.62
64.39

100.00
99.05
98.48
98.30
97.35
95.64
92.61
89.96
84.28
77.84
69.89

Fig. 3. Pairs on which the min-recall of linear SVM with directionalfeatures is at least 10% higher than that of the naive classiﬁer.

Table 4
Pairs with linear SVM min-recall below .80
Min-recall

x-recally-recall

0.10420.10420.9553

x

y

.3548

Max SV fraction

0.10420.12500.10421.00001.00000.1250

.1935
.1935

Table 3
SVM performance on confusing pairs, using density features

0.33330.42110.33330.42111.00000.9306

.3548
.2934

Min-recall

SVM linear (%)

SVM Gaussian (%)

SVM cubic (%)

>0
>.5
>.6
>.7
>.8
>.9
>.95
>.97
>.99
>.995
>.999

100.00
98.86
98.48
98.11
97.16
94.70
91.10
88.83
82.58
78.03
67.80

99.60
97.18
96.98
96.18
94.97
91.75
86.92
80.08
71.03
67.00
55.33

96.21
94.51
94.32
92.99
91.10
88.07
83.33
78.22
70.83
66.29
51.14

0.57500.58430.99070.58430.57500.8704

.0698
.1718

0.59090.59091.0000.1400

0.6136

0.61361.0000.1200

0.72220.72220.9297.3768

0.74070.74071.00000.98260.74070.7407

.1236
.0356

0.74070.99430.7407.0300

0.78330.95350.7833.2556

Author's personal copy

C. Malon et al. / Pattern Recognition Letters 29 (2008) 1326–1332

ods based on one-versus-one classiﬁers (the max-wins anddirected acyclic graph approaches). Each approach has
well-known drawbacks, and none is suited to utilize a priori
information about the likelihood of alternatives, though
the directed acyclic graph method requires an order forthe candidates to be chosen, whose implications are far
from obvious.
Any of these methods could be applied directly to a confusion cluster, but instead, we use a method that utilizes the
ranking of alternatives in a confusion cluster, to make itlikely that the most likely misrecognitions will be tested
with an SVM.
For an Infty recognition result i, the confusion cluster
C(i) of misrecognition candidates is partially ordered bylikelihood, as explained in Section 3. Let C0 (i) be the subsetof alternatives j 2 C(i) for which a binary SVM comparing jand i was constructed. After the pure Infty engine recognizes a character as i, our method starts to apply the
SVM for (j, i) for each j 2 C0 (i), starting with the most
likely j. When any j wins over i in the SVM classiﬁcation,
the testing is stopped, and j is reported as the classiﬁcation.
If no j wins, i is kept as the classiﬁcation.
This method requires us only to train SVM on confusing
pairs; other one-versus-one approaches would require us totrain SVM on all pairs of letters that appear together insome confusion cluster. Of course, testing complexity isalso linear in the number of letters in a cluster.
Without SVM, the pure Infty engine recognizes characters with 96.10% accuracy on our testing data set. Using
SVM by this method, the recognition rate rises to
97.70%, so that the number of misrecognized characters
falls by 41%.
When Infty makes the correct choice and our method
does not, it always means that an SVM’s decision was at
fault. If neither Infty nor our method chooses correctly,
three phenomena can explain the mistake. The SVM testing the Infty’s choice against the right alternative may havechosen the wrong result when it was reached (we count thecases where an SVM was not trained, because of insuﬃcient data, as such a case). The confusion of Infty’s guess
for the correct answer might not have occurred in the training data, so that the right alternative was not represented in
the confusion cluster; we call this situation an ‘‘unprecedented mistake”. The ﬁnal alternative is called ‘‘shadowing”. On an instance of testing data for which Infty
guesses i, and the correct answer is k, we say that an
SVM is ‘‘shadowed” if some other alternative j occurs
before k in the confusion cluster, and j defeats i, so thatthe i versus k classiﬁer is never run.
Altogether, the classiﬁcation on the 70,637 testing samples may be synopsized as follows: Infty right, output right: 67,100,
 Infty wrong, output right: 1912,
 Infty right, output wrong: 784,
 Infty wrong, output wrong, SVM wrong or not trained:
399,

1331

 Infty wrong, output wrong, unprecedented mistake: 280,
 Infty wrong, output wrong, SVM shadowed: 162.
If shadowing happened frequently, our multi-class strategy would be inappropriate, but this data shows that it
happens quite rarely.
6. Style distinction
One novel aspect of our single-character recognition
problem is the distinction of a letter in Roman, italic, calligraphic, and blackboard bold styles, regardless of its font.
The eﬃcacy of SVM on this aspect of the problem is compared to that of other techniques in Table 5.
The decrease in the number of confusing pairs meansthat the SVM can distinguish certain styles with 100%
accuracy that pose confusion to other classiﬁers. The total
number of style mistakes decreases from Infty to SVM by agreater margin than the misrecognition rate overall.
With occasional mistakes, the naive classiﬁer typically
can distinguish calligraphic and blackboard bold fromother styles. Its main weakness is the distinction of italic
characters. The linear SVM shows signiﬁcant improvement
Table 5
Style misrecognitions on testing data

Total number of confused pairs
Confused pairs representing style mistakes
Total number of misrecognitions
Style recognition errors

Naive

Infty

SVM

315
46
3,832
254

321
51
2,753
219

256
37
1,625
116

FailuresSuccesses

Successes

Failures

FailuresSuccesses

Successes

FailuresFig. 4. Classiﬁcation of the same letters in diﬀerent styles.

Author's personal copy

1332

C. Malon et al. / Pattern Recognition Letters 29 (2008) 1326–1332

in this regard. In Fig. 4, we display three italic pairs thatare markedly improved with SVM.
The only case where linear SVM performed remarkably
worse than the naive classiﬁer was in the distinction oflower case italic l from script lower case ‘.
7. Summary
We have demonstrated the eﬀectiveness of SVM on alarge multi-class problem, with many similar symbols andmany classes with little training data. The SVM managed
to learn many binary classiﬁcations well for which there
was a paucity of training data. Though all SVM kernels
provided about the same performance on directional features, the linear classiﬁer had superior performance on density features. Generally, SVM trained on directionalfeatures performed marginally better than SVM trainedon density features. The SVM excels at distinguishing stylesof characters, particularly italic and nonitalic variants,
which are indistinguishable to simpler methods using thesame sets of features.
We have integrated these SVM into the solution of alarge multi-class problem, by testing only pairs of symbols
mistaken by an existing OCR system. The complexity islow, and the most likely confused alternatives are preferredby the algorithm. The single-character misrecognition rateof the OCR system falls by 41% with the introduction of
SVM. We note that we do not omit pairs often regarded
as indistinguishable without size information (lower andupper case versions of C, O, P, S, U, V, X, and Z) in reporting our recognition rate.
Many of the mistakes that remain after the applicationof SVM represent characters that are truly indistinguishable without contextual information (such as the character’s size relative to surrounding characters), or that
represent degraded character images. We will try to
improve the use of contextual information in Infty, anddevelop better methods for the treatment of touching andbroken characters, in future work.
Acknowledgement
This work was supported by the Kyushu University 21stCentury COE Program, ‘‘Development of Mathematics
with High Functionality”.
References
Anderson, R., 1968. Syntax-directed recognition of hand-printed twodimensional mathematics. Ph.D. Thesis, Harvard University.
Chang, C.-C., Lin, C.-J., 2001. LIBSVM: A library for support vector
machines. <http://www.csie.ntu.edu.tw/%7Ecjlin/libsvm>.
Chang, F., Lin, C.-C., Chen, C.-J., 2004. Applying a hybrid method tohandwritten character recognition. In: ICPR ’04: Proc. 17th Internat.
Conf. on Pattern Recognition, vol. 2. IEEE Computer Society,
Washington, DC, USA, pp. 529–532.
Chan, K.-F., Yeung, D.-Y., 2000. Mathematical expression recognition: asurvey. IJDAR 3 (1), 3–15.

Cortes, C., Vapnik, V., 1995. Support-vector networks. Mach. Learn. 20(3), 273–297.
DeBacker, S., 2006. Stable distributions supported on the nilpotent cone
for the group G2. In: The Unity of Mathematics; Honor of the 90th
Birthday of I.M. Gelfand, Progress in Mathematics, vol. 244.
Birkhäuser, Boston.
Dong, J.-X., Krzyzak, A., Suen, C., 2005. An improved handwritten
Chinese character recognition system using support vector machine.
Pattern Recognition Lett. 26 (12), 1849–1856.
Eto, Y., Suzuki, M., 2001. Mathematical formula recognition using virtual
link network. In: ICDAR ’01: Proc. 6th Internat. Conf. on Document
Analysis and Recognition. IEEE Computer Society Press, pp. 430–437.
Garain, U., Chaudhuri, B.B., Ghosh, R.P., 2004. A multiple-classiﬁer
system for recognition of printed mathematical symbols. In: ICPR ’04:
Proc. 17th Internat. Conf. on Pattern Recognition. IEEE Computer
Society, Washington, DC, USA, pp. 380–383.
Hsu, C.-W., Chang, C.-C., Lin, C.-J., 2003. A practical guide to support
vector classiﬁcation. <ttp://www.csie.ntu.edu.tw/%7Ecjlin/papers/
guide/guide.pdf>.
Hsu, C.-W., Lin, C.-J., 2002. A comparison of methods for multi-class
support vector machines. IEEE Trans. Neural Networks 13, 415–425.
Kanahori, T., Suzuki, M., 2003. Detection of matrices and segmentationof matrix elements in scanned images of scientiﬁc documents. In:
ICDAR ’03: Proc. 7th Internat. Conf. on Document Analysis andRecognition. IEEE Computer Society, Washington, DC, USA, pp.
433–437.
Kimura, F., Wakabayashi, T., Tsuruoka, S., Miyake, Y., 1997. Improvement of handwritten Japanese character recognition using weighted
direction code histogram. Pattern Recognition 30 (8), 1328–1329.
Malon, C., Uchida, S., Suzuki, M., 2007. Separation of touching
characters using DP matching. In: IEICE Technical Report
PRMU2006: Proc. IEICE Conf. Pattern Recognition and Machine
Understanding, pp. 13–18.
Meshesha, M., Jawahar, C., 2005. Recognition of printed Amharic
documents. In: ICDAR ’05: Proc. 8th Internat. Conf. on Document
Analysis and Recognition. IEEE Computer Society, Washington, DC,
USA, pp. 784–788.
Murakami, M., Suzuki, M., 2002. Improvement of mathematical structural analysis by Center-Band. IEICE Technical Report PRMU2001270, pp. 203–210.
Suzuki, M., Kanahori, T., Ohtake, N., Yamaguchi, K., 2004. An
integrated OCR software for mathematical documents and its output
with accessibility. In: Computers helping people with special needs, 9thInternat. Conf. on ICCHP 2004, Paris (July 2004). Lecture Notes in
Computer Science, vol. 3119. Springer, pp. 648–655.
Suzuki, M., Malon, C., Uchida, S., 2007. Databases of mathematical
documents. Research Reports on Information Science and ElectricalEngineering of Kyushu University 12(1), 7–14.
Suzuki, M., Tamari, F., Fukuda, R., Uchida, S., Kanahori, T., 2003.
Infty: an integrated OCR system for mathematical documents. In:
DocEng ’03: Proc. 2003 ACM Symposium on Document Engineering.
ACM Press, New York, NY, USA, pp. 95–104.
Suzuki, M., Uchida, S., Nomura, A., 2005. A ground-truthed mathematical character and symbol image database. In: ICDAR ’05: Proc. 8thInternat. Conf. on Document Analysis and Recognition (Washington,
DC, USA, 2005). IEEE Computer Society, pp. 675–679.
Suzuki, M., 2006. InftyCDB-3: a ground truthed database of words/
formulae images, third distribution. <http://www.inftyproject.org/en/
database.html>.
Tapia, E., Rojas, R., 2003. Recognition of on-line handwritten mathematical formulas in the E-Chalk system. In: ICDAR ’03: Proc. 7thInternat. Conf. on Document Analysis and Recognition (Washington,
DC, USA). IEEE Computer Society, pp. 980–984.
Uchida, S., Nomura, A., Suzuki, M., 2005. Quantitative analysis ofmathematical documents. Internat. J. Document Anal. Recognition 7(4), 211–218.
Vapnik, V., 1995. The Nature of Statistical Learning Theory. SpringerVerlag, New York Inc., New York, NY, USA.