Offline ICR with Deep Learning
Ting XU , Wei MING

要旨

, Masahiro OZAWA

Abstract

手書き文字認識（Intelligent Character Recognition:

Handwriting recognition, or Intelligent Character Recognition

筆記体の単語単位の認識まで発展してきている。しかし，

acter to recognizing cursive whole words, which few com-

ICR）の研究は，一文字単位の英数字の認識から始まり，
現在でも実用的な商用システムは数えられる程度である。
しかも，これらの商用システムは性能を向上するために，辞書，または言語モデルを用いていることが多い。この「語彙内」制約により，ICR 技術を多くの実社会の文書（臨床記録，財務書類，流通の取引記録など）に適用する
上で制限が発生している。それは，辞書に登録されていない語句が頻繁に含まれることで，認識率の低下に繋がるためである。

mercial systems can recognize well. Moreover, these systems
are often guided by a dictionary or a language model toenhance performance. With the success of Deep Learning,many recent works on ICR extract handwriting features using
Convolutional Neural Networks (CNN) provided that all possible words are prescribed by a dictionary. This “in-vocabulary” constraint, however, prevents ICR’s application fromclinical forms, financial reports, logistic transaction records,

この問題に対処するために，辞書に登録されていない

手書きの語句を認識する ICR エンジンを開発した。これにより幅広い分野への ICR 技術の適用が可能になり，辞書に登録されていない名前，住所，略語，数字または通貨を含む文書に対しても，既存の商用システム以上の高精度な認識を実現できる。開発したエンジンの有効性

は，複数の適用分野で確認できている。RPA（Robotic

Process Automation）システムに統合することで，作業の生産性と従業員の満足度が大幅に向上することが分かっている。

(ICR), has started from recognizing single alphanumeric char-etc., which frequently contain named entities, address, acronyms, digits, or currency that are not found in a dictionary.
To address this problem, we developed a dictionary-free
ICR engine that opens broader ICR applications, including
documents containing named entities, address, acronyms,
digits, or currency, with better performance compared toleading competitors. The effectiveness of the proposed engine
has been proved in multiple applications. When integrated
into RPA (Robotic Process Automation) system, the work productivity and employee satisfaction are increased significantly.

＊ Konica Minolta Laboratory U.S.A., Inc＊＊ Konica Minolta, Inc.

26

KONICA MINOLTA TECHNOLOGY REPORT VOL. 16 (2019)1 Introduction

2 Previous Work

Despite the widespread creation of electronic doc-

Handwriting recognition is the task of transforming

uments and the promotion of paperless in office envi-

a language represented in its spatial form into its

ronment, paper documents are still heavily used insymbolic representation. The brief introduction on

many fields, such as medical interview forms, gov-

the recognition system and approaches will be given

ernment office application forms, banking application

as follows.

forms, etc. The operations, like data entry and verification on these paper-based documents are thetedious daily tasks performed by human workers.

2. 1 Online Recognition vs. Offline Recognition
Based on input device, handwriting recognition

The resource turnover on data processing is also a

can be classified into two types, offline character rec-severe issue. Thus, automating these operations has

ognition and online character recognition. Online

great economic and social needs. It will not only

recognition refers to the methods and techniques that

result in time reduction, cost reduction and error

automatically convert a writing trajectory into sym-

reduction, but also solve the problem of resource

bolic representation when it is written using a digi-

shortage and relieve people from boring tasks to cre-tizer or a PDA device, where the pen positions withtemporal order information are recorded. Fig. 1 gives

ative work.
Automating office data processing is getting more

an example of touch based and tracking based online

and more attentions these years. There are satisfac-

systems. Since the rich information and distinguish-

tory solutions on market that convert cleanly printed

able features can be obtained and used in recogni-

documents into digital form through Optical Charactertion, the online recognition has achieved satisfactory

Recognition (OCR). Yet, the solution to convert hand-accuracy and is widely used in various PDA devices.

written documents (ICR) still face many challenges.
Needless to say, the technology of handwriting recognition is the top challenge among them. Although
there are some solutions on market that support written documents, they stay at the level of handprint
character recognition of words that can be found in adictionary.
Putting user needs on top of head, we target to therecognition of a word or a phrase that is an arbitrary
sequence of characters that include digits and special characters. We focused on the development ofrecognition engine with deep learning technology,

(a)

(b)

Fig. 1 (a) Touch based input device.
(b) Tracking based input device.

and meantime developed entire processing workflow,

In contract to online recognition, offline handwrit-from pre-processing, layout analysis to post-process-ing recognition refers to the process of recognizing

ing, in order to provide higher readability. In experi-

writings that have been scanned or captured withments, we have achieved the state-of-the-art recogni-

camera from a piece of paper or a surface (ex. white-tion accuracy of 96 % and 83 % for character andboard), after the writing process. Thus, the temporal

word level respectively. In this report, we focus on aand other helpful information, such as the order ofdeep CNN based method for handwritten recognition

pen-on and pen-off, the number of strokes, move-engine, targeting the challenging problem of uncon-

ments, the direction and speed of writing, the pres-strained transcription of handwritten word images.

sure applied, etc. do not exist. The only information

The rest of the paper is organized as follows: Inavailable in recognition is a static image.

Section 2, we briefly introduce past research andSince the information obtained in online and offline

development on handwriting recognition technology.

handwriting processes are very different, the recogni-

In Section 3, we describe the proposed solution totion approaches are very different as well. Generally,recognizing words in which characters have no cor-

the time-sequence based analysis and image feature

relation with each other. In Section 4, we present

based analysis are the major differences in these two

preliminary evaluation results. Conclusions are given

cases. Fig. 2 gives an example of different featuresin Section 5.

used in two different analysis.

KONICA MINOLTA TECHNOLOGY REPORT VOL. 16 (2019)27

Lexicon-based approaches have been shown effecy

tive on scene text recognition (OCR on natural images)(a)[4, 5]. However, the requirement of a predefined dicx
0

10

20

time 30

40

tionary poses serious limitation for many real-word
applications. For example, transcribing people’s names,
phone number, date, address, email, etc. requires
recognizing out-of-vocabulary (OOV) words.

(b)

Attribute based methods achieve state-of-the-art
Fig. 2 ( a) Time sequence features.
(b) Topologic and statistic features.

recognition accuracy across several standard handwritten word recognition benchmarks, which arepreviously dominated by sequence based methods.

Technically, offline handwriting recognition is much

Compared to RNN, CNN has less training and infer-

more challenging than online recognition. It is not only

ence time because the computation can be parallel-because less information can be used in offline recog-

ized. Meanwhile, attribute based learning uses train-nition than in online recognition, but also because

ing data more efficiently than lexicon-based approachinconsistent or unstable capture conditions make thebecause many attributes are shared thus not every

recognition much more difficult. So far, satisfactory

word needs to appear in the training set. For exam-

products are limited to a few domain applications

ple, in a training set, a word “abstraction” mayunder specific conditions, such as recognition of

appear only 3 times, but the attribute “does TION

postal address and amount on bank checks on speci-

appears in the second half of the word?” may appear

fied locations or within boxes.

many more times.

2. 2 Related Work on Offline Recognition

over 90% word accuracy in IAM and RIMES bench-

Even though CNN predicted attributes can achieve
As mentioned above, offline handwriting recogni-

marks [6], a predefined lexicon is still required fortion is one of the most challenging topics in patterntranscription because by design, these attributes can-recognition and image processing fields. Although

not be directly converted back into word strings. In

the recognition of isolated characters has become

fact, all current designs of lexical attribute vectors

mature (Convolutional Neural network based tech-and their label embedding processes require pre-nology can achieve about 99 % of recognition rate fordefined lexicons ([7-9]).

isolated characters for both western and Asian languages [1-3].), the word or character sequence recognition has yet reached to this level of accuracy.

3 Unconstrained Word Recognition

Moreover, most of current work on offline handwrit-

Unconstrained word refers to a text or digit string

ten recognition assume all the possible words to bethat does not exist in a dictionary, like bank accountrecognized are known beforehand.

numbers, claim numbers, etc. We propose here an
invertible label embedding (encoding) algorithm to

Most of the current word recognition methods canembed character strings into an Euclidean vector
space as attribute vectors, and uses a CNN to learn

be classified into three types:
1. Lexicon-based approaches where word recogni-

and predict attribute vectors of handwriting images intion is modeled as a classification problem andthis Euclidean vector space, and then directly decodes

solved using Convolutional Neural Networks

a predicted attribute vector into a character string using(CNN). Here all words that need to be recognizeda decoding algorithm without requiring a lexicon.

are among a predefined dictionary or lexicon;
2. Sequence based approach where a word istreated as a character sequence which is recognized using Recurrent Neural Networks (RNN);
3. Attribute based approach where lexical attributes of a word are learnt by Convolutional Neural
Networks (CNN). Learned attributes are thenmapped to a word string using RNN or optimizing a specific objective function.

28

Training
image ofhandwritten
word

Image
embeddingusing CNN
being trained

Euclidean vector
space V

Attribute vector v1of embedded image



Label
embedding(encoding)Training
label

Attribute vector v0of embedded label

Fig. 3 Image embedding and label embedding during training.

KONICA MINOLTA TECHNOLOGY REPORT VOL. 16 (2019)

Test image ofhandwritten
word to berecognized

CNN would otherwise require input images to have

Euclidean vector
space V
Image
embeddingusing
trained CNN

Recognizedword

Directdecoding(inverse of encoding)Predicted attribute vector v2of the input image

the same size, while the length of the word image
may vary greatly as compared to its height.
To recognize a target handwriting image (Fig. 4), thetarget image is inputted into the trained neural network to predict an attribute vector v2 in the Euclidean
vector space V (“image embedding”). A decoding pro-

Fig. 4 Direct decoding of lexical attribute vectors during inference.

cess is then applied to the predicted attribute vector

As shown in Fig. 3, a convolutional neural networkof the encoding (label embedding) algorithm used in(CNN) is trained to perform image embedding to

the training process. The result of the decoding pro-embed images of handwritten words into an Euclidean

cess is the recognition result, i.e. the character string

vector space V. During training, in each iteration, athat is recognized. This image recognition process istraining image is inputted into the neural network tosummarized in Fig. 6.

v2, using a decoding algorithm which is the inverse

calculate a first attribute vector v1 in the Euclidean
vector space V (“image embedding”), and the training

Input image

label (the word) is embedded into the same Euclidean
vector space V using the invertible encoding algo-

Trained CNN Modelrithm (described in detail later) as a second attribute
vector v0 (“label embedding (encoding)”). The weights

Predicted attributesof the neural network are updated to minimize a loss
function which measures the distance between the

Inverse of label embedding

attribute vectors v1 and v0 in the Euclidean vector
space. Thus, during training, the label embedding
step is used to construct the ground truth of training

Transcribed text

samples: the ground truth word label is encoded into

Fig. 6 Recognition process (inference) using a trained model.

its corresponding attribute vector as ground truth.
The trained neural network is able to predict an attri-

The invertible encoding (label embedding) andbute vector from an input word image. This neural

decoding algorithms will be referred to as “invertible

network training process is summarized in Fig. 5.

Pyramidal Histogram of Characters” (iPHOC) fromnow on. It is assumed that all characters of the string

Input image

Ground truth label

CNN Model

Label embedding

attribute vector uses a recursive bisection and histogram computation process. In each bisecting step, if
acters, its middle character is omitted in the next

Ground truth attributes

level child strings. Thus, the child strings at eachlevel always have the same number of characters. If
the string being bisected has an even number of char-

Loss Function

Iterative update

Σ. The encoding of a character string into an iPHOC

the string being bisected has an odd number of char-

Weight Decay
Predicted attributes

being encoded belong to a known and fixed alphabet

acters, it is deemed to have an omitted middle character that is an empty character.

Gradient Descent

For example, in Fig. 7 (a), “success” (level 0) is

Fig. 5 Training process of a CNN model.

bisected into two child strings “suc” and “ess” (omitting the middle character “c”) (level 1), which are

In this development, a L_2 loss function and sto-

further bisected into smaller child strings “s” and “c”,

chastic gradient descent are used to train a 11-layerand “e” and “s” (again omitting the respective middle

VGG-based CNN [10], which includes a horizontal

characters) (level 2). The next level bisections (level

Spatial Pyramid Pooling layer [11] before the fully-3) are all empty, as indicated by the quotation marks.

connected layers to enable arbitrary input image size

Fig. 7 (b) shows the bisection of a string that is not ain the horizontal direction. This is helpful because acommon word.

KONICA MINOLTA TECHNOLOGY REPORT VOL. 16 (2019)29

success

Lv 0

3
2

“success”

1

ess

suc
s

c

Lv 1

3

“”

“”

(a)

“”

3

s
“”

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

u

v

w

x

y

z

a

b

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

u

v

w

x

y

z

Lv 2

“ ” Lv 3

“”

1
a

b

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

u

v

w

x

y

z

2

“s”

1
a

b

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

u

v

w

x

y

z

a

b

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

u

v

w

x

y

z

2

F3NP20X!

Lv 0

“c”

1

3
2

Lv 1
3

a

b

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

u

v

w

x

y

z

2

“s”

1

F3

NP

20

X!

Lv 2

Lv 2
“e”

1

20X!

F3NP

Lv 1
“ess”

2

3

Lv 0

“suc”

1

e

“”

b

2

3

“”

a

a

b

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

u

v

w

x

y

z

Fig. 8 Character histogram of each node in the binary bisection tree.

F

3

N

P

2

0

X

!

Lv 3

(b)

For an iPHOC encoding of k levels, there will be(2 -1) histograms; and with an alphabet of size n, thekFig. 7 B isection process of a word.
(a) A common word of length 7.
(b) An out-of-vocabulary word of length 8.

attribute vector’s dimension will be (2k-1)*n. To decodeThis bisecting process can be represented as a

individual histograms each of size n, using the samebinary tree, where the root of the tree is the originalorder in which the histograms are concatenated instring and the other nodes are the child strings. This

the encoding algorithm.

a CNN-predicted iPHOC attribute vector of dimension (2k-1)*n, the vector is divided to obtain (2k-1)

binary tree can also be seen as a coarse-to-fine pyra-

For each node of the decoding binary tree, the his-

mid where each level focuses on smaller and smaller

tograms of its left and right child nodes are subtracted

child strings.

from the histogram of the current node to obtain a

For each node of the binary tree, a histogram ofdifference histogram. The difference histogram ischaracters is calculated from the character string of

decoded to obtain a decoded character. If the maxi-

that node, which is a histogram with n values (n being

mum histogram value is greater than the threshold of

the size of the alphabet Σ), each value being theconfidence τ, the decoded character is the character

number of times the corresponding character occurs

having the maximum histogram value; if the maxi-

in the string. Fig. 8 shows the histograms of levels 0

mum histogram value is less than or equal to theto 2 for the example of Fig. 7 (a) (in this example, allthreshold of confidence τ, the decoded character ischild strings at level 3 are empty, so all level 3 histo-

an empty character.

grams have zero values and are not shown in Fig. 8.

As a result, a decoded character (which may be an

Note that the omission of the middle character

empty character) is generated for each node of thewhen bisecting odd-length strings does not cause any

decoding binary tree. Fig. 9 illustrates the decoded

loss of information. During decoding, the omitted

characters organized in a “decoding binary tree”,

middle characters can always be recovered by find-corresponding to the example of Fig. 7 (a).

ing the difference between a histogram of a node andthe sum of the two histograms of its left and rightcchild node (and if there is no difference, then theomitted middle character is empty). For example, the

Lv 0

s

u

central “c” in “success” can be found by subtracting
the sum of two level 1 histograms (for “suc” and “ess”)sfrom the level 0 histogram (for “success”) (see Fig. 8).

c

Lv 1

e

s

Lv 2

After the bisection is completed, all histograms for
all nodes of the tree (including the zero histograms)
are concatenated to form the attribute vector.

30

“”

“”

“”

“”

“”

“”

Fig. 9 A decoding binary tree.

KONICA MINOLTA TECHNOLOGY REPORT VOL. 16 (2019)“”“” Lv 3

The decoded characters for all the nodes of thedecoding binary tree are concatenated together, based

5 Conclusions and Future Workon an order that is the reverse of the recursive bisec-

In this report, we proposed a handwriting recogni-tion used in the encoding algorithm, to obtain a char-tion method that enables unconstrained transcription

acter string that is the final decoding result.

of handwritten word images. The method resolves

As can be seen, the invertible label encoding andthe technical difficulty of transcribing a textual image

decoding method is a one-to-one mapping between

that may contain arbitrary text. The method provides

character strings and a set of grid points of the Euclidean

a deterministic procedure to encode and decode text

vector space. This method is simple and fast, as it

embedding without using optimization or machine

does not require an optimization (nearest neighborlearning based methods as reported in the literature.

search) process for decoding. To the contrary, theThis is possible because the label embedding uses an

SPOC [8] and PHOC [7] do not provide an invertible

invertible coding scheme which is a one-to-one map-

decoding method. In these methods, each words ofping between valid grid points of the Euclidean vec-the lexicon is mapped to a grid point, but not uniquely.

tor space and character strings. In the future, wewould like to extend this method to recognizing awhole line of handwritten text. This would be desir-

4 Experiments

able in the situation where word segmentation is

To test the performance of the proposed iPHOC attri-

hard.

butes with CNN, we report the word recognition accuracy on 7 test samples from writers whose handwritings
have never been seen (Fig. 10). Each sample has 56
handwritten words that contain digital values and special characters (“$”, “%”, “,”, “.”, “-”, “(”, “)”). Somesamples may have less words because miswritten words
are not counted. The samples are scanned at 300 ppi.

Table 1 shows the word recognition accuracy on 7
samples. Comparing with a leading competitor, ourmodel shows significant better accuracy, a 22 % higher
on average. Meanwhile, the accuracy of our model is
on par with start-of-the-art handwritten digital sequence
recognition, which uses a similar test dataset [12].

Fig. 10 Example of test samples.
Table 1 Evaluation results in comparison with a leading competitor.
Sample (#)

Leading Competitor

Ours›

1

48%

73%

+25%

2

77%

93%

+16%

Difference

3

59%

86%

+27%

4

56%

75%

+19%

5

77%

89%

+12%

6

63%

89%

+26%

7

50%

79 %

+29%

AVG

61%

83%

+22%

References[1]  Wang Yutao, Qin Tingting, Tian Ruixia, Yang Gang,“Recognition of license plate character based on wavelet
transform and generalized regression neural network”,
Control and Decision Conference (CCDC), 2012 24th
Chinese, 1881-1885[2]  Hong-Ming Yang, Xu-Yao Zhang, Fei Yin, Zhenbo Luo,
Cheng-Lin Liu: Unsupervised Adaptation of Neural Networks
for Chinese Handwriting Recognition. ICFHR 2016: 512-517[3]  Alex Graves, J. Schmidhuber, “Offline Handwriting Recognition
with Multidimensional Recurrent Neural Networks”, NIPS
2009[4]  Jaderberg, Max, et al. “Synthetic data and artificial neuralnetworks for natural scene text recognition.” In NIPS Deep
Learning workshop (2014).
[5]  Jaderberg, Max, et al. “Reading text in the wild with convolutional neural networks.” International Journal of Computer
Vision 116.1 (2016): 1-20.
[6]  Poznanski, Arik, and Lior Wolf. “Cnn-n-gram for handwriting word recognition.” Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition. 2016.
[7]  Almazán, Jon, et al. “Word spotting and recognition withembedded attributes.” IEEE transactions on pattern analy︲
sis and machine intelligence 36.12 (2014): 2552-2566.
[8]  Rodriguez-Serrano, Jose A., Florent Perronnin, and France
Meylan. “Label embedding for text recognition.” Proceedingsof the British Machine Vision Conference. 2013.
[9]  Wilkinson T, Brun A. Semantic and verbatim word spotting
using deep neural networks. In Frontiers in Handwriting
Recognition (ICFHR), 2016 15th International Conference
on 2016 Oct 23 (pp. 307-312). IEEE.
[10] K. Simonyan et al., Very Deep Convolutional Networks For
Large-Scale Image Recognition, ICLR 2015.
[11] He, Kaiming, et al. “Spatial pyramid pooling in deep convolutional networks for visual recognition.” European confer︲ence on computer vision. Springer, Cham, 2014.
[12] Wang, Qingqing, and Yue Lu. “A sequence labeling convolutional network and its application to handwritten string
recognition.” Proceedings of the 26th International Joint
Conference on Artificial Intelligence. AAAI Press, 2017.

KONICA MINOLTA TECHNOLOGY REPORT VOL. 16 (2019)31