Noname manuscript No.
(will be inserted by the editor)

Bi-directional Extraction and Recognition of Scene Text
with Layout Consistency
Ryota Hinami · Xinhao Liu · Naoki Chiba · Shin’ichi Satoh

Received: date / Accepted: date

Abstract Text recognition in natural scene images isa challenging task that has recently been garnering increased research attention. In this paper, we propose amethod of recognizing text by utilizing the layout consistency of a text string. We estimate the layout (fourlines of a text string) using initial character extraction
and recognition result. On the basis of the layout consistency across a word, we perform character extraction
and recognition again using four lines, which is moreaccurate than the first process. Our layout estimation
method is diﬀerent from previous methods in terms of
exploiting character recognition results and its use of aclass-conditional layout model. More accurate and robust estimation is achieved and it can be used to refinecharacter extraction and recognition. We call this twoway process—from extraction and recognition to layout, and from layout to extraction and recognition—”bi-directional” to discriminate it from previous feedback refinement approaches. Experimental results demonstrate that our bi-directional processes provide a boost
to the performance of word recognition.
Keywords Scene text recognition – Character
recognition – Word recognition – Layout consistency
R. Hinami
The University of Tokyo
E-mail: hinami@nii.ac.jp
X. LiuTokyo Institute of Technology
E-mail: liuxinhao@ok.ctrl.titech.ac.jp
N. Chiba
Rakuten, Inc
E-mail: naoki.a.chiba@mail.rakuten.com
S. Satoh
National Institute of Informatics
E-mail: satoh@nii.ac.jp

1 Introduction
As the number of vision systems (e.g., mobile cameras) increases, so too does the need for information
retrieval from images. Text in natural scene images contains valuable information pertaining to street signs,
store names, product brands, and so on. Recognizing
the text in these images is becoming more important
than ever before.
Although conventional optical character recognition(OCR) has been successfully commercialized, its application is limited to the realm of business documents andfactory automation. Recognizing text in natural scenes
is still a challenging problem and has been garneringa significant amount of attention from academia. Such
images frequently contain blurred text on complicatedbackgrounds, and the uncontrolled lighting conditions
can cause reflection, highlights, and/or shadows on thetext, which makes scene text recognition more diﬃcult
than OCR.
Text recognition research can be divided into two
categories: text detection and word recognition. The
method we propose in this work falls into the categoryof word recognition, specifically, the recognition of aword from a given cropped rectangular region in animage. Word recognition is regarded as the key part ofend-to-end scene text recognition.
Character extraction is a critical stage in the word
recognition of scene text. Character extraction is difficult, especially for scene text because scene text is
not easily segmented or binarized due to various factors such as noise, blur, and complicated background.
Two types of methods are mainly used in character extraction: connected component-based (CC-based) andclassifier-based. CC-based methods (e.g., MSER) extract characters by connected component analysis where

2

Ryota Hinami et al.

text pixels with similar properties are grouped to conthat we can use to estimate layout is limited. We therestruct component candidates. Classifier-based (e.g., slidfore propose a novel layout estimation method that can
ing window) methods find the candidate character by
estimate four lines from a limited number of characters.
applying a character classifier to sub-windows in mulOur layout estimation method is diﬀerent from previtiple scales through all locations of an image. Choosing
ous methods in that it exploits character recognition
which of these two methods to select is a system deresults and uses a class-conditional layout model thatsign issue. There is a trade-oﬀ between accuracy andis trained in advance. On the basis of four lines estiprocessing time. Although sliding window methods are
mated by the proposed method, we refine the result ofgenerally more accurate than CC-based methods, they
character extraction and recognition.
are more computationally expensive.
There are two main contributions in this paper. The
Character recognition is also a fundamental probfirst is accurate and robust layout (four line) estimation
lem of text recognition. Scene text’s variation of font,methods. The proposed method can estimate layout
noise, and distortion makes this problem a very chalfrom only a few candidates by using recognition results
lenging task. Various features and classifiers have been
and trained statistical layout model. In addition, ourinvestigated as character classifiers (e.g., HOG feature
method can easily be adapted to other types of layout+ SVM classifier, convolutional neural networks). Geosuch as curve; i.e., it is not limited to linear. The second
is a layout consistency-based feedback refinement prometric information (position of characters) is also used
to improve the accuracy of character recognition. For
cess. Using the four lines estimated by our method, we
example, in scene text recognition, Bissacco et al. [6]perform the following two feedback refinements: charadded the geometric feature (top and bottom positions
acter re-extraction by sliding window with four lines
relative to the height of text line) in addition to HOG
and re-scoring of character classification using relative
features, while Neumann and Matas [28] corrected theposition to the four lines. These two contributions are
initial recognition results by comparing the height withregarded as top-down (character extraction and recogother characters to diﬀerentiate the upper and lower
nition → word layout) and bottom-up (word layout →cases. Since vertical position and height are importantcharacter extraction and recognition) processes, respecfeatures that characterize alphabets (e.g., the upper
tively. We combine these two processes into what we calland lower cases of some characters such as ’o’ are disa bi-directional process.
criminated only by these features), incorporating precise geometry features is important to improve the accuracy of character recognition. However, in scene text,
2 Related Works
since it is possible to know the relative vertical position for only a few characters, it is diﬃcult to estimate
In this section, we review works that are related to our
precise geometry information for character recognition.
method. We first review scene text recognition, espeIn this work, we address the two problems mencially word recognition. We then focus on geometric estioned above—loss of character geometric information
timation related to our line estimation method. Finally,
and trade-oﬀ between two character extraction approaches—we review the style consistency (our layout consistency
by focusing on layout consistency. Following the typogcan be regarded as a kind of style consistency) andraphy, we exploit four lines of a text string (Fig. 2)
feedback refinement related to our re-extraction and reas layout consistency. Since fonts are designed on thescoring processes.
basis of these four lines (layout consistency), they canbe regarded as one of the important style that define
the vertical position of characters. We can solve the two
2.1 Scene Text Recognition
problems by performing character extraction and recognition using four lines: 1) we can perform eﬃcient and
The problem of scene text recognition has been adaccurate character extraction even with sliding window
dressed as two tasks: text detection and word recognibecause character size and position are constrained bytion. Text detection methods tackle the problem of lofour lines, and 2) geometry information can be incorcalizing words in whole natural scene images and word
porated into character recognition by referring to therecognition methods recognize the cropped word imrelative position between the four lines and characters.
ages obtained in the text detection stage. End-to-end
Accurate estimation of four lines is indispensable insystems that integrate both methods are addressed interms of making use of them for character extraction[39, 41, 3, 14, 28, 29, 6]. Since our method focuses on theand recognition. However, since only a few charactersword recognition problem, we review the work related
can be extracted reliably from one word, information
to word recognition here.

Bi-directional Extraction and Recognition of Scene Text with Layout Consistency

3

Typical word recognition systems starts with candiet al. [6] use the HOG feature as an input of the deep
date character extraction. Character extraction methneural network to find character segmentation points
ods can be categorized into two types: connected component-as well as the character classifier.
based (CC-based) and classifier-based. For connected
Some recent work [2, 12, 14, 11] performs word recogcomponent-based methods, MSER [24] is the most popnition by a holistic approach, in contrast to other characterular method in scene text recognition [31, 29, 28, 30] bebased recognition approaches. Goel et al. [11] use wholecause it achieves high recall compared with the binaimage features to recognize words by comparing themrization that is often used in document processing. Forwith the synthetic images of lexicon words. Almazan etclassifier-based methods, a sliding window consistingal. [2] embed word images and text strings in a comof a character classifier is often used [40, 39, 41, 27, 34].
mon vectorial subspace and cast a word recognition task
Since multi-scale sliding window produces many overas a nearest neighbor problem. Gordo et al. [12] exlapped candidates, non-maximum suppression is applied
tend this work by using supervised mid-level features
to them to reduce the number of candidate regions.
for word image representation. Jaderberg et al. [15] inSome recent works such as those by Bissacco et al. [6]put a whole word image into a deep convolutional neuand Alsharif and Pineau [3] use over-segmentation meth- ral network trained with a huge amount of synthetic
ods that find character segmentation points with a charimages. In an extension of this work [14], they integrate this word recognition framework into an end-toacter classifier, while Weinman et al. [44] fully integrate segmentation and recognition into a probabilistic
end text recognition pipeline. Since these methods canframework. In these methods, since the number of candeal with style consistency such as character alignment
didates is not so large compared to multi-scale sliding
in a holistic way, they are extremely eﬀective in tasks
window, they do not use NMS.
where a candidate lexicon is available as prior knowlCharacter candidates extraction is usually performed edge. However, it is diﬃcult for them to recognize a newwith high recall and appropriate character regions areword out of lexicon.
selected in the next inference stage. Previous works
have addressed character extraction using pictorial struc2.2 Text Line Estimationture [39], conditional random field (CRF) [27, 34], weighted
finite-state transducers (WFST) [31], and hidden Markov
Estimation of text line has been used to improve text
models(HMM) [3]. Although the important components
recognition accuracy. In handwriting recognition, Benof CRF are energy functional and likelihood training,gio and LeCun [5] estimate four tied curves for each
Mishra et al. [27] and Shi et al. [34] use only energy
word to normalize a word image. The model of the fourfunctional. Most models solve the word inference probcurves is represented by six parameters and the mostlem by a cost minimization problem with a Viterbi-likelikely parameters are inferred by a probabilistic framealgorithm. A language model is often incorporated into
work with an EM algorithm. Caesar et al. [7] estimate
this inference to check the consistency of language. Lanfour tied lines based on the contour processing of a biguage models have been used since the appearance ofnary image with iterative regression analysis.
OCR literature such as that by Jones et al. [16] usIn scene text recognition, Neumann and Matas ining n-grams as an OCR post-processor and by Bazzi et[28] introduced a method to detect top and bottom
al. [4] integrating n-grams in an OCR system. In scene
lines for geometric image normalization while using atext recognition, Thillou et al. [37] used n-grams for
typographic model by measuring the height of upperpost-processing and Weinman et al. [42, 45] integratedcase and lower-case letters for correctly recognizing inbi-grams for word recognition while Bissacco et al. [6]terchangeable letters. Neumann and Matas in [29] use
integrated n-grams. The consistency of not only lanfour lines of a text string to verify text localization.
guage but also style is incorporated into this process,
They use the vertical extrema of character regions toas we discuss in the next subsection.
fit four lines. They first estimate the common slope (diDeep neural networks have recently been used inrection) of four lines by fitting bottom points and thenscene text recognition to improve the recognition accufind four lines that minimize the square error. Weinracy. Wang et al. [41] use convolutional neural networks
mann et al. in [44] identified a pair of guidelines for a(CNN) with raw image input as a character classifier.
text string to normalize the region before recognizing
Alsharif and Pineau [3] use a maxout network and furwords. They can infer the extrema in which the guidether improve the performance of CNN’s character claslines intersect even if characters are inclined throughsification. Specifically, they use the maxout network for
an iterative refinement approach.
segmentation and character classification together by
These methods use extrema of character regions tointegrating them into hybrid HMM models. Bissacco
estimate lines without recognizing characters. They con-4

sequently require many candidate regions to estimate
the four lines accurately as the exact typograhic stylethat we need. Since sometimes only a few characters canbe extracted reliably from scene text, these methods are
not suited for our purpose. Our method, in contrast, estimates lines using character recognition results. In addition, we use a character-dependent layout model that
we train in advance. Our method can therefore extract
a lot of information to estimate the word layout (fourlines) and achieves accurate and robust layout estimation from only a few candidate regions.

Ryota Hinami et al.

Input image(a)
4. Sliding window
re-extraction

1. CC-based extractionxSt

2. Character recognition

x
p

(b)

o

5. Recognition withre-scoring

Regionlevel

Character
level

Word
level
3. Four-line estimation

2.3 Style Consistency and Feedback Refinement
Style consistency is important information and has been
widely used in text recognition [33, 36, 4]. In terms ofscene text, Weinman et al. in [42, 45] incorporate character similarity as style consistency under the assumption that the appearance of characters with the sameclass is similar if the source (font) is the same. They
incorporate this information into a probabilistic framework so that characters with similar appearance are
given the same label and character with dissimilar appearance are given diﬀerent labels. Weinman et al. [43]incorporate the bi-gram model of character width into
the inference to find the best parse, wherein correlation
between the widths of a character pair is statisticallymodeled with reference to character class. Novikova et
al. [31] use color and vertical position as style consistency. Specifically, they use color information and the
distance from the top and bottom points of characterto the baseline as a part of the word inference process.
We conclude this section with a brief review of works
related to our feedback refinement, i.e., the re-extraction
and re-scoring processes. Huang et al.[13] split the connected characters detected by MSER using sliding window with a CNN classifier after estimating the text linefrom initial candidates. Although this process is related
to our re-extraction using sliding window, it cannot recover missing characters, unlike our method. Mishra etal. [27] use character classifier re-scoring based on aspect ratio, which is related to our method using verticalgeometry positions to re-score character classification
although Mishra et al. does not use word-level knowledge to re-score. Neumann and Matas [28] correct theinitial recognition results by feedback loop to diﬀerentiate the upper and lower cases of certain letters bycomparing the height with other recognized characters.

6. Word inference

Spot
Fig. 1: The pipeline of the proposed word recognition method.

3 Overview of the Approach
Our method begins by extracting of reliable charactersand using them to estimate four lines using them (Section 4). Character re-extraction and re-scoring are thenperformed using the four lines and the optimal word
is inferred from obtained candidates with a cost minimization framework (Section 5). Figure 1 shows thepipeline of the proposed method.
We first extract reliable characters whose scores are
suﬃciently high. To extract a reliable character, we use
CC-based methods (MSER and Otsu binarization) on
an input word image and apply a character classifier
to extracted components. We then estimate the layout
consistency (four lines of a text string) on the basisof the reliable characters. Unlike previous estimation
methods, which use character extraction results to obtain constraints [28], we use character recognition results to obtain layout consistency. Each character recognition result predicts the distribution of the line positions by using a character class dependent model thatis trained in advance, which allows accurate estimationeven if only a few candidates can be extracted.
We then perform character extraction and recognition again under the layout consistency defined byfour lines. We use two top-down cues, indicated by the
red arrows in Fig. 1. (a) Re-extraction. We extract
character candidate regions by using layout consistency
to concentrate on promising regions. Although we use
sliding windows, we can perform a computationally efficient scan with four lines, unlike conventional multi-Bi-directional Extraction and Recognition of Scene Text with Layout Consistency

Table 1: Character types
Ascenderbdfhiklt
Descender
gjpqy
Small lower case acemnorsuvwxz
Upper case
ABCDEFGHIJKLMNOPQRSTUVWXYZ
Digit
0123456789

scale sliding window. (b) Re-scoring. Among multiple
character classes for each region, we re-calculate scoreson the basis of layout consistency, giving higher scoresto character classes consistent with the style and lower
scores to classes with less consistency.
Candidate regions obtained by first and second scans
are processed jointly in the word inference process. We
find the optimal labeling (character or non-character,and its character class as well) that minimize the costfunction. We use a cost function similar to the cost function in Mishra et al. [27], which is defined by classification score, spatial constraints, and linguistic model.
The labeling that minimizes the cost function is inferredby the TRW-S algorithm [19].

3.1 Character Classifier
Here, we describe the character classifier used in ourpaper. To recognize characters, we apply a characterclassifier to every candidate region. It generates classification probabilities (scores) for multiple characterclasses. We use HOG features [9] along with the aspectratio of the bounding box of the character. Since alphabets tend to have certain aspect ratios for specific characters, aspect ratio is important information to identify
character class. We confirmed a performance increase
due to using aspect ratio (accuracy increase from 80.3%
to 81.8% on the ICDAR 2003 dataset). An RBF kernel
SVM is used for the classifier with the one-versus-all
multiclass SVM setting. We give special treatment tocase-identical characters (c-o-s-v-w-x-z) that are identical in upper and lower cases. Instead of 62 classes
corresponding to upper and lower case alphabets anddigits, we use 55 classes by merging the upper and lower
cases of case-identical characters into one class. These
classes are then separated into upper and lower casesin the re-scoring process described in Section 5.2.

4 Estimation of Four Lines
The proposed method starts with the estimation of fourlines—ascender line, mean line, baseline, and descenderline (Fig. 2)—using the initial recognition result. We

Ascender line
Mean line
Baseline
Descender line

5

Style

Fig. 2: Four lines of a text string.

first extract reliable characters that are used for line
estimation. To estimate the lines from reliable characters, we train the Gaussian distribution model of theline positions depending on each character class in advance. We then estimate the equation of the four lines
via maximum likelihood estimation by using the positions of reliable characters and trained models.

4.1 Extraction of Reliable CharacterTo obtain layout consistency (four lines of a text string),
we use reliably extracted characters, i.e., reliable characters, whose scores are above the predetermined threshold. The idea of using reliable outputs to guide recognition has appeared in various literature on handwriting
and speech recognition (e.g., Miller and Viola [25]). We
use a connected component-based (CC-based) method
to extract initial candidate regions. We extract candidate regions by two methods: MSER and Otsu binarization. Following the approach of Neumann andMatas [28], the MSER region detector [24] is used to
extract the set of candidates. We also use Otsu binarization [32], where the connected components of binary images are used as another set of candidates. We
combine the results of MSER and Otsu binarization toform the final result of CC-based extraction. We confirmed that the combination of these two methods delivers higher recall rates in the character extraction (seeSection 6.3).
We then select reliable characters from candidate
character regions by using a character classifier. We apply the character classifier to every candidate, which
outputs a character class along with its score. The regions with scores above a certain threshold are selected
as reliable characters. These selected regions are thenused to estimate four lines of a text string in the next
step. Since two or more reliable characters are required
for the line estimation, if less than two characters withscores above the threshold are extracted, the two characters with the two highest scores are selected as reliable characters.

6

Ryota Hinami et al.

Line

Reliable characters

abc
Gaussian models

Step 1. Estimate baseline

Line

Line

Estimated line position

L3: y=kx+b3

Step 2. Estimate other lines

Line
x1
x2
x3
x4
c1=‘L’ c2=‘i’ c3=‘n’ c4=‘e’

L1: y=kx+b1
L2: y=kx+b2
L3: y=kx+b3
L4: y=kx+b4

Fig. 3: Estimation of four lines. For each reliable character, distributions of line position are obtained from trained models.
Equations of lines are then estimated by maximum likelihood estimation.

4.2 Training of Line Position Models
In this section, we describe how to train the Gaussian
distribution model. For training we used the word image datasets with ground truth of character boundingboxes and their classes. We used the training part of
the IIIT 5K-Word [26] and ICDAR2013 [18] datasets.
In addition to these, we used the synthetic word images as training data because some character classes
such as ’q’ scarcely appear in public training sets. We
used 30 words generated from diﬀerent fonts and eachsynthetic word contains 62 characters (0..9, a..z, and
A..Z). In total, 2,577 words containing 30∼577 characters for each character class were used for training. The
detailed training procedure is shown in Algorithm 1. In
summary,1) Determine the ground truth lines of training images
from given training data (lines 5–9).
2) Calculate the line positions relative to each character (lines 10–16).
3) Calculate Gaussian parameters µlc , σlc from obtained
positions (lines 19–24).
We first determine four lines for each of the training
word images. The lines that a character touches dependon the class of the character, as shown in Table 1—forexample, upper case characters touch the ascender line
and the baseline. On the basis of this fact, lines are fit to
the centers of the upper and lower bases of the boundingboxes of characters using the least-mean-square (LMS)
algorithm. This requires more than two points available
for each line. If not available, corresponding lines will
not be drawn and the information will not be taken into
account for the corresponding training word image.
We then compute vertical positions of lines relative
to each character region. The intersections between fourlines and the vertical center line of a character are usedas the line positions for the character. These positions
are then normalized on the basis of the height of eachcharacter region.
We construct a statistical model of line positions
using the position values obtained from many word images. We assume normalized line positions yield theGaussian distribution for each line and character class.
Therefore, the means and variances of normalized line
positions are obtained as the suﬃcient statistics of theGaussian distributions for each of the 62 classes × 4lines, namely, µlc and σlc as the mean and variance foreach line l and character class c. l can be 1, 2, or 4
corresponding to ascender line, mean line, and descender line, respectively; also, l = 3 for the baseline forcompleteness. These models are used to estimate theline positions for each extracted region. Note that although Algorithm 1 explicitly use the set of values Plc ,we can calculate the mean and variance eﬃciently by
maintaining not Plc but rather just the sum and sumof squares and updating them for each iteration.

Algorithm 1 Training of line position models
Input: Number of training word images N , Ground truth

data: character class and bounding box.
Output: {µlc }, {σlc },j =62
1: initialize {Pij }ii=4=1,j =1 ⇐ 62 × 4 empty set
2: for i = 1 to N do3:
C ⇐ set of characters in ith word image
4:
for l = 1 to 4 do
5:D ⇐ set of characters that touch line Ll
6:
if size(D ) < 2 then7:
continue8:
end if9:
Ll ⇐ Least-square fitting(D)10:for k = 1 to size(C) do
11:c ⇐ character class of Ck
12:b ⇐ bounding box of Ck ([x y width height])
13:p ⇐ y-position of Ll at x=bx + bwidth /2
14:p̃ = (by − y )/bheight15:
add p̃ to Plc
16:
end for17:
end for
18: end for
19: for c = 1 to 62 do
20:
for l = 1 to 4 do
21:
µlc ⇐ mean(Plc )22:
σlc ⇐ variance(Plc )23:
end for
24: end forBi-directional Extraction and Recognition of Scene Text with Layout Consistency

7

4.3 Estimation of Line Equations
We now estimate equations of lines from reliable character extraction results by using a trained model of line
position. For each character region, we can estimate thepositions of lines, as shown in Fig. 3. Let us denotes theequation of l th lines as Ll : y = kx + bl . The estimation of line equations is performed with the following
two steps:
1) Estimate the equation of baseline (common slope k
and intercept of baseline b3 ).
2) Estimate the intercept of other three lines (b1 , b2 ,and b4 ).

Fig. 4: Examples of line estimation results. The red rectangles
are reliable characters used for the estimation of four lines.

lines are used to re-extract missed character candidate
regions and to re-score the initial scores from the character classifier. Figure 4 shows examples of line estimation results.

We first determine a baseline L3 . Since most characters touch the baseline, its estimation is expected tobe more reliable than others. Neumann and Matas [29]also estimate baseline prior to the other three lines. The
vertical location of the baseline at each reliable charac4.4 Curved Text
ter is obtained from line position models as µ3c , where cis its character class. The baseline is inferred by fitting
Although we typically assume that characters in text
these locations using LMS.
are aligned in a straight fashion, scene images someNext, other lines are estimated. Since the slope oftimes contain curved text. To extend the proposed method
these line is the same as that of the baseline, we only esfor curved text, we construct a four-concentric circle
timate the intercepts for each of three lines (b1 , b2 , andmodel in addition to the existing four-line model withb4 ). We estimate the intercepts by maximizing likeliadaptive model selection.
hood under the constraint of the slope k. Note that
We first judge from reliable characters whether acase-identical characters (c-o-s-v-w-x-z) are excluded from given word is straight or curved. Since the slope of areliable characters to estimate lines other than the basecurved text line is gradually changed from left to right,
line. Given N reliable characters in a word, let c1 , c2 , · · · , cN we check if the text is curved by the following steps.
be corresponding character classes and x1 , x2 , · · · , xN
First, for each reliable character, the vertical location
be horizontal locations at the centers of the characters,of the baseline is estimated from a trained Gaussian
as shown in Fig. 3 right. Further assuming p(yil |ci ) tomodel. Adjacent locations are connected by lines andbe the probability that the intercept of the lth line at xi
their slopes are calculated. We then calculate the subis yil when the character class of the ith character is ci ,traction between two slopes for every adjacent pair of
the intercepts that maximize likelihood can be obtainedlines. If these diﬀerence values are all positive or allasnegative, text is inferred as curved and a circle modelN
is used for layout estimation.
!
bl = arg max
ln p(yil |ci )Next, we explain how we estimate four concentricbli=1
circles. First, a circle corresponding to the baseline (baseN!
circle) is estimated using LMS. The other three circles(µ˜lci − yil )2= arg max−
(1)
2
are then estimated. Since we assume the four circles are2σ˜lcibl
i=1
concentric, the centers of the circles are the same as that"N1(µ˜lci − kxi ))of the basecircle. The radius of each circle is obtained
i=1 σlc
˜ i2=,"Nby maximum likelihood estimation. Examples of curve
1i=1 σlc
˜ i2estimation results are shown in Fig. 5. On the basis ofwhere µ˜lci , σ˜lci are the mean and variance of the posifour estimated circles, re-extraction and re-scoring aretion of the lth line for ci , normalized by the location
performed similar to the straight cases. Our evaluationand size of the ith reliable character.
of this system is described in Section 6.5.
By considering the variance σlci , we vary the priorThe limitation of our method is that it does not
ity of characters for estimation. For example, in Fig. 3,
work well for largely oriented text, since we used only
‘L’ is considered more important to estimate the asthe vertical extrema of a connected component as thecender line than the other characters. These estimatedintersection point; for oriented characters, vertical ex-

8Fig. 5: Examples of four concentric circles estimation for
curved text. The red rectangles are reliable characters used
for estimation.

Fig. 6: Examples of the eﬀect of sliding window re-extraction.
The top row shows the extraction results by CC-based extraction (MSER and Otsu). The middle row shows the four lines
estimated by our method. The bottom row shows the extraction results with sliding window re-extraction along fourlines.

trema is not always the point that intersect with fourlines. Weinman et al. [44] address this problem and infer
the exact extrema that the guidelines intersect by using
an iterative refinement method. However, the method
they used cannot estimate guidelines from few characters (e.g., ”Dry” in Fig. 7 of that work [44]) because it
estimates the top and bottom lines without recognizing character class. We assume that accurate lines canbe estimated even from largely oriented text with few
characters by combining their iterative process with our
method using the character recognition result.

5 Feedback Refinement with Four Lines
5.1 Sliding Window-based Re-extraction
The connected component-based method (Section. 4.1)
sometimes misses characters, such as those shown in the
top row of Fig. 6. The sliding window-based method, in
contrast, can find such characters, but this comes at thecost of a huge number of candidate regions. Moreover,since windows need to be scanned at various scales,
aspect ratios, and locations in a given image, a huge
number of false alarms is produced and computationalcosts may become high.

Ryota Hinami et al.

Although we also use sliding window to re-extract
missed characters, our method significantly reduces thenumber of windows to be checked by imposing layout
consistency. We can reduce the number of windows for
two reasons: (1) our method scans windows in limited
space excluding reliable character regions, as shown in
Fig. 7, and (2) we slide windows in the horizontal direction only because vertical positions are determinedby word layout (four lines).
Here, we describe the details of the sliding window
along four lines. Figure 8 illustrates the sliding window re-extraction. We change the vertical position andheight of the windows in accordance with the following three cases: (1) windows to detect ascenders, upper cases, and digits between ascender line and baseline, (2) windows to detect small lower cases between
mean line and baseline, and (3) windows to detect descenders between mean line and descender line. The
width of the window is determined by selecting an aspect ratio from a set of predetermined ratios. The windows are then scanned along lines. The regions wherereliable characters are detected will be skipped to further reduce the number of scans. In this way, charactersthat are missed in the first extraction are re-extracted,
as shown in the bottom row of Fig. 6. Our two-way
character extraction—a hybrid of CC-based and sliding
window extraction—significantly reduces the computational cost while achieving high recall.
The final set of candidate character regions is obtained by combining MSER, Otsu binarization, andsliding window re-extraction. Previous sliding windowbased methods [41, 27, 34] typically apply non-maximum
suppression (NMS) to reduce the number of candidate
regions before they are fed into a word inference stage.
However, when we tried using NMS after our sliding
window re-extraction, it was not as eﬀective as in previous works because our method already suﬃciently
reduces the number of character candidate regions bylayout consistency, and thus the diﬀerence in the performance (word recognition accuracy and processingtime) does not change much between with and without NMS. We therefore decided not to use NMS following the method without sliding window, the same as
Novikova et al. [31] who use MSER. False extractions
are discarded by solving the cost minimization problem
described below.
5.2 Re-scoring with Four Lines
Character classification scores are modified on the basisof on layout consistency before the word inference process. The new scores are then calculated on the basisof goodness of fit between the estimated lines and theBi-directional Extraction and Recognition of Scene Text with Layout Consistency

9

5.3 Cost Minimization

Fig. 7: Sliding window re-extraction. The yellow region is thespace to be scanned, the green lines are the four estimatedlines of a text string, and the red rectangles are boundingboxes of reliable characters.

A

(1)

(2)

(3)

g

(1) For upper case, ascender, digit(2) For small lower case(3) For descenderFig. 8: The vertical position of windows in sliding window
re-extraction.

The application of character recognition to each candidate character region generates character classes withscores. However, the given set of character regions often
contains false positives, and corresponding characterclasses include classification failure. We select appropriate character regions as well as promising characterclasses by an energy minimization framework.
We first construct a graph with candidate character regions as nodes and place edges between suﬃciently
close nodes. Each node is represented by a random variable Xi that takes a label xi corresponding to a character class including non-character label ϵ. We then definea cost function by making use of character recognition
scores, spatial constraints, and linguistic knowledge andinfer the most likely word by minimizing the cost function.
We use a cost function similar to the one by Mishra
et al. [27]. The cost function E(x) is represented as asum of unary and pairwise terms, as follows:
E(x) =n!

Ei (xi ) +i=1

trained models of line positions, as follows:4#$(µ˜lcj − yil )2S(xi = cj ) = p(cj |xi )exp −σ˜lcj 2
l=1

%

,

(2)

where p(cj |xi ) is the original classification score for ith candidate classified as class cj . The scores for caseidentical characters, i.e., S(xi = cUj ) for the score ofL
the upper case and S(xi = cj ) for that of the lower,
are then calculated as
$
%
4
(µlc
˜ Uj − yil )2#
U
S(xi = cj ) = p(cj |xi )
exp −
, (3)
σlc
˜Uj 2
l=1
$
%
4
(µlc
˜ Lj − yil )2#
L
S(xi = cj ) = p(cj |xi )
exp −
. (4)
σlc
˜ Lj 2l=1This re-scoring process enables more accurate character recognition because geometric information (verticallocation) ignored in the first scan can be taken into account. For example, for a window between ascender line
and baseline, the score of the window classified as label ‘C’ does not decrease because estimated lines matchwith the trained line positions of ‘C’. In contrast, thescore of ‘c’ decreases because estimated lines do notmatch with the models. As an additional benefit, windows located far from four lines will be assigned lowscores for all character classes, which helps remove false
positives in the word inference stage.

!

Ei,j (xi , xj ),(5)εwhere x = {xi |i = 1, 2, ..., n} is the set of all randomvariables, Ei (xi ) is the unary cost, Eij (xi , xj ) is the
pairwise cost, and ε represents the set of all neighboring
pairs of nodes, which is determined by the structure of
the constructed graph. We use the same pairwise costas Mishra et al. [27], using a bi-gram model and spatialconstraint. We also use a similar formulation for unary
cost as Mishra et al. [27] but with a layout-based score(2) instead of raw classifier confidence:&
1 − S(xi = cj )if cj ̸= ϵ,
Ei (xi = cj ) =(6)
λmaxj S(xi = cj )if cj = ϵ,where S(xi = cj ) is the score calculated in (2) and λis the parameter of penalty for the node taking a nulllabel, which is set to 10 in our experiments. Given theseunary and pairwise terms, the word recognition result
is acquired by minimizing the cost function. To do this,following Mishra et al. [27] and Shi et al. [34], we use
the TRW-S algorithm [19].

6 Experiments
6.1 Datasets
We used several challenging public datasets—ICDAR
2003 [23], 2013 [18], 2015 [17], Street View Text (SVT)[39, 40], and IIIT 5K-Word [26]—to evaluate the word
recognition performance. The ICDAR datasets were scene10

text recognition datasets created for the ICDAR competition. The ICDAR 2015 dataset [17] was created forthe task of ”Challenge on Incidental Scene Text” introduced in ICDAR 2015, which is a more challenging
dataset than previous ICDAR datasets such as ICDAR
2003 and 2013. It contains more than 1,500 images acquired with wearable devices; thus, the images are often arbitrarily rotated and include bad conditions (e.g.,
blur, noise), unlike the focused scene text in previous
ICDAR datasets. The detailed evaluation protocol ofword recognition is described in Section 6.4. For ICDAR 2003, a 50-word lexicon is provided for each image
by Wang et al. [39].
The SVT dataset contains images taken from Google
Street View. Following the experimental protocol ofprevious works [39, 40], we used the SVT-WORD subset that contains 647 words, where a 50-word lexicon isassociated with each word.
The IIIT 5K-Word dataset is the largest dataset
for cropped word recognition, consisting of 2,000 word
images for training and 3,000 word images for testing.
Each test image has an associated 50- and 1000- word
lexicon for evaluation on a closed-vocabulary task.
We also used the training part of the ICDAR 2003
and IIIT 5K-Word datasets for training character classifiers because ground truth bounding boxes and character labels are provided on these datasets. Chars74K[10] was also used for training the character classifier.
6.2 Evaluation of Character Classifier
In this section, we present a detailed implementationand evaluation of the character classifier used in our experiments. HOG features were computed with a block
size of 2 × 2 cells and a cell size of 7 × 7 pixels using
nine bins after resizing each image to a 28 × 28-pixel
window. When the input image height was larger than
64 pixels, the input image was smoothed by a 4 × 4 median filter before resizing. We used the standard LIBSVM package [8] for training and testing the SVM classifier. The parameters of RBFSVM were determined by
five-fold cross validation on the training data; C=6.72
and γ=1.5 were obtained. We combined the ICDAR
2003 [23] training dataset, Chars74k [10] dataset, and
IIIT 5K-Word [26] training dataset as training data, resulting in a total of 27,715 characters. In all reported
experiments, we used a 55-class SVM trained on thesedatasets unless otherwise specified.
To evaluate the performance of the character classifier, we tested the recognition rate on the ICDAR 2003
testing dataset. Table 2 shows the error rates of thecharacter recognition compared to other methods. For
this comparison, the result using a 62-class SVM (lower

Ryota Hinami et al.

Table 2: Error rates of character recognition on ICDAR 2003
dataset.
MethodError (%)
Conv-Maxout [3]
CNN [41]
HOG + non-linear SVM (ours, 55-class)
HOG + non-linear SVM (ours, 62-class)Conv. Co-HOG [35]Co-HOG + linear SVM [38]TSM (49-class) [34]HOG + NN [40]ABBYY [1]

(a)

14.5
16.1
16.89
18.23
19
20.6
22.14
48.5
73.4

(b)

Fig. 9: Eﬀect of threshold of reliable characters

Table 3: Character extraction results on IIIT 5K-Worddataset.
Extraction method
Recall (%)
MSER + Otsu + re-extraction
MSER + Otsu
MSER
Otsu

81.25

72.96
68.24
55.29

and upper cases of case-identical characters are trained
separately) is also shown. The results indicate that ourcharacter classifier is comparable to the latest methods such as TSM [34] and Co-HOG [38, 35] but behind
Conv-Maxout [3] and CNN [41]. When we changed thetraining set to the combination of ICDAR 2003 training and Chars74K (same as Tian et al. [38] and Su etal. [35]), the error rate of our method on the 62-class
SVM increased to 19.4% but was still comparable to themethods with Co-HOG [38, 35]. We assume that the difference of performance between our method (HOG +non-linear SVM) and HOG + NN is due to the diﬀerence of classifier: non-linear SVM and nearest neighbor.

6.3 Evaluation of Character Extraction
After character candidate regions were obtained by MSER
and connected component analysis, we applied a char-Bi-directional Extraction and Recognition of Scene Text with Layout Consistency

Fig. 10: Relation between the number of scanned windows
per word and recall of character extraction on IIIT 5K-Worddataset. N is the number of vertical scales on conventionalsliding window.

acter classifier to obtain character classes with scores.
Reliable characters that had scores above a threshold(set to 0.8) were then selected. Although this threshold
was manually determined, we confirmed that the word
recognition performance is not very sensitive to thisvalue. Figure 9 shows the eﬀect of the threshold value
on word recognition performance (both open and closed
vocabulary tasks) and runtime tested on ICDAR 2003
dataset. The error rate is minimized with the thresholdof 0.5∼0.9 and the runtime of sliding window increases
gradually as the threshold increases.
We then estimated four lines of a text string fromreliable characters using the trained models of line positions. We then performed sliding window-based reextraction. We used 1.1, 1.4, 1.7, and 2.0 as the set of
possible aspect ratios. The step size of the scan was
fixed to 1/10 of the height of the window.
To evaluate the performance of character extraction
from cropped words, we measured the recall of character extraction on the IIIT 5K-Word testing dataset. We
computed the intersection over union (IoU) measureof a detected window compared to the ground truth
with a threshold of 60%. Table 3 summarizes the results. Otsu, MSER, and MSER + Otsu show the recall using only first CC-based extraction and MSER
+ Otsu + re-extraction shows the recall using both
first CC-based extraction and second sliding window reextraction. The combination of MSER and Otsu achieveda recall of 72.96%, which is far better than using only
MSER or Otsu. The recall was further improved by
8.29% (30.66% relative error reduction) by combination

11

with our sliding window re-extraction. We also measured the percentage of the words wherein all characters are correctly extracted by the proposed method(Otsu + MSER + re-extraction). We confirmed that
all characters in 72.93% of words (2,188 words / 3,000
words) are correctly extracted by the proposed methods, which shows that the character extraction errors
are concentrated on the other 27.07% diﬃcult words.
To determine the eﬃciency of our two-stage extraction method, we compared it with conventional multiscale sliding window extraction. We measured the recall of character extraction and evaluated the eﬃciency
by recall for the number of scans because computation
time is mainly occupied by computing features and applying classifiers. In the conventional sliding window
method, the height of the window was varied by thenscale s as follows: s = 2− 4 (n = 0, 1, ..., N − 1), where
N is the number of scales. The height of the window was
set to s times the height of the word image. The step
size was varied among 13 diﬀerent values (from 0.01 to10) times the height of the window for both (proposedand conventional) methods. Note that we do not apply NMS following the proposed method, because theperformance with NMS highly depends on the character classifier and the threshold of NMS and thus faircomparison is diﬃcult with NMS.
Figure 10 shows the relation between the number ofscanned windows per word and the recall of character
extraction (IoU >60% is used). The conventional sliding window method needed about 250 scans per word toachieve 80% recall while the proposed method achieved
80% with about 40 scans including the results of MSER
and Otsu. These results demonstrate that our method
achieves high computational eﬃciency while retaining
high recall. Although the conventional sliding window
method can exceed 90% recall with more than 1,000
scans for a given word image, it does so at a huge
computational cost (more than 25 times that of our
method).
6.4 Cropped Word Recognition
In this section, we provide a detailed evaluation of cropped
word recognition. To determine the pairwise cost, we
use a large English dictionary with around 0.5 million
words provided by Weinman et al. [45]. We evaluate our
performance on both a closed (limited) vocabulary task
and an open (unlimited) vocabulary task. In the closed
vocabulary task, we selected the word with the smallest edit distance in the lexicon as the final result. The
lexicon consists 50 words in the tasks of ICDAR03(50),
SVT, and IIIT5K(Small). All words in the test set areused as lexicon in ICDAR(FULL) and IIIT5K(Medium)12

has a 1,000-word lexicon. Open vocabulary tasks measure the performance by case sensitive accuracy without edit distance correction. We evaluate ICDAR 2013
and 2015 in an open vocabulary task following the evaluation of the competition. We also evaluate ICDAR
2003 with an open vocabulary task (referred to as ICDAR03(Open)) in addition to ICDAR03(50) and ICDAR03(FULL). IIIT5K(Large) can also be considereda case insensitive open vocabulary task because lexicon
with ground truth is not given. In the open-vocabularytask, to compare accuracy with another method thatuses a strong language model, we implemented a simple post-processing method based on a 4-gram-based
language model similar to the secondary language scoring in Bissacco et al. [6]. The language model is trainedwith 0.5 million words independent from the groundtruth lexicon.
Following previous papers [39, 27, 31, 34], words with
two or fewer characters or with non-alphanumeric characters were excluded on ICDAR 2003. However, on ICDAR 2013/2015, we followed the exact evaluation protocol of the competition for a fair comparison.
6.4.1 Evaluation of Sliding Window Re-extraction
We have shown in Section 6.3 that sliding window reextraction improves the recall of character extraction
eﬃciently. To investigate the eﬀect of re-extraction on
the accuracy of word recognition, we evaluated the accuracy of cropped word recognition by changing extraction methods. We performed evaluation of the following four tasks: ICDAR03(50), ICDAR03(FULL), ICDAR03(Open), and ICDAR13.
Table 4 shows the performance using diﬀerent extraction methods. The combination of MSER and Otsu
achieved much higher performance than either MSER
or Otsu alone. Moreover, sliding window re-extraction
further improved MSER+Otsu extraction. Figure 11
shows sample results to demonstrate the eﬀect of sliding window re-extraction. Missed and connected characters in the MSER+Otsu extraction were successfully
extracted by the proposed method, demonstrating theeﬀectiveness of the proposed re-extraction based on layout consistency.
6.4.2 Evaluation of Text Line-Based Re-scoring
To confirm the eﬀect of the re-scoring on the performance of character classification, especially for caseidentical characters, we evaluated the performance withand without the re-scoring on a case sensitive word
recognition task on the ICDAR 2003 and 2013 datasets.
We evaluated on following four tasks: ICDAR03(50),

Ryota Hinami et al.

Table 6: Word recognition results on ICDAR 2013.
Normalized
Methodedit distance Error (%)
PhotoOCR [6]

122.7

17.17

Proposed

329.5
332.4
360.1
539.0

41.00
42.01
35.80
54.70

PicRead [31]NESP [21]ABBYY [1]Table 7: Word recognition results on ICDAR 2015.
Normalized
Methodedit distance Error (%)
MAPS [20]NESP [21]DSM
Proposed1128.01164.61178.81232.7

67.07
68.32
74.15
69.81

ICDAR03(FULL), ICDAR03(Open), and ICDAR13. In
the method without the re-scoring, we used a 62-class
SVM as a character classifier.
The recognition rates on the ICDAR 2003/2013 datasets
are shown in Table 5. Although the performance of theclosed-vocabulary task was increased slightly by the rescoring, the performance of the open-vocabulary task
was increased significantly. This indicates that our rescoring stage corrects falsely recognized characters effectively although its eﬀect becomes weak when vocabulary is limited.
To investigate the contribution of a re-scoring stage
to each character class, we measured the character wise
recognition performance on the IIIT 5K-Word dataset.
Figure 12 shows the character wise recognition ratewith and without re-scoring. The rate increase by rescoring is also shown and the graph is arranged in theorder of this amount. Character classes that do not contain enough test data items (less than 15) are excluded
from the graph. As shown, the re-scoring has a positiveeﬀect on the classification, especially for case-identicalcharacters (c-o-s-v-w-x-z). On the other hand, performance of numeric characters decreases because theirposition relative to other characters varies widely andthis variation cannot be learned with limited training
data. Although performance decreased for some characters, the overall character level recognition error rate
was reduced from 26.79% to 24.35% (9.11% relative error reduction).
Figure 13 shows examples of the results to demonstrate the eﬀect of the re-scoring. These results indicatethat the re-scoring stage is eﬀective to eliminate false
positives and to distinguish confusing characters suchas case-identical characters.

Bi-directional Extraction and Recognition of Scene Text with Layout Consistency

13

Table 4: Word recognition error rates on ICDAR 2003 dataset with diﬀerent extraction methods.
MethodICDAR03(50) ICDAR03(FULL) ICDAR03(Open) ICDAR13
MSER + Otsu + re-extraction
MSER + Otsu
MSER
Otsu

12.21

15.58

38.57

41.00

13.72
15.70
21.51

17.91
22.56
26.28

45.93
57.90
54.65

41.83
50.68
54.16

Without re-detection

CILANEISPWP

tOLATON

GzRAG

factryWith re-detection

CLEANERS

PULP

KOLESTONGARAGE

factory

Fig. 11: Diﬀerences of word recognition results with and without re-extraction. These examples are from ICDAR 2003/2013,
SVT, and IIIT 5K-word datasets. The recognition results are obtained under an open vocabulary condition.

Fig. 12: Character wise recognition error rate on IIIT 5K-Word dataset. The error rates with and without re-scoring are shownby points and the relative error reduction by re-scoring is shown by bars. The graph is arranged in the order of relative error
reduction.

6.4.3 Comparison with Other Methods
We compared our method with several competing word
recognition methods. We first discuss the evaluation on
the ICDAR 2013/2015 dataset. We followed the sameevaluation protocol as other competitors for fair comparison. The results on the ICDAR 2013 dataset are
shown in Table 6. The proposed method with re-scoring

was ranked second among all methods tested according
to the oﬃcial metrics of the normalized edit distances
to the correct answers, bested only by PhotoOCR [6].
Although Jaderberg et al. [14] also report the accuracy(90.8%), their evaluation protocol is diﬀerent: groundtruth lexicon is given and words with two or fewer characters or with non-alphanumeric characters were excluded.

14

Ryota Hinami et al.

Table 5: Word recognition error rates on ICDAR 2003 to show the eﬀect of text line-based re-scoring.
MethodICDAR03(50) ICDAR03(FULL) ICDAR03(Open) ICDAR13
With re-scoring
Without re-scoring

12.21

15.58

38.57

41.00

13.02

16.40

52.91

47.58

Without re-scoring

wivenhOLe

GIFrs

vour

shoP

drts

ENr

With re-scoring

Wivenhoe

GIFTS

your

Shop

arts

ENT

Fig. 13: Diﬀerences of word recognition results with and without re-scoring. These examples are from ICDAR 2003/2013, SVT,
and IIIT 5K-word datasets. The recognition results are obtained under open vocabulary condition.

Table 8: Word recognition error rates of proposed method and other methods on ICDAR 2003, SVT, and IIIT 5K-Word.
MethodProposed
Wang et al. [41]
Novikova et al. [31]Almazan et al. [2]Gordo et al. [12]Alsharif and Pineau [3]Jaderberg et al. [14]Bissacco et al. [6]Yao et al. [46]Lee et al. [22]Shi et al. [34]Mishra et al. [26]ABBYY [1]ICDAR03(FULL)ICDAR03(50)

SVT

IIIT5K(Large)

IIIT5K(Medium)

IIIT5K(Small)

15.58
16
17.2
11.4

12.21
10
6.9

23.49
30
27.1
12.99
25.7

51.53

19.77
24.40
13.4
-

12.60
11.43
6.7
-

1.4

1.3

4.6

7.3

2.9

19.67
24
20.70
45

11.52
12
12.56
19.72
44

9.61
24.11
20
26.49
26.43
65

30.7
42.50
-

19.8
31.75
75.67

The results on the ICDAR 2015 dataset are shown in
Table 7. Our method is behind the other three submitted methods, although there is no big diﬀerence in error
rates. MAPS [20] and NESP [21] both perform segmentation first and the binarized image is processed using OCR software. DSM used a deep structured modelwith convolutional neural networks for local character scoring. It seems our method cannot extract reliable characters with simple MSER and Otsu binarization, because ICDAR 2015 datasets contain many
ill-conditioned words with noise and blur. To improve
the accuracy of our method for ill-conditioned text,
we should use a more sophisticated character extraction method such as the binarization techniques usedin NESP and MAPS.
The performances on the ICDAR 2003, SVT, and
IIIT 5K-Word datasets are shown in Table 8. Our method
outperforms Mishra et al. [27], Shi et al. [34], Yao et
al. [46], Novikova et al. [31], and ABBYY [1] and iscompetitive with Wang et al. [41] and Lee et al. [22].
However, our method is still behind Almazan et al. [2],

61.7
55.70
-

Gordo et al. [12], Alsharif and Pineau [3], Bissacco etal. [6], and Jaderberg et al. [14]. These methods take awhole word-based approach [2, 12, 14] and/or use deepneural networks [6, 3]. This is discussed in detail laterin this subsection.
Our method outperforms Mishra et al. and Shi etal. [26, 34], which uses the similar cost function, whichsuggests that our candidate regions and accompanying classification score are better than these methods.
There are two main reasons for this: (1) our two-stage
character extraction achieves high recall with a small
number of false alarms, and (2) the unary cost is computed by using four estimated lines, which improves
character classification accuracy while reducing false
positives.
Examples of successful recognition results are shownin Fig. 14(a). As we can see, the proposed method wasable to recognize scene text with low resolution, diﬀerent fonts, and distortions. Figure 14(b) shows the casesof incorrect recognition. The proposed method with thefour-line model could not recognize words with a curved

Bi-directional Extraction and Recognition of Scene Text with Layout Consistency

FamousStewartSTARBUCKS

Greenstead

DEBENHAMSPENALTY

15

from

CLEANED

BROW

GHOST

CIRCLE

Wivenhoe(a)
HvtYARYXOUERBX

SrRE

iUsTORaALfA(b)Fig. 14: Examples of word recognition results of proposed method with four-line model. (a) Successfully recognized words. (b)Incorrectly recognized words.

text line, largely inclined characters, and unconstrained
a character classifier and achieve a significantly higher
style.
character classification performance, as shown in TaLimitations of whole word-based approach.
ble 2. Since our method is compatible with any characThe performance of our method is behind Gordo etter classifier, we believe it can achieve an even betteral. [12], Almazan et al. [2], and Jaderberg et al. [14],performance if we replace our character classifier withwhich recognize text from the whole word images. Howa more powerful one. Moreover, since our re-extraction
ever these methods have some limitations. First, theyand re-scoring is separated from the final inference proface diﬃculty for cases without a lexicon, i.e., an opencess, our method can be incorporated with other methvocabulary condition. Gordo et al. [12] and Almazan
ods (if not the whole word-based method) to refine theet al. [2] deal with the word recognition problem as a
candidate regions.
retrieval problem from a given limited lexicon. JaderIn addition, these methods require a huge amountberg et al. [14] train the model with a pre-defined lexiof training images. Bissacco et al. use 2.2 million handcon (about 90K words). The result is that words out oflabeled character images and Jaderberg et al. [14] use 9lexicon cannot be recognized by these methods. Since
million synthetic word images. Wang et al. [41] and Alscene texts often contain words that are not listed insharif and Pineau [3] use about 130k character images
the dictionary and are impossible to know in advance,
while we use 27,715 characters. Although training ima method that can handle the open-vocabulary task isages can be prepared by synthetic generation, as Wanguseful.
et al. [41] and Jaderberg et al. [14] do, training deep
Second, these methods cannot handle significant changesneural networks with large datasets imposes a muchin layout because they jointly and implicitly handle laylengthier training time and high computational power.
out consistency (character alignment) in a holistic way.
Since we explicitly handle layout consistency, we argue
6.5 Curved Text
that our method is advantageous for use with significant changes in layout (e.g., from linear to curved) by
In this section, we provide an evaluation of the curved
explicit adaptation, whereas the holistic approaches retext recognition discussed in Section 4.4. We first judge
quire re-learning.
if a given word is straight or curved. The selected style
Deep neural networks. Bissacco et al. [6], Alis then estimated and re-extraction and re-scoring aresharif and Pineau [3], Wang et al. [41], and Jaderberg
performed using the estimated style.
et al. [14] exploit deep neural networks and outperform
Figure 15 shows a comparison of the curved word
our method. This is mainly due to the high performancerecognition
results by the four line-based method, whichof the CNN-based character classifier. For example, Alcould not recognize curved words, and the four circlesharif and Pineau use convolutional maxout networks asbased method, which could. We then measured the over-16

Ryota Hinami et al.

Table 9: Comparison of word recognition performance between method with adaptive model selection and only fourline model. (a) Tested on full datasets. (b) Tested on the subset that contains curved cases only. (c) Tested on the subsetthat contains linear cases only.
(a) All cases
Dataset
IIIT5K (Small)
IIIT5K (Medium)
IIIT5K (Large)
SVT
ICDAR03(50)
ICDAR03(FULL)

Four-line model

Adaptive model

12.23
19.10
52.00
23.49
12.21
15.58

11.13
17.77
51.67
23.80
12.44
15.93

(b) Curved cases only
Dataset
IIIT5K (Small)
IIIT5K (Medium)
IIIT5K (Large)

Four-line model

Adaptive model

49.10
61.21
89.09

29.70
38.79
83.03

Four-line model

Adaptive model

49.84
16.54
10.05

49.84
16.61
10.09

(c) Linear cases only
Dataset
IIIT5K (Small)
IIIT5K (Medium)
IIIT5K (Large)

Table 10: Average computational time (per word) of diﬀerent
components of the proposed method.
Procedure

Processing time

Connected component-based extraction
First recognition
Line estimation
Sliding window re-extraction
Line-based re-scoring
CRF

0.031 sec
1.30 sec
0.0016 sec
1.59 sec
0.005 sec
0.080 sec

cessing time is dominated by the character classifier,which accounts for 96% of the overall time including
first recognition and re-extraction. Although our additional stages of the pipeline appear to involve costly
operations, every stage except for first recognition andre-extraction is computationally eﬃcient. Moreover, ourmethod reduces the number of classifier evaluations, as
discussed in Section 6.3 (Fig. 10). If our character classifier is replaced with a more eﬃcient one, the processing
time will be reduced significantly and our method could
be scalable to large datasets.

all performance on the IIIT 5K-Word, SVT, and ICDAR 2003 datasets. We counted the number of curved
words manually for each datasets: IIIT 5K-Word and7 Conclusion
SVT contain 165 words (5.5%) and 10 words (1.55%),
respectively, while ICDAR 2003 contains none. The method
In this paper, we presented a method that can improvewith adaptive model selection is compared with theperformance by conducting character extraction andmethod using only a four-line model. The performancesword recognition in a bi-directional way. Our key novon several datasets are shown in Table 9. In IIIT5K,
elty resides in the estimation of four lines of text string
we evaluate the performance tested on the subsets thatby character recognition results, not character extraccontain curved cases or linear cases only, shown in Tation results. We utilize the estimated four lines to refineble 9(b), (c). The error rates were decreased in the IIIT
the character extraction and recognition results on the
5K-Word full dataset and a significant error decreasebasis of the layout consistency of a text string. The
can be seen in Table 9 (b), which shows the errors
benefits of this are that 1) we can take into account acevaluated on curved cases only. Accuracies were largely
curate geometric information to recognize character byunchanged in ICDAR 2003, which includes no curved
re-scoring process and 2) the hybrid of CC-based andcases, indicating that style selection is successful andsliding window-based extraction achieves both compuperformance does not decrease even if other styles aretational eﬃciency and high recall of character extracconsidered (Table 9(c) also supports this). Since curvedtion.
cases in SVT were very diﬃcult to recognize because
Although the final accuracy of our method was inof complicated background or sharp curves, we cannot
ferior to that of state-of-the-art methods, each comporecognize only one word among them, even with ournent is easily convertible to another stronger compoadaptive model.
nent because our method does not depend on any specific character classifier, character extraction method,or cost function. Moreover, since our re-extraction and
6.6 Computational Timere-scoring is separated from the final inference process,We analyzed the computational time of diﬀerent comour method can be incorporate with other methods inponents of the proposed method. The processing timedependent of other components to refine the candidate
for each procedure is summarized in Table 10. The proregions.

Bi-directional Extraction and Recognition of Scene Text with Layout Consistency

17

Four lines

SrRE

RANAPA

ROSSQAGWELSC

Four circles

STREETGRANARYPROSPECTCHELSEA

Fig. 15: Diﬀerences of word recognition result between four-line model and four-circle model. These examples are from IIIT
5K-word datasets. The recognition results are obtained under an open vocabulary condition.

References
1. Abbyy finereader 9.0. http://www.abbyy.com.
2. J. Almazan, A. Gordo, A. Forn, and E. Valveny. Word
Spotting and Recognition with Embedded Attributes.
PAMI, pages 1–17.
3. O. Alsharif and J. Pineau. End-to-End text recognition
with hybrid HMM maxout models. ICLP, oct 2013.
4. I. Bazzi, R. Schwartz, and J. Makhoul. An omnifont open-vocabulary ocr system for english and arabic.
PAMI, 21(6):495–504, 1999.
5. Y. Bengio and Y. LeCun. Word normalization for on-line
handwritten word recognition. In IAPR, 1994.
6. A. Bissacco, M. Cummins, Y. Netzer, and H. Neven. Photocr: Reading text in uncontrolled conditions. In ICCV,2013.
7. T. Caesar, J. M. Gloger, and E. Mandler. Estimating thebaseline for written material. In ICDAR, 1995.
8. C.-C. Chang and C.-J. Lin. LIBSVM : A library forsupport vector machines. ACM Transactions on Intelligent
Systems and Technology (TIST), 2011.
9. N. Dalal and B. Triggs. Histograms of oriented gradientsfor human detection. In CVPR, 2005.
10. T. de Campos, B. R. Babu, and M. Varma. Character
recognition in natural images. In VISAPP, 2009.
11. V. Goel and W. Ecole. Whole is greater than sum ofparts : recognizing scene text words. In ICDAR, 2013.
12. A. Gordo. Supervised mid-level features for word image
representation. In CVPR, 2015.
13. W. Huang, Y. Qiao, and X. Tang. Robust scene text
detection with convolution neural Network induced mser
trees. In ECCV, 2014.
14. M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman. Reading Text in the Wild with Convolutional Neural Networks. IJCV, pages 1–20, 2014.
15. M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman. Synthetic Data and Artificial Neural Networks for
Natural Scene Text Recognition. NIPS deep learning workshop, pages 1–10, 2014.
16. M. A. Jones, G. A. Story, and B. W. Ballard. Integrating multiple knowledge sources in a bayesian ocr postprocessor. ICDAR, 1991.
17. D. Karatzas, L. Gomez-bigorda, A. Nicolaou, S. Ghosh,A. Bagdanov, M. Iwamura, J. Matas, L. Neumann, V. R.
Chandrasekhar, S. Lu, F. Shafait, S. Uchida, and E. Valveny. ICDAR 2015 Competition on Robust Reading. In
ICDAR, 2015.
18. D. Karatzas, F. Shafait, S. Uchida, M. Iwamura, L. G. I.
Bigorda, S. R. Mestre, J. Mas, D. F. Mota, J. A. Almazan, and L. P. de las Heras. ICDAR 2013 robust reading competition. In ICDAR, aug 2013.
19. V. Kolmogorov. Convergent tree-reweighted message
passing for energy minimization. PAMI, oct 2006.

20. D. Kumar, M. Prasad, and A. Ramakrishnan. Maps: midline analysis and propagation of segmentation. In Proceedings of the Eighth Indian Conference on Computer Vision,
Graphics and Image Processing, page 15. ACM, 2012.

21. D. Kumar, M. A. Prasad, and A. Ramakrishnan. Nesp:
Nonlinear enhancement and selection of plane for optimal segmentation and recognition of scene word images.
In IS&T/SPIE Electronic Imaging, pages 865806–865806.
International Society for Optics and Photonics, 2013.
22. C.-Y. Lee, A. Bhardwaj, W. Di, V. Jagadeesh, and R. Piramuthu. Region-based discriminative feature pooling forscene text recognition. In CVPR, 2014.
23. S. Lucas, A. Panaretos, L. Sosa, A. Tang, S. Wong, andR. Young. ICDAR 2003 robust reading competitions. In
ICDAR, number Icdar, 2003.
24. J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust
wide-baseline stereo from maximally stable extremal regions. Image and Vision Computing, 22(10):761–767, sep
2004.
25. E. G. Miller and P. A. Viola. Ambiguity and constraint
in mathematical expression recognition. In AAAI/IAAI,
pages 784–791, 1998.
26. A. Mishra, K. Alahari, and C. V. Jawahar. Scene text
recognition using higher order language priors. In BMVC,
2012.
27. A. Mishra, K. Alahari, and C. V. Jawahar. Top-down
and bottom-up cues for scene text recognition. In CVPR,
2012.
28. L. Neumann and J. Matas. A method for text localization
and recognition in real-world images. In ACCV, 2010.
29. L. Neumann and J. Matas. Text localization in realworld images using eﬃciently pruned exhaustive search.
In ICDAR, 2011.
30. L. Neumann and J. Matas. Real-time scene text localization and recognition. In CVPR, 2012.
31. T. Novikova, O. Barinova, P. Kohli, and V. Lempitsky. Large-lexicon attribute-consistent text recognitionin natural images. In ECCV, 2012.
32. N. Otsu. A threshold selection method from gray-level
histograms. Automatica, 1975.
33. P. Sarkar and G. Nagy. Style consistent classification of
isogenous patterns. PAMI, 2005.
34. C. Shi, C. Wang, B. Xiao, Y. Zhang, S. Gao, andZ. Zhang. Scene text recognition using part-based treestructured character detection. In CVPR, 2013.
35. B. Su, S. Lu, S. Tian, J.-H. Lim, and C.-L. Tan. Character recognition in natural scenes using convolutional
co-occurrence hog. In ICPR, 2014.
36. J. B. Tenenbaum and W. T. Freeman. Separating styleand content with bilinear models. Neural computation,
12(6):1247–1283, 2000.
37. C. Thillou, S. Ferreira, and B. Gosselin. An embedded application for degraded text recognition. EURASIP Journal
on applied signal processing, 2005:2127–2135, 2005.

18
38. S. Tian, S. Lu, and B. Su. Scene text recognition using cooccurrence of histogram of oriented gradients. In ICDAR,
2013.
39. K. Wang, B. Babenko, and S. Belongie. End-to-end scene
text recognition. In ICCV, 2011.
40. K. Wang and S. Belongie. Word spotting in the wild. In
ECCV, 2010.
41. T. Wang, D. J. Wu, and A. Y. Ng. End-to-end text
recognition with convolutional neural networks. In ICPR,
2012.
42. J. Weinman and E. Learned-Miller. Improving recognition of novel input with similarity. CVPR, 2006.
43. J. J. Weinman. Typographical features for scene text
recognition. ICPR, 2010.
44. J. J. Weinman, Z. Butler, D. Knoll, and J. Feild. Toward
integrated scene text reading. PAMI, 2014.
45. J. J. Weinman, E. Learned-miller, and A. R. Hanson.
Scene text recognition using similarity and a lexicon withsparse belief propagation. PAMI, 2009.
46. C. Yao, X. Bai, B. Shi, and W. Liu. Strokelets : A learned
multi-scale representation for scene text recognition. In
CVPR, 2014.

Ryota Hinami et al.