Could Scene Context be Beneﬁcial for Scene Text
Detection?
Anna Zhua,∗, Renwu Gaob , Seiichi Uchidaba Huazhong University of Science and Technology, State Key Lab for Multispectral

Information Processing Technology , School of automation, Wuhan, Chinab Kyushu University, Human Interface Laboratory, Information Science and ElectricalEngineering, Fukuoka, Japan

Abstract
Scene text detection and scene segmentation are meaningful tasks in the computer vision ﬁeld. Could the semantic scene segmentation assist scene text
detection in any degree? For example, can we expect the probability of a region
being text is low if its surrounding segment, i.e., its context, is labeled as sky? In
this paper, we have a positive answer by constructing a scene context-based text
detection model. In this model, we use texton features and a fully-connectedconditional random ﬁeld (CRF) to estimate pixel-level scene class’s probability
to be considered as image’s context feature. Meanwhile, maximally stable extremal regions (MSERs) are extracted, integrated and extended as image patches of character candidates. Then, each image patch is fed to a simple two-layer
convolutional neural network (CNN) to automatically extract its character feature. The averaged context feature of the corresponding patch is considered as
the patch’s context feature. The character feature and context feature are fusedas the input into a support vector machine for text/non-text determination. Finally, as a post-processing, neighboring text regions are grouped hierarchically.
The performance evaluation on ICDAR2013 and SVT databases, as well as apreliminary evaluation on a patch-level database, proves that the scene context
can improve the performance of scene text detection. Moreover, the compara∗ Corresponding author: Tel.: +86-186-2770-8137;

Email address: annakkk@live.com (Anna Zhu)

Preprint submitted to Journal of Pattern Recognition

March 24, 2016

tive study with state-of-the-art methods shows the top-level performance of our
method.
Keywords: Scene text detection, Fully connected CRF, Convolutional neuralnetwork, Character feature, Context feature.

1. Introduction
Recently, scene text reading captures much attention in the computer visionﬁeld [1, 2]. It refers to an attempt to recognize text from camera-captured
images and contains two parts: scene text detection and scene text recognition.
5

The technologies of this area can be applied to various applications, such as
language translation system, visual-based navigation, and content-based web
image search. Accurate detection of text in natural scene images is an essential
step and primary work for successfully recognizing scene text. Therefore, we
focus our attention on scene text detection in this paper.

10

Scene text detection meets great challenge [3, 4]. The diﬃculty comes fromthe huge variations of scene text. Speciﬁcally, scene text (1) varies in size, color,
font, and style; (2) has no clear layout; (3) tends to have complex background,and (4) undergoes non-uniform lighting, partial occlusion, blur, rotation, perspective distortion, and low resolution.

15

To tackle with those diﬃculties, we propose a framework to detect scene text
by means of scene context information. The idea arises from the observationthat text appear frequently in certain scene context and rarely in others. For
example, they are always embedded in signboard, cars, books and other surfaces,
but rarely appear in rivers, the sky, trees or grass. If this idea is valid, a better

20

detection performance can be expected by using the framework shown in Fig. 1.
In this framework, we use not only the character feature but also context
feature for discriminating a connected component (CC) into text or non-text.
In this situation, even if the character feature suggests that a non-text CC is acharacter or a part of a character, it may still be discriminated into non-text,

25

when the context feature suggests its surrounding segment is “sky”. Conversely,2

Original RGB image

Connected
component (CC)

CharacterfeatureContext
featureSemantic segmentation result

signboard

Char/non-charclassificationof the CC

Scene context
around the CC

tree
Figure 1: Our hypothesis: confuse context information for classiﬁcation.

even if the character feature suggests that a text CC is non-text, it can bedetected as text, when the context feature suggests its surrounding segment is“signboard”.
Scene context information represents the scene class attribution of pixels
30

in the images, which can be extracted through semantic scene segmentation.
Though semantic scene segmentation has been another diﬃcult task, recent researches [5, 6] show very promising results. In our approach, we employ Philipp’s
work [7], in which images are segmented and semantic labeled in pixel-level by
TextonBoost and then reﬁned by a fully-connected conditional random ﬁeld

35(CRF). From the segmentation result, we obtain K-dimensional feature vectorof each pixel, namely the scene context, where K is the number of scene classes(such as sky and signboard) and the kth element of the vector is a probability
that the pixel lies on the kth scene class. The averaged scene context probabilities of a CC region is considered as this CC’s context feature. The context

40

feature is then combined with the character feature generated by a two-layer
convolutional neural network (CNN) [8] and fed to a support vector machine(SVM) classiﬁer for text/non-text classiﬁcation.
The contribution of our work is threefold. The ﬁrst contribution is to propose
a new idea of utilizing scene context for scene text detection. In traditional3

45

scene text detection researches, they focus only on “text-ness” at each region
in a scene image. Speciﬁcally, they assume several heuristics about texts, suchas, high contrast, high spatial frequency, dense edges, uniform stroke width,
and then use them for detection. Even though these heuristics work reasonably,
they cannot avoid false alarms and false negatives. False alarms will be detected50around text-like shapes in scene. For example, car tires can be detected as “o”
because it satisﬁes most of the above heuristics. False negatives will happen on
texts with irregular appearance. For example, texts printed in a fancy font may
not have uniform stroke width and texts on complex background may not haveenough contrast. Some recent methods are free from those heuristics by using

55

machine learning techniques, but they learn text-ness only and thus still have
similar false alarms and false negatives. The introduction of scene text will bean essential solution for this problem.
The second contribution is to emphasis the usefulness of semantic segmentation for scene text detection. Semantic segmentation is a rather new technology

60

and developed mainly for scene object recognition and scene understanding. To
our best knowledge, semantic segmentation has never been used for scene text
detection, i.e., there is no class “text” in the task of semantic scene segmentation. This may come from that most semantic segmentation methods assume
some smoothness and prevent detecting very ﬁne and thin structures, i.e., texts.

65

We will see that semantic segmentation is still very useful for scene text detection by the help of CC-based text detection strategy, even though semantic
segmentation has no enough resolution to detect individual text by itself.
The third contribution of our work is that, to prove our hypothesis, four
variable controlled comparison experiments, which control the variable whether

70

to use context feature and how to train CNN, are performed to provide adequate
and valid proof for test. Besides, a patch-level database is given in this work.
In this database, all the patch images are manually labeled and cropped fromtext regions in non-overlapping images of the ICDAR 2011 and ICDAR 2013’s
training databases. The results on this patch-level database and ICDAR 201375

database achieve convincible performance.
4The following section gives an overview of our pipeline. We review related
work in two directions in Sect. 2. Sect. 3 presents our proposed method in
detail. In Sect. 4, we give the experimental results which include the details of
databases and the experimental setup. Finally, Sect. 5 gives a summarization80

and conclusion of this paper.

2. Related work
In this section, we summarize some related previous work. Since our work
refers to scene segmentation but focus on scene text detection, we introduce
the related works on scene segmentation brieﬂy and text detection methods
85

emphatically.
2.1. Scene segmentation methods
Scene image segmentation aims to label every pixel in the image with several predetermined classes, thus concurrently perform recognition and segmentation of multiple classes. The segmentation techniques can be classiﬁed as90

follows: graph-based approaches, region-based approaches, boundary detection
approaches, perceptual organization approaches, multi-class image segmentation, and hybrid approaches.
The graph-based image segmentation approach deﬁnes the boundaries between regions by measuring the dissimilarity between the neighboring pixels by

95

a graph, where the node representing each pixel and the weights denoting thedissimilarity between pixels [9]. Region-based techniques make use of common
patterns in intensity values within a cluster of neighboring pixels and group
regions according to their anatomical or functional roles [10]. The boundary detection approach refers to a contour in the image plane that represents dissimilar

100

pixels between the neighboring segments [11]. Perceptual organization refers toa basic capability of the human visual system to obtain relevant groupings andstructures from an image without having prior knowledge of the image’s contents [12]. Multi-class image segmentation uses one of a number of classes (e.g.,5road, sky and water, etc) for labeling every pixel in an image. Many state-of105

the-art methods ﬁrst over-segment the image into superpixels (or small coherent
regions) and then classify each region [13, 14]. Hybrid techniques combine theabove segmentation approaches [15] to take advantages of them. Our scene segmentation method belongs to hybrid method, because it combines multi-classimage segmentation methods and graph-based methods together.

110

2.2. Scene text detection methods
Scene text detection involves extracting regions that contain text and ﬁltering out regions without text in the images. It is a typical classiﬁcation problem.
We classify the text detection methods into three categories: global-classiﬁcation
methods, local-classiﬁcation methods, and hybrid methods.

115

The global-classiﬁcation methods split images to region-based patches andattempt to classify these patches into text/non-text by using feature, such as
Histogram of Oriented Gradients (HOG) [16], Local Binary Pattern (LBP) [17],and learns classiﬁers [18, 19] on these features to discriminate text and non-text.
Those patches are selected by scanning the whole image with sliding windows.

120

To resist diﬀerent sizes of text, multi-scale strategy are typically used. Afterclassiﬁcation, the patches containing text are merged into text blocks. Chen etal. [20] trained a cascade of 4 strong AdaBoost classiﬁers containing 79 informative features to classify the regions. Wang et al. [21] classiﬁed all the patches
extracted from the sliding window by using a random ferns classiﬁer trained on

125

HOG features. Neumann et al. [22] used sliding windows for stroke detection
by convolving the gradient ﬁeld with a set of oriented bar ﬁlters. Mishra etal. [23] exploited bottom-up cues that classify individual character regions by a
HOG-based SVM classiﬁer and top-down cues that prune false detected regionswith a CRF model. Wang et al. [8] normalized all the extracted regional patch-130

es to input into a CNN classiﬁer and showed that a deep feature-based CNN iseﬀective for character patch classiﬁcation. These kinds of methods treat text
detection as object detection tasks. They can capture the inter-relationships oftext, by which text can be detected even if there are noises. However, they often
6use exhausted searching strategy, which results in the signiﬁcant computational
135

cost.
The second category ﬁlters out most non-text pixels initially and uses distinct features based on CC level to extract the candidate characters on the rest
pixels. It groups spatial pixels to CCs conditionally. Either CC level features orextended region-based features are inputted into trained classiﬁer or heuristic

140

rules to discriminate text components from non-text components. Candidate
characters are grouped into words based on their relationship like location, text
line, distance, stroke width and color features. Epstein et al. [24] proposed
the Stroke Width Transform (SWT) algorithm which computed per pixel thewidth of the most likely stroke containing the pixel to obtain text CCs. Many145

modiﬁed SWT methods [25, 26] were proposed with the same idea. The MSER
method [27] and ER method [28] extracted some extremal regions as candidate
characters. These kinds of methods showed its outstanding eﬀectiveness andhigh eﬃciency on text detection area. Yin et al. [29] exploited a MSER tree
to extract character candidates and learned the distance weights and cluster-150

ing threshold to cluster the character candidates into text candidates. Further,
they proposed a multi-orientation text detection system [30] to construct text
candidates through three sequential coarse-to-ﬁne grouping steps with adaptive
clustering and distance metric learning. Sun et al. [31] extracted color-enhanced
CERs as character candidates in six component-trees which were built from the155

gray scale, hue and saturation channel images in a perception-based illumination invariant color space, respectively. Because all of the features extracted
from these methods are normalized in process, it is convenient in dealing withtext of variant scales and orientations. Besides, the detected result can be useddirectly for text recognition.

160

The third category combines the advantages of the two above categories inorder to achieve higher robustness while keeping lower computational cost. In
Pan’s work [32], a region-based method ﬁrst estimated the text existing conﬁdence, and then non-text components are ﬁltered out by CRF. Huang et al. [33]used MSERs by incorporating with a text saliency map which is generated by a7

Figure 2: The ﬂowchart of the proposed method.

165

strong CNN classiﬁer to eﬃciently prune the trees of Extremal Regions. Thesehybrid methods ﬁrst roughly select candidate text regions with one category
and then use other strategies to conﬁrm the conﬁdence of text regions and ﬁlter
out non-text regions.

3. The proposed method
170

3.1. Overview of our approachA ﬂowchart of our text detection framework is presented in Fig. 2. Our
method belongs to local-classiﬁcation category as it extracts MSERs in images.
To extract the context feature of each pixel, a fully connected CRF modelthat is deﬁned on the complete set of pixels in an image is used as a tool to

175

update pixel-level context probabilities, which are initialized by TextonBoost.
It is a global-classiﬁcation process, but only context feature of pixels who lay inthe extended rectangle regions of MSERs are utilized.
Four sets of parameters are used to get the MSER maps. We can get four
diﬀerent MSER maps by processing on the original gray-scale image, then inte-

180

grate them to one map. The same process is performed on the 0-255 reversed
gray-scale image and ﬁnally there are two integrated MSERs maps without multiple repeating structures. All the integrated MSERs are converted to boundingbox regions and then extened to larger size based on the bounding box’s width8and height. They are normalized to the same size and fed to CNN for character
185

feature extraction.
For text/non-text classiﬁcation, the CNN classiﬁer is selected because of its
excellent performance on object classiﬁcation [34] and character classiﬁcation[35, 36]. There are two schemes to combine context feature. One is using
context features and image patches to train CNN concatenated with a SVM

190

classiﬁer together. The parameters of CNN and SVM ﬁne tuning sequentially.
The other is to train a SVM classiﬁer with character features which is generated
by a pre-trained CNN from image patches and add the corresponding context
features. The parameters in CNN and SVM are trained separately. These two
schemes correspond to scheme (2) and (3) in the experiment section. And both

195

schemes illustrate that scene context features can aﬀect the text detection result
to some degree.
After classiﬁcation, a hierarchical grouping process is proposed to retrieve
missing characters and connects character to words. This provides the possibility for improving detection results on the basis of the former stage. In the200

following sections, each part will be described in detail.
3.2. Context feature extraction
In this section, the context feature of each pixel is computed by TextonBoost and reﬁned by a fully-connected CRF. It is a 14-dimensional vector which
presents the probability of pixel belonging to K=14 semantic scene labels. The

205

14 semantic scene labels are sky, trees, grass, rivers, houses, books, cars, airplanes, signboards, human faces, human clothes, walls, roads and text streamers.
We statistically compute a histogram of all the objects appeared in scene text
images of ICDAR 2011 and ICDAR 2013 training databases. Scene classes withhigh frequency of appearance in the databases and high probability of containing

210

text are selected. Finally we obtain the 14 classes.

93.2.1. Preliminary context feature generation by TextonBoost
The initial context feature, namely the probabilities, is derived from TextonBoost [13, 37]. The features are obtained by convolution the input image
with a 17-dimensional ﬁlter bank suggested by Shotton et al. [13]. And then,215

follow the work from Ladick et al. [37] by adding color, HOG, and pixel location features. 12 objects categories’ training images are obtained from theMSRC-23 database [13] and the training images for wall and text streamer are
either downloaded from Google images or ICDAR211 database. The output of
TextonBoost is considered as the unary potentials of Fully-connected CRF.

220

3.2.2. Updating pixel-level context probability using fully-connected random ﬁeld
Basic CRF models are composed of unary potentials on individual pixels or image patches, and binary potentials on neighboring pixels or patches[14, 38, 39]. The resulting adjacency CRF structure is limited in its ability
to model long-range connections within images and generally results in exces-

225

sive smoothing of object boundaries. To avoid the above problem, we use afully-connected CRF. It establishes pairwise potentials on all pixels, by which
it can greatly reﬁne segmentation and labeling. Here, we brieﬂy introduce thefully-connected CRF model.
Consider a random ﬁeld Y deﬁned over a set of variables {Y1 , · · · , YN }. The

230

domain of each variable is a set of labels L = {l1 , · · · , lK }, K is the number ofclasses. Consider also a random ﬁeld X deﬁned over variables {X1 , · · · , XN }.
In our setting, X ranges over possible input image’s size N and Y ranges overpossible pixel-level image labeling. Xj is the observation vector of pixel j and
Yj is the label assigned to pixel j.

235

A CRF (X; Y) is characterized by a Gibbs distribution, as in Eq. 1:()∑
1
P (Y|X) =exp −ϕc (Yc |X) ,
Z(X)

(1)

c∈CG

where each clique c in a set of cliques CG in G induces a potential ϕc . G = (V, E)

10

is a graph on Y. The Gibbs energy of a labeling y ∈ LN is:
E(y|X) =∑ϕc (yc |X) .

(2)

c∈CG

The maximum a posteriori (MAP) labeling of the random ﬁeld is y∗ =argmaxy∈LN P(y|X). For notational convenience we will omit the conditioning
in the rest of the paper and use ψc (yc ) to denote ϕc (yc |X).
In the fully-connected CRF model, G is the complete graph on Y and CG
240

is the set of all unary and pairwise cliques. The corresponding Gibbs energy ispresented as Eq. 3:

E(y) =∑

ψu (yi ) +i∑ψp (yi , yj ) ,

(3)

i<j

where i and j range from 1 to N. ψu (yi ) is the unary potential obtained from
TextonBoost. The pairwise edge potentials ψp (yi , yj ) are deﬁned by a linear
combination of Gaussian kernels in Eq. 4:
ψp (yi , yj ) = µ(yi , yj ) ΣKω m k m (fi , fj ),| m=1 {z}(k(fi ,fj ))()|Ii − Ij ||pi − pj ||pi − pj |22−,k(fi , fj ) = ω exp −+ω exp −
2θα2
2θβ2
2θγ2
|
{z
}
|
{z
}
2

2

1

appearance kernel(4)smoothness kernel

where vectors fi and fj are feature vectors deﬁned in terms of color vectors Ii and
Ij and position vectors pi and pj for pixel i and j, ω m are linear combination
weights, and µ is a label compatibility function. The appearance kernel is245

inspired by the observation that nearby pixels with similar color are likely to
be in the same class. The degrees of nearness and similarity are controlled bylearned parameters θα and θβ . The smoothness kernel which can remove small
isolated regions.
A mean ﬁeld approximation [40] is applied to the CRF distribution for its

250

high eﬃciency. Some of the segmentation results are shown in Fig. 3(c). Com-

11

Sky
House
Water
Signboard
Wall

Road
Grass
Book

(a)

(b)

(c)

Figure 3: Scene context feature extraction. (a) The original images. (b) Segment results after
TextonBoost. (c) Segment results after fully connected CRF.

pared to the coarse segmentation result of TextonBoost in Fig. 3(b), we cansee that some initially incorrect segmented pixels are changed to correct categories. To utilize the segmentation result yielded by the fully-connected CRF,
we need to update the context probability of the pixels whose category attri255

bution changes. We denote the original context probability of a pixel as P ={p1 , p2 , · · · , pK }, K is the total object categories. If its category attributionchanged to a diﬀerent label lu after fully-connected CRF, we updated the context probability of plu to 0.8, and set other plv ∈L,lv ̸=lu to 0.2 ×

plv ∈L,lv ̸=lu
. This1−pluprocess assigned a larger probability to the updated object labeling, meanwhile
260

keeping the sum of each category’s probabilities to 1.
3.3. Text/Non-text classiﬁcation
In this stage, we confuse the context features and character features to classify text and non-text. Our method belongs to the local-classiﬁcation method.
The MSER-based text detection methods [41, 42, 43] are state-of-the-art meth-265

ods for their outstanding eﬀectiveness and high eﬃciency. It deﬁnes an extremal
region as a CC whose pixels have intensity contrast against its boundary pixels
in the image. We applied various sets of parameters of MSER to extract as
many components as possible, and then integrated them into two maps to overcome the overlapping problem. The classiﬁcation process focuses on the two

270

maps. All the CCs on the two maps are extended to region-based patches. A
12well trained CNN classiﬁer with two layers is used to classify the patches to text
or non-text. For the character adjoining problem, the width and height ratio
is a judgement. Therefore, a coarse segmentation is implemented for adjacent
joint character regions. The total classiﬁcation scores of all patches in the cor275

responding adjacent joint character region will determine whether it is text ornot.
3.3.1. Maximally stable extremal regions (MSERs) integration
In the work[44], Matas deﬁned the extremal region as a connected component of an image whose pixels have either higher or lower intensity than its outer280

boundary pixels. The Maximally stable extremal region is an extremal region
whose size remains virtually unchanged over a range of intensity levels. Consider an extremal region Qt , and the branch of the tree rooted at T (Qt ) = {Qt ,Qt+1 , · · · , Qt+∆ }. The instability of Qt is deﬁned as ν(Qt ) = |Qt+∆ − Qt |/|Qt |.
|Q| denotes the number of pixels in region Q. Qt is a maximally stable ex-285

tremal region if its instability is more lower than its parent Qt−1 and child
Qt+1 . Extracting MSERs in an image involves ﬁve parameters that control its
performance. Their deﬁnition and meaning are listed below:(1) ∆; It compares (sizet+∆ -sizet )/sizet ,(2) maxArea; It prunes the area bigger than maxArea,

290(3) minArea; It prunes the area smaller than minArea,(4) maxVariation; It prunes the area to have a similar size to its children,(5) minDiversity; It traces back to cut oﬀ MSERs with diversity less thanset value.

Table 1: Four sets of parameters in MSER.

PARAM

∆

maxVariation

minDiversity

1

2

0.5

0.5

2

5

0.5

0.5

3

5

0.1

0.9

4

5

0.5

0.5

13

Algorithm 1 Integrate MSER map.
Input: MSER maps I = {I1 , · · · , In } generated by the n parameter sets andoriginal RGB image. I(x, y)represents the value in coordinate point (x, y).
Initialize: Transform I to 0-1 map.
Output: Integrated map M.
1. Itemp (x, y) = I1 (x, y);
2. for i: 2∼n do
3. Itemp (x, y) = Itemp (x, y) + Ii (x, y);
4. Label CCs of Itemp (x, y) = 1 (denote as C1 ) and get each CC’s height
H and width W; Label CCs of Itemp (x, y) = 2 (denote as C2 );5. for each c ∈ C2
6.
Search for the adjacent two CCs ci and cj in C1 ;7.
if Wi + Wj > min(Hi , Hj ), and Itemp (x, y) =1
8.
Itemp (x, y) = 0;
9.
else
10.
Compute the adjacent pixels’ average RGB vector V in c, ci , cj ;11.
if max(|Vc - Vci |, |Vc - Vcj |, |Vcj - Vci |)>10, and Itemp (x, y) =1
12.
Itemp (x, y) = 0;13.
end if14.
end if
15. end for
16. Remove CCs in C1 with area size less than 10;
17. if Itemp (x, y) =2
18.
Itemp (x, y) =1;
19. end
20. end for
21. M = Itemp ;Diﬀerent parameters result in diﬀerent MSER regions. For diﬀerent images,295

the same parameters applied to MSER may generate diﬃdent results. The
characters may adjoin the background or removed. It is diﬃcult to set ﬁxed or
adaptive parameters for obtaining the perfect performance on all the images.
To robustly extract MSERs in all the images, four diﬀerent combinations ofparameters are set. Among them, the minArea is ﬁxed to 10 and maxArea = 1/3

300

× image size. The four sets of the other three parameters are listed in Tab. 1. If
all the MSERs from the four sets are classiﬁed in the classiﬁcation step, a high

14

(a)

(b)

(c)

Figure 4: Integrated MSER map for the four parameter sets. (a) The examples of scene text
images. (b) Four MSER maps generated by the four parameter sets. (c) Integrated MSER
map.

recall will be achieved. However, it costs more time for computation since there
are many repeating MSERs or similar MSERs of the same text in the images. To
avoid this problem, we integrate all the MSERs. The details of the integration
305

steps are given in Algorithm 1. The same process is applied to 0-255 reversed
gray-scale images. Finally, we obtain two maps with non-overlapping CCs. The
adjoining characters of some CCs will separate to individual characters after
this process as shown in Fig. 4. This method successfully avoids the repeatingproblem as mentioned in [31]. It can also extract as many text CCs as possible

310

and also preserve them eﬃciently.
3.3.2. Patch-based character/non-character classiﬁcation
Scene text detection in natural image is a high level visual task, which isdiﬃcult to be solved completely by a set of low-level operations or manuallydesigned features. Therefore, we exploit convolutional neural networks (CNN)315

to extract character feature in our work, since a deep network is capable ofleaning meaningful high-level features and semantic representations for visual
recognition through a hierarchical architecture with multiple-layers of feature
convolutions. The deep structure of the CNN allows it to reﬁne feature representation and abstract sematic meaning gradually. Applications of applying320

CNN have achieved great success on digit and hand-written character recognition [45, 46].
A simple two-layer CNN structure connected with a SVM classiﬁer in thelast layer is adapted as shown in Fig. 5. The input of this classiﬁcation system

15

feature map
Character feature
96@5×564@2×2（2×2×64-dim）Figure 5: The classiﬁcation system.

is the region-based patch of CCs on the integrated MSER map and an aver325

aged context feature in this region. The output is a vector of two dimensions
representing the score of text and non-text.
Since the input of the CNN is region-based patches, we need to transfer theCCs of the integrated MSER map to patches. For the CCs obtained in theformer step, connections may exist, as shown in Fig. 6. The height and width

330

ratio R is used to discriminate whether the CCs are connected or not. If R <TR (in our work, we set the threshold TR to 1.8), the CCs are non-connected.
Otherwise, they are connected CCs. For a non-connected CC region, we directly
turn it to a patch which is the region of bounding box. However, to extract more
context information of it, we extend the patch region to larger size by adding 1/3

335

× CC region’s height to the boundaries. For connected CCs, many characters
are adjoined to words or strings. It is diﬃcult to estimate the size of eachcharacter. So, accurate segmentation meets great challenge. In our method, wecoarsely segment the characters based on the CC region’s height and extract
many overlapped regions with two strides: 0.6 × CC’s height and 0.7 × CC’s

340

height. The same extension is processed as the individual CCs. The average

16Figure 6: Patch extension for individual CCs and connected CCs

text score of all patches in the connected region is considered as its score.
In this classiﬁcation system, the input patches of the CNN are normalized
to 32 × 32. The settings of the CNN architecture used in our work is displayedin Tab. 2. 96 and 64 nodes are designed in the ﬁrst and second layers. We
345

use average pooling and rectiﬁed activation. The stride sizes of convolution andpooling are both set to 1. Therefore, we can get 5 × 5 and 2 × 2 size featuremaps in the ﬁrst and second layers of the CNN respectively. In the ﬁrst layerof the CNN , we train the ﬁlters in an unsupervised learning algorithm [8]. Ina given set of 32 × 32 training images, we randomly select 8 patches with size350

8 × 8. Then, perform contrast normalized and whitened zero-phase component
analysis (ZCA) [47] of all these patches to form 64-dimension input vectors ofeach patch. A K-means clustering method is used to learn a set of low-level ﬁlters
D∈R64×n . Here we set n = 96 which is equal to the ﬁrst layer’s node number.
For a normalized and whitened 8×8 patch x, we compute its ﬁrst layer responses

355

by performing inner product with the ﬁlter bank followed by a scalar activation
function with response = max{0, |DT x|-α}, here α = 0.5 is a hyperparameter. In
the second layer, 14-dimensional averaged context feature vector is confused withthe projected 2 × 2 × 64-dimensional character feature vector and cascaded toan SVM classiﬁer. The network is discriminatively trained by backpropagating
17

Table 2: The parameters used in CNN architecture.

LayersLayer-1
Layer-2

360

Kernel size
82

Pooling size
5
2

Nodes
96
64

the L2 -SVM classiﬁcation error. Only the parameters in second layer of CNN
are updated in the ﬁne-tuning.
After classiﬁcation, the CCs are labeled with probability of text. We ﬁlter
out some of the non-text CCs by setting a threshold T = 0 to the output scoreof the CNN.

365

3.4. Hierarchical grouping process
After classiﬁcation, the CCs classiﬁed to text are deﬁned as text components(TCs). However, there still are false negatives and false positives. To tackle thisproblem, we use a hierarchical grouping method to retrieve the false negatives
and remove the false positives.

370

First, we deﬁne four types of the candidate TCs: left TC, right TC, middle
TC and isolated TC. A right or a left TC has no adjacent right or left TC respectively, a middle TC has both adjacent left and right TCs, and an isolated TC
has no adjacent TCs. Since the similar geometric properties, color appearance
and context information are shared by the text CCs in the same word or text

375

line, they can be used to measure the adjacent relationship and discriminate theCC’s type. For a certain candidate text CC, we search its left and right regions
from near to far with the distance of two times of its height to ﬁnd candidate
text CCs. If they satisfy condition (1) or condition (2) listed below, they are
adjacent TCs.

380

Condition (1): |Ci - Cj | < Tc1 ; min(Ti - Tj , Bi - Bj ) < min(Hi , Hj ).
Condition (2): |Ci - Cj | < Tc2 ; |SCi - SCj | <1; Ai /Aj > Ts and Ai /Aj < 1/Ts ;(Ti - Tj ∥ Bi - Bj ) < min(Hi , Hj )/2.
Here, T and B represent the top coordinate and bottom coordinate respectively. H is the height of TC. A is the area size, C is the averaged RGB color

18

(a)

(b)

(c)
Figure 7: Example of some grouping results. (a) shows the classiﬁcation results before grouping. (b) shows the grouping results by connecting adjacent characters’ centers with red lines.
(c) The ﬁnal text detecting results.

385

vector, and SC is the averaged context feature. In all of our framework, we set
Tc1 =20, Tc2 =40 and Ts =0.3.
For a middle TC, if its right and left adjacent TCs’ inter-character distancediﬀerence is lager than one third of the three TCs’ middle height value, there
might be missed characters or string breaks. We search the CCs on the inte-

390

grated MSER map in this region and discriminate whether to retrieve the CCs
by the following rules.
(1) If the CC satisﬁes HX > max(Hi , Hj )/3, HX < max(Hi , Hj )× 1.5, XT> min(Ti , Tj ) and XB < max(Bi , Bj ), retrieve it, otherwise, turn to (2).
(2) Count CX of this CC limited in the region max(Ti , Tj ), min(Bi , Bj ),395

XL and XR . If |CX - Ci |< Tc1 or |CX - Cj |< Tc1 , retrieve this CC.
Here, i and j are neighbored TCs of X. X is the CC between i and j which
locates on the integrated MSER map. XT , XB , XL and XR represent the top,bottom, left and right boundary coordinate of X.

19

Sky
Signboard
Text streamer
Tree
Wall

Other objects

Figure 8: Pixel level labeling of wall and text steamer images.

After that, we connect all the adjacent TCs to obtain the detection result.
400

For isolated CCs, we compute their parallel edge pair ratio [48]. If the edge pairratio is larger than 0.5, they are considered as text. Otherwise, we classify themas background and remove them. Some grouping results and ﬁnal detection
results are shown in Fig. 7.

4. Experiments and results
405

In this section, we introduce the two collected databases for scene segmentation and scene text detection. The evaluation and analysis on scene text
detection are presented.
4.1. Data collocation
4.1.1. Database for scene segmentation

410

In our method, 14 scene categories are used for segmentation and labeling.
The images of 12 scene categories are extracted from the MSRC-23 database[13]. Each categories contains 30 images with the corresponding ground truthlabels. Besides that, the two additional object images (wall and text streamer)are downloaded from Google Images or selected from the ICDAR 2011 database415[49]. Each ground truth image is fully annotated at pixel-level, with carefullabeling around complex boundaries as shown in Fig. 8. Each object categorycontains 30 images for both training and testing.

20

4.1.2. Database for scene text detection
To train the parameters of CNN and SVM, we collected 440 images from420

the ICDAR 2011 and ICDAR 2013 text location training databases. We manually labeled the character regions with the bounding boxes in each image. By
extending them to larger size based on the process in Section 3.3.2, we obtained
8317 text patches. For background patches, we performed an MSER algorithm
to the databases and removed the CCs belonging to text region. The non-text

425

CCs were extended to patches and cropped the same way as text patches. In
total, we got 15440 background patches. The process of collection was shownin Fig. 9. They compose the patch-based database and were used to evaluate
four variable controlled classiﬁcation methods. Meanwhile, we store the context
features of each patch region in a matrix. All of the patches with the correspond-

430

ing context feature matrices are randomly divided to two sets: 15000 training
patches and 8757 test patches. The training patches contain 6000 text patches
and 9000 background patches. The rest of the parts belong to the test patches.
To compare our proposed method with other scene text detection methods, thepublic ICDAR 2013 database and SVT database [21] are used for evaluation.

435

4.2. Scene context feature extraction performance with fully-connected CRF
To evaluate the performance of TextonBoost and the fully-connected CRF,
we randomly selected 6 images in each object category from the scene segmentation database for test. The others were split into training images. The unarypotentials and parameters of the CRF model were learned on the training set

440

and the training time cost was about 5 hours on an Intel i7 processor clocked
at 12.0GHz. It yielded the global score of 82.5% and average score of 74.3%,
where the global score denotes the overall percentage of correctly classiﬁed image
pixels and the average score is the unweighted average of per-category classiﬁcation accuracy [13, 50]. The trained parameters are used to extract the context

445

features in the scene text detection databases. The examples of segmentation
results in scene segmentation test database and scene text detection databases
are given in Fig. 10(a) and Fig. 10(b). However, it may fail for some scene im21

Training images

Manually labeled

Training set of text patc hes(a)Training images

By MSER

Training set of non-text patches(b)Figure 9: Patch extraction. (a) Extracted text patches from manually labeled text regions.
(b) Extracted background patches from non-text regions generated by MSER.

ages as shown in Fig. 10(c). Some text regions can not be segmented and some
objects are incorrect classiﬁed to other objects. For example, in the ﬁrst and450

third images of Fig. 10(c), the signboard on the wall and the text streamer on
the airplane are not correctly segmented. The background in the second image
are classiﬁed to a book and the airplane in the forth image is classiﬁed to acar. Since the training images in scene segmentation databases are limited, it is
not so robust for segmenting all types of object categories. Because some back-455

grounds are very complicated, it is diﬃcult to deﬁne them to certain categories.
In the scene text detection database, there are some objects undeﬁned, such asa microwave, sculpture, mouse cursor, etc., these objects will be classiﬁed toother objects in the 14 object categories.

22

Sky

Tree

Air plane

Wall

Text streamer

Grass

Road

House

Car

Signboard

Water

Book

ClotheHuman face

(a)

(b)

(c)
Figure 10: Scene segmentation performance of some example images. (a) Results in scene
segmentation database. (b) Results from scene text detection database. (c) Some incorrect
segmented images.

23

4.3. Eﬀect of context feature for text detection
460

The segmentation results are used to update the context features, namely
the probabilities of the pixels. Then, they are averaged as the input context
feature of the region to be classiﬁed. Three evaluations are performed in our experiment: evaluation on patch-based classiﬁcation, evaluation on public ICDAR2013 database and SVT database.

465

4.3.1. Results of patch-based classiﬁcation by CNN
Four variable controlled experiments are implemented on patch-based database.
We conclude them to four schemes as below:(1) Train CNN+SVM without context features;(2) Train CNN+SVM with context features;

470(3) Train SVM with pre-trained CNN features and context features;(4) Train SVM with pre-trained CNN features and averaged equal fake context features (All are 1/14).
The input of Scheme (1) is only patch-level images. Scheme (2) is the frameof our work. In training process, the The pre-trained CNN features in Scheme

475(3) and (4) means that the parameters in the CNN are not trained with the
SVM but trained in advance with the patch-level training images. Then, after
training, we combine the output of the CNN features and corresponding context
features to train the SVM classiﬁer. To ﬁgure out whether the context features
and the CNN’s structure aﬀect the classiﬁcation result, we add fake context

480

features with all the values as 1/14 in Scheme (4).
Since the patches in the database belong to text or background, it is a twoclass classiﬁcation problem for the patch-based images. We deﬁne an accuracy
that equals to the correctly classiﬁed patch number SU M (Pcorrect−classif ied ) divides by all of the patch number SU M (P ), namely accuracy =485

SU M (Pcorrect−classif ied ),SU M (P )to measure the classiﬁcation performance. Then, we give the results of eachschemes in Tab. 3. The training accuracy and test accuracy is the accuracy forthe training patches and test patches respectively. In Fig. 11, the misclassiﬁedpatches generated by our proposed method and non-context feature method in
24

Figure 11: Misclassiﬁed text patches and background patches.

Table 3: The patch-based classiﬁcation accuracy on four schemes.

Scheme number

1

2

3

4

Scene context

No

Yes

Yes

Equal-fakeTrain CNN with scene context

no

no

yes

yes

Train accuracy

99.99%

99.77%

99.99%

99.99%

Test accuracy

97.84%

98.44%

97.88%

97.77%

Scheme (1) are given. Less background patches are misclassiﬁed in our method,
490

thus, with the context information, it can remove more manually labeled background regions, which can increase the precision in text detection task. We cansee that, our proposed method, namely Scheme (2), achieves the best performance on test set and the pre-trained CNN features combining context feature
method ranks second. They all performed better than the method without con-495

text features. The patch-based experimental result proves that context features
can be of beneﬁt for scene text detection.

25

(a)

(b)

(c)

Figure 12: The output results after text and non-text classiﬁcation by non-context and withcontext methods. The diﬀerent gray-scale value represents the text veriﬁcation degree. (a)The original images. (b) Classiﬁcation results by with-context method. (c) Classiﬁcation
results by non-context method.

4.3.2. Comparing the without-context method to our framework on scene text
detection
After the patch-based test, we evaluate the Scheme (1) and (2) on the IC500

DAR 2013 test database and SVT test database. Scheme (1) is the method
without using context features and Scheme (2) is the proposed method andachieves the best performance on the patch-based database. The experiments
were implemented on the gray-scale channel and multi-channels that contain aand b channels of Lab color space. We follow the same evaluation scheme of505

the ICDAR-2013 competition which makes use of the framework proposed by
Wolf and Jolion [51], and classify the matching to one-to-one, one-to-many andmany-to-one matches.
Tab. 4 displays the comparison results on ICDAR 2013 database. Fromthe comparison, we found that the recall of the non-context method is lower

510

than the with-context method, but the precision is higher when evaluating on
gray-scale images. Since the recall measures how many true text regions are26

Table 4: Comparison Results of Scheme (1)and Scheme (2) on ICDAR 2013 Database.

Method name

Precision

RecallF-scoreScheme 1 (gray)88.35%

64.38%

74.49%

Scheme 2 (gray)88.26%

66.12%

75.60%

Scheme 1 (gray+a+b)

85.76%

71.05%

77.72%

Scheme 2 (gray+a+b)

85.80%

74.34%

79.66%

Table 5: Comparison Results of Scheme (1)and Scheme (2) on SVT Database.

Method name

Precision

RecallF-scoreScheme 1 (gray)38.21%

29.21%

33.11%

Scheme 2 (gray)42.33%

28.60%

34.14%

Scheme 1 (gray+a+b)

36.19%

35.09%

35.63%

Scheme 2 (gray+a+b)

40.55%

34.27%

37.15%

detected, the result illustrates that the with-context method can detect more
text regions. The precision measures that the detected text region divide by all
detected regions, so we can infer that there are more false detected regions in515

the with-context method. It may result from the scene segment results. Someundeﬁned background regions are segmented to text streamers, cars or books onwhich the text appear with high probability. This might be a reason of the low
precision. The segmentation results of the text regions are almost correct, so it
results in high recall when adding context features. The comparison results on

520

SVT database is shown in Tab. 5. Images in SVT database contains a lot ofbuildings, street and billboard. Scenes classed are simpler than that in ICDAR
database. But text of SVT database are mostly in low resolution, that results inincorrect segmented scene. Thus, with the context, our method could exclude
more background and get higher precision. Meanwhile, some text are classiﬁed

525

to background with the incorrect scene context that decreases the recall. In

27

Fig. 12, a comparison result of the two methods is displayed. The intensityof the detected CCs represent the text score of the classiﬁer’s output. In theimages of ﬁrst and second lines, when the context are text streamers, the realtext components can be detected. And the CCs in the wall of the third image
530

are classiﬁed to non-text.
4.3.3. Comparison with state-of-the-art methods
The performances of the proposed algorithm as well as other methods on
the ICDAR 2013 database and SVT database are shown in Tab. 6 and Tab. 7
respectively. We use the combined channels, gray-scale, a and b channels forcomparison. The average running time of our method on MATLAB platform isTable 6: Comparison of diﬀerent methods on ICDAR 2013 database.

Method name

Precision

RecallF-scoreXu’s Method [52]

85.0%

78.0%

81.0%

Our method85.80%

74.34%

79.66%

Liu’s Method [53]85.39%

68.72%

76.15%USTB TexStar

88.47%

66.45%

75.89%TextSpotter

87.51%

64.84%

74.49%CASIA NLPR

78.89%

68.24%

73.18%

Baseline

60.76%

34.74%

44.21%

535

about 263 seconds by using a standard PC with 12 GHz Intel processor. Since
we do not use a GPU to accelerate the CNN, the time cost is high. Otherwise,
it could speed up by more than 20 times. SVT database should be used for
cropped lexicon-driven word recognition or full image lexicon-driven word de540

tection and recognition [21]. In our method, we do not use any lexicon system
for text detection. The low resolution of images results in large scale of text candidates missing. Consequently, it causes low recall on SVT database byusing our method. However, comparing to other state-of-the-art text detection
algorithms, our method is comparable. This proves the achievement by applying
28

Table 7: Comparison of diﬀerent methods on SVT database.

545

Method name

Precision

RecallF-scoreSnooperText [54]

36.0%

54.0%

43.0%

Our method

40.55%

34.27%

37.15%

Neumann[28]

19.0%

33.0%

26.0%

TESSFRONE+HOG

15.0%

15.0%

1.0%

TESSFRONT

4.0%

18.0%

6.0%

context information to scene text detection task. In other words, if we combine
those methods with scene context, we can expect to enhance the performancefurther.

5. Conclusion
In this paper, we had the hypothesis that scene context could be of beneﬁt
550

for scene text detection. Meanwhile, we proposed a novel approach to prove it.
It combined character feature generated by CNN with context feature extraction by TextonBoost and fully-connected CRF. The experiment results on thepatch-based database, the ICDAR 2013 database and SVT database proved ourhypothesis and also illustrated that the context based approach could perform

555

better than some state-of-the-art methods in scene text detection.
Our work contained two ﬁelds of computer vision: scene understanding andscene text detection. We tried to build the connection between them to ﬁnd the
beneﬁts. The results are encouraging. However, there are still two problems
to be solved: more accurate scene segmentation and non-connected character

560

detection. They correspond to improve precision and recall in text detection
task respectively. Therefore, we will focus on the two aspects in our future work
to ﬁnd more beneﬁts of them.

29References[1] L.-J. Li, R. Socher, L. Fei-Fei, Towards total scene understanding: Clas565

siﬁcation, annotation and segmentation in an automatic framework, in:Proceedings of the CVPR, 2009, pp. 2036–2043.
[2] A. Geiger, M. Lauer, C. Wojek, C. Stiller, R. Urtasun, 3D traﬃc scene
understanding from movable platforms, IEEE Trans. PAMI 36 (5) (2014)
1012–1025.

570[3] Q. Ye, D. Doermann, Text detection and recognition in imagery: A survey,
IEEE Trans. PAMI 37 (7) (2014) 1480–1500.
[4] H. Zhang, K. Zhao, Y.-Z. Song, J. Guo, Text extraction from natural scene
image: A survey, Neurocomputing 122 (2013) 310–323.
[5] T. Scharwächter, M. Enzweiler, U. Franke, S. Roth, Eﬃcient multi-cue

575

scene segmentation, Pattern Recognit. 8142 (2013) 435–445.
[6] C. Liu, D. Wang, J. Zhu, B. Zhang, Learning a contextual multi-thread
model for movie/tv scene segmentation, IEEE Trans. Multimedia 15 (4)(2013) 884–897.
[7] P. Krähenbühl, V. Koltun, Eﬃcient inference in fully connected crfs with

580

gaussian edge potentials, arXiv preprint arXiv:1210.5644.
[8] T. Wang, D. J. Wu, A. Coates, A. Y. Ng, End-to-end text recognition
with convolutional neural networks, in: Proceedings of the ICPR, 2012,
pp. 3304–3308.
[9] P. F. Felzenszwalb, D. P. Huttenlocher, Eﬃcient graph-based image seg-585

mentation, Int. J. Comput. Vis. 59 (2) (2004) 167–181.
[10] C. Pantofaru, C. Schmid, M. Hebert, Object recognition by integrating
multiple image segmentations, in: Proceedings of the ECCV, 2008, pp.
481–494.

30

[11] D. Hoiem, A. N. Stein, A. Efros, M. Hebert, et al., Recovering occlusion
590

boundaries from a single image, in: Proceedings of the CVPR, 2007, pp.
1–8.
[12] C. Cheng, A. Koschan, C.-H. Chen, D. L. Page, M. Abidi, et al., Outdoor
scene image segmentation based on background recognition and perceptualorganization, IEEE Trans. Image Process. 21 (3) (2012) 1007–1019.

595[13] J. Shotton, J. Winn, C. Rother, A. Criminisi, Textonboost for image understanding: Multi-class object recognition and segmentation by jointly
modeling texture, layout, and context, Int. J. Comput. Vis. 81 (1) (2009)
2–23.
[14] S. Gould, J. Rodgers, D. Cohen, G. Elidan, D. Koller, Multi-class seg-

600

mentation with relative location prior, Int. J. Comput. Vis. 80 (3) (2008)
300–316.
[15] A. Levin, Y. Weiss, Learning to combine bottom-up and top-down segmentation, Int. J. Comput. Vis. 81 (1) (2009) 105–118.
[16] N. Dalal, B. Triggs, Histograms of oriented gradients for human detection,605

in: Proceedings of the CVPR, 2005, pp. 886–893.
[17] D. Huang, C. Shan, M. Ardabilian, Y. Wang, L. Chen, Local binary patterns and its application to facial image analysis: a survey, IEEE Trans.
Syst. Man Cybern. 41 (6) (2011) 765–781.
[18] J.-J. Lee, P.-H. Lee, S.-W. Lee, A. Yuille, C. Koch, Adaboost for text

610

detection in natural scene, in: Proceedings of the ICDAR, 2011, pp. 429–
434.
[19] K. I. Kim, K. Jung, J. H. Kim, Texture-based approach for text detection
in images using support vector machines and continuously adaptive mean
shift algorithm, IEEE Trans. PAMI 25 (12) (2003) 1631–1639.

31

615

[20] X. Chen, A. L. Yuille, Detecting and reading text in natural scenes, in:Proceedings of the CVPR, 2004, pp. 366–373.
[21] K. Wang, B. Babenko, S. Belongie, End-to-end scene text recognition, in:Proceedings of the ICCV, 2011, pp. 1457–1464.
[22] L. Neumann, J. Matas, Scene text localization and recognition with orient-620

ed stroke detection, in: Proceedings of the ICCV, 2013, pp. 97–104.
[23] A. Mishra, K. Alahari, C. Jawahar, Top-down and bottom-up cues for scene
text recognition, in: Proceedings of the CVPR, 2012, pp. 2687–2694.
[24] B. Epshtein, E. Ofek, Y. Wexler, Detecting text in natural scenes withstroke width transform, in: Proceedings of the CVPR, 2010, pp. 2963–

625

2970.
[25] A. Mosleh, N. Bouguila, A. B. Hamza, Image text detection using a bandletbased edge detector and stroke width transform., in: Proceedings of the
BMVC, 2012, pp. 1–12.
[26] H. Chen, S. S. Tsai, G. Schroth, D. M. Chen, R. Grzeszczuk, B. Girod,

630

Robust text detection in natural images with edge-enhanced maximallystable extremal regions, in: Proceedings of the ICIP, 2011, pp. 2609–2612.
[27] L. Neumann, J. Matas, A method for text localization and recognition in
real-world images, in: Proceedings of the ACCV, 2011, pp. 770–783.
[28] L. Neumann, J. Matas, Real-time scene text localization and recognition,635

in: Proceedings of the CVPR, 2012, pp. 3538–3545.
[29] X.-C. Yin, X. Yin, K. Huang, H.-W. Hao, Robust text detection in naturalscene images, IEEE Trans. PAMI 36 (5) (2014) 970–983.
[30] X.-C. Yin, W.-Y. Pei, J. Zhang, H.-W. Hao, Multi-orientation scene text
detection with adaptive clustering, IEEE Trans. PAMI 37 (9) (2015) 1930–

640

1937.

32

[31] L. Sun, Q. Huo, W. Jia, K. Chen, A robust approach for text detection
from natural scene images, Pattern Recognit. 48 (9) (2015) 2906–2920.
[32] Y.-F. Pan, X. Hou, C.-L. Liu, A hybrid approach to detect and localize
texts in natural scene images, IEEE Trans. Image Process. 20 (3) (2011)645

800–813.
[33] W. Huang, Y. Qiao, X. Tang, Robust scene text detection with convolution
neural network induced mser trees, in: Proceedings of the ECCV, 2014, pp.
497–511.
[34] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectiﬁers: Surpass-650

ing human-level performance on imagenet classiﬁcation, <arXiv preprint
arXiv:1502.01852>.
[35] A. Bissacco, M. Cummins, Y. Netzer, H. Neven, Photoocr: Reading text inuncontrolled conditions, in: Proceedings of the ICCV, 2013, pp. 785–792.
[36] M. Jaderberg, K. Simonyan, A. Vedaldi, A. Zisserman, Reading text in the655

wild with convolutional neural networks, Int. J. Comput. Vis. (2014) 1–20.
[37] L. Ladicky, C. Russell, P. Kohli, P. H. Torr, Associative hierarchical randomﬁelds, IEEE Trans. PAMI 36 (6) (2014) 1056–1077.
[38] B. Fulkerson, A. Vedaldi, S. Soatto, Class segmentation and object localization with superpixel neighborhoods, in: Proceedings of the ICCV, 2009,

660

pp. 670–677.
[39] B. Triggs, J. J. Verbeek, Scene segmentation with crfs learned from partiallylabeled images, in: Proceedings of the NIPS, 2007, pp. 1553–1560.
[40] T. Hotta, Mean-ﬁeld approximation, in: Nanoscale Phase Separation andColossal Magnetoresistance, 2003, pp. 157–167.

665[41] X. Yin, X.-C. Yin, H.-W. Hao, K. Iqbal, Eﬀective text localization in natural scene images with mser, geometry-based grouping and adaboost, in:Proceedings of the ICPR, 2012, pp. 725–728.
33

[42] H. I. Koo, D. H. Kim, Scene text detection via connected component clustering and nontext ﬁltering, IEEE Trans. Image Process. 22 (6) (2013)
670

2296–2305.
[43] C. Shi, C. Wang, B. Xiao, Y. Zhang, S. Gao, Scene text detection usinggraph model built upon maximally stable extremal regions, Pattern Recognit. Lett. 34 (2) (2013) 107–116.
[44] J. Matas, O. Chum, M. Urban, T. Pajdla, Robust wide-baseline stereo from675

maximally stable extremal regions, Image Vis. Comput. 22 (10) (2004) 761–
767.
[45] K. Chellapilla, S. Puri, P. Simard, High performance convolutional neuralnetworks for document processing, in: Proceedings of the International
Workshop on FHR, 2006.

680[46] P. Sermanet, S. Chintala, Y. LeCun, Convolutional neural networks applied
to house numbers digit classiﬁcation, in: Proceedings of the ICPR, 2012,
pp. 3288–3291.
[47] A. Hyvärinen, E. Oja, Independent component analysis: algorithms andapplications, Neural Networks 13 (4) (2000) 411–430.

685[48] A. Zhu, G. Wang, Y. Dong, Detecting natural scenes text via auto image
partition, two-stage grouping and two-layer classiﬁcation, Pattern Recognit. Lett. 67 (2015) 153–162.
[49] A. Shahab, F. Shafait, A. Dengel, Icdar 2011 robust reading competition
challenge 2: Reading text in scene images, in: Proceedings of the ICDAR,

690

2011, pp. 1491–1496.
[50] P. Kohli, P. H. Torr, et al., Robust higher order potentials for enforcing
label consistency, Int. J. Comput. Vis. 82 (3) (2009) 302–324.
[51] C. Wolf, J.-M. Jolion, Object count/area graphs for the evaluation of object
detection and segmentation algorithms, Int. J. Doc. Anal. Recognit. 8 (4)695(2006) 280–296.
34

[52] H. Xu, F. Su, Robust seed localization and growing with deep convolutionalfeatures for scene text detection, in: Proceedings of the ACM on ICMR,
2015, pp. 387–394.
[53] J. Liu, H. Su, Y. Yi, W. Hu, Robust text detection via multi-degree of700

sharpening and blurring, Signal Processing.
[54] R. Minetto, N. Thome, M. Cord, N. J. Leite, J. Stolﬁ, Snoopertext: A text
detection system for automatic indexing of urban scenes, Comput. Vis. andImage Understanding 122 (2014) 92–104.

35

Anna Zhu received the B.S. degree from Huazhong University of Science andTechnology, of China in 2011. She is currently pursuing Ph.D in School of Automation, Huazhong University of Science and Technology. She visited Human
Interface Laboratory, Kyushu University of Japan in 2015. Her current researchinterests include text detection, image processing, and machine learning.
Renwu Gao received his M.S. degree from Nanjing university of science andtechniques in 2012, and PhD degree from Kyushu university in Japan in 2015.
His current interest researches include text detection, image processing andpattern recognition.
Seiichi Uchida received B.E. and M.E. and Dr. Eng. degrees from Kyushu
University in 1990, 1992 and 1999, respectively. From 1992 to 1996, he joined
SECOM Co., Ltd., Japan. Currently, he is a professor at Kyushu University.
His research interests include pattern recognition and image processing. Dr.
Uchida is a member of IEEE and IPSJ.

1