Reading Scene Text in Deep Convolutional Sequences

We develop a Deep-Text Recurrent Network (DTRN) that regards scene text reading as a sequence labelling problem. We leverage recent advances of deep convolutional neural networks to generate an ordered highlevel sequence from a whole word image, avoiding the difficult character segmentation problem. Then a deep recurrent model, building on long short-term memory (LSTM), is developed to robustly recognize the generated CNN sequences, departing from most existing approaches recognising each character independently. Our model has a number of appealing properties in comparison to existing scene text recognition methods: (i) It can recognise highly ambiguous words by leveraging meaningful context information, allowing it to work reliably without either preor post-processing; (ii) the deep CNN feature is robust to various image distortions; (iii) it retains the explicit order information in word image, which is essential to discriminate word strings; (iv) the model does not depend on pre-defined dictionary, and it can process unknown words and arbitrary strings. It achieves impressive results on several benchmarks, advancing the-state-of-the-art substantially. Text recognition in natural image has received increasing attention in computer vision and machine intelligence, due to its numerous practical applications. This problem includes two sub tasks, namely text detection (Huang, Qiao, and Tang 2014; Yin et al. 2014; He et al. 2015; Zhang et al. 2015; Huang et al. 2013; Neumann and Matas 2013) and text-line/word recognition (Jaderberg, Vedaldi, and Zisserman 2014; Almazán et al. 2014; Jaderberg et al. 2014; Bissacco et al. 2013; Yao et al. 2014). This work focuses on the latter that aims to retrieve a text string from a cropped word image. Though huge efforts have been devoted to this task, reading text in unconstrained environment is still extremely challenging, and remains an open problem, as substantiated in recent literature (Jaderberg et al. 2015b; Almazán et al. 2014). The main difficulty arises from the large diversity of text patterns (e.g. low resolution, low contrast, and blurring), and highly complicated background clutters. Consequently, individual character segmentation or separation is extremely challenging. Most previous studies focus on developing powerful char∗Authors contributed equally Figure 1: The word image recognition pipeline of the proposed Deep-Text Recurrent Networks (DTRN) model. acter classifiers, some of which are incorporated with a language model, leading to the state-of-the-art performance (Jaderberg, Vedaldi, and Zisserman 2014; Bissacco et al. 2013; Yao et al. 2014; Lee et al. 2014). These approaches mainly follow the pipeline of conventional OCR techniques by first involving a character-level segmentation, then followed by an isolated character classifier and post-processing for recognition. They also adopt deep neural networks for representation learning, but the recognition is still confined to character-level classification. Thus their performance are severely harmed by the difficulty of character segmentation or separation. Importantly, recognizing each character independently discards meaningful context information of the words, significantly reducing its reliability and robustness. First, we wish to address the issue of context information learning. The main inspiration for approaching this issue comes from the recent success of recurrent neural networks (RNN) for handwriting recognition (Graves, Liwicki, and Fernandez 2009; Graves and Schmidhuber 2008), speech recognition (Graves and Jaitly 2014), and language translaar X iv :1 50 6. 04 39 5v 2 [ cs .C V ] 2 0 D ec 2 01 5 tion (Sutskever, Vinyals, and Le 2014). We found the strong capability of RNN in learning continuous sequential features particularly well-suited for text recognition task to retain the meaningful interdependencies of the continuous text sequence. We note that RNNs have been formulated for recognizing handwritten or documented images (Graves, Liwicki, and Fernandez 2009; Graves and Schmidhuber 2008; Breuel et al. 2013), nevertheless, the background in these tasks is relatively plain, and the raw image feature can be directly input to RNN for recognition, or the text stroke information can be easily extracted or binarized at pixel level, making it possible to manually design a sequential heuristic feature for the input to RNN. In contrast, the scene text image is much more complicated where pixel-level segmentation is extremely difficult, especially for highly ambiguous images (Fig. 1). Thus it is non-trivial to directly apply the sequence labelling models to scene text. Consequently, the second challenge we need to resolve is the issue of character segmentation. We argue that individual character segmentation is not a ‘must’ in text recognition. The key is to acquire strong representation from the image, with explicit order information. The strong representation ensures robustness to various distortions and background clutters, whilst the explicit order information is crucial to discriminate a meaningful word. The ordered strong feature sequence computed from the sequential regions of word image allows each frame region to locate the part of a character, which can be stored sequentially by the recurrent model. This makes it possible to recognize the character robustly by using its continuous parts, and thus successfully avoid the character segmentation. To this end, we develop a deep recurrent model that reads word images in deep convolutional sequences. The new model is referred as Deep-Text Recurrent Network (DTRN), of which the pipeline is shown in Fig. 1. It takes both the advantages of the deep CNN for image representation learning and the RNN model for sequence labelling, with the following appealing properties: 1) Strong and high-level representation without character segmentation – The DTRN generates a convolutional image sequence, which is explicitly ordered by scanning a sliding window through a word image. The CNN sequence captures meaningful high-level representation that is robust to various image distortions. It differs significantly from manually-designed sequential features used by most prior studies based on sequence labelling (Breuel et al. 2013; Graves, Liwicki, and Fernandez 2009; Su and Lu 2014). The sequence is generated without any low-level operation or challenging character segmentation. 2) Exploiting context information In contrast to existing systems (Bissacco et al. 2013; Jaderberg, Vedaldi, and Zisserman 2014; Wang et al. 2012) that read each character independently, we formulate this task as a sequence labelling problem. Specifically, we build our system on the LSTM, so as to capture the interdependencies inherent in the deep sequences. Such consideration allows our system to recognize highly ambiguous words, and work reliably without either preor post-processing. In addition, the recurrence allows it to process sequences of various lengths, going beyond traditional neural networks of fixed-length input and output. 3) Process unknown words and arbitrary strings With properly learned deep CNNs and RNNs, our model does not depend on any pre-defined dictionary, unlike exiting studies (Jaderberg et al. 2015b; Jaderberg, Vedaldi, and Zisserman 2014; Wang et al. 2012), and it can process unknown words, and arbitrary strings, including multiple words. We note that CNN and RNN have been independently exploited in the domain of text recognition. Our main contribution in this study is to develop a unified deep recurrent system that leverages both the advantages of CNN and RNN for the difficult scene text recognition problem, which has been solved based on analyzing character independently. This is the first attempt to show the effectiveness of exploiting convolutional sequence with sequence labeling model for this challenging task. We highlight the considerations required to make this system reliable and discuss the unique advantages offered by it. The proposed DTRN demonstrate promising results on a number of benchmarks, improving recent results of (Jaderberg, Vedaldi, and Zisserman 2014; Almazán et al. 2014) considerably.