Recognition of cursive video text using a deep learning framework

IET Image ProcessingVolume 14, Issue 14 p. 3444-3455 Research ArticleFree Access Recognition of cursive video text using a deep learning framework Ali Mirza, Corresponding Author Ali Mirza alimirza@bahria.edu.pk Department of Computer Science, Bahria University, Islamabad, PakistanSearch for more papers by this authorImran Siddiqi, Imran Siddiqi Department of Computer Science, Bahria University, Islamabad, PakistanSearch for more papers by this author Ali Mirza, Corresponding Author Ali Mirza alimirza@bahria.edu.pk Department of Computer Science, Bahria University, Islamabad, PakistanSearch for more papers by this authorImran Siddiqi, Imran Siddiqi Department of Computer Science, Bahria University, Islamabad, PakistanSearch for more papers by this author First published: 14 October 2020 https://doi.org/10.1049/iet-ipr.2019.1070Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract This study focuses on cursive text recognition appearing in videos, using a complete framework of deep neural networks. While mature video optical character recognition systems (V-OCRs) are available for text in non-cursive scripts, recognition of cursive scripts is marked by many challenges. These include complex and overlapping ligatures, context-dependent shape variations and presence of a large number of dots and diacritics. The authors present an analytical technique for recognition of cursive caption text that relies on a combination of convolutional and recurrent neural networks trained in an end-to-end framework. Text lines extracted from video frames are preprocessed to segment the background and are fed to a convolutional neural network for feature extraction. The extracted feature sequences are fed to different variants of bi-directional recurrent neural networks along with the ground truth transcription to learn sequence-to-sequence mapping. Finally, a connectionist temporal classification layer is employed to produce the final transcription. Experiments on a data set of more than 40,000 text lines from 11,192 video frames of various News channel videos reported an overall character recognition rate of 97.63%. The proposed work employs Urdu text as a case study but the findings can be generalised to other cursive scripts as well. 1 Introduction With the tremendous increase in the amount of digital video data, the traditional tag-based video search engines need to be replaced with intelligent content-based retrieval systems. The conventional video retrieval systems rely on matching the queried words with user-assigned annotations and, ignore the rich information in videos that can be exploited for effective indexing and subsequent retrieval. The content-based search systems may exploit the visual information (objects and buildings), audio (spoken words), textual content (news tickers, names, subtitles etc.) or a combination of these to support smart retrieval. Examples of typical queries to such intelligent systems include retrieving all videos where a particular individual has appeared or all instances where a particular keyword (for example Breaking News) has been flashed. Among various search modalities, the focus of our present study lies on the textual content appearing in videos. Textual content in videos can be categorised into two different classes, scene text and caption text (also known as artificial/graphics text). Scene text is captured through the camera during video recording and may not always be co-related with the content. Examples of scene text include signboards, advertisement banners, building names, text on T-shirts and play cards etc. It may also include handwritten text. Scene text is useful for applications like robot navigation and systems to assist the visually impaired. Caption text or artificial text, on the other hand, is superimposed on videos and, in most cases, is more related to the content. Typical examples of the artificial text include news tickers, anchors' names, scorecards and movie credits etc. The correlation of caption text with the actual content makes it more appropriate for indexing and retrieval applications. An example video frame containing occurrences of the scene as well as caption text is illustrated in Fig. 1. Fig. 1Open in figure viewerPowerPoint Video frame with instances of scene and caption text From the viewpoint of text-based video indexing and retrieval, textual content in video frames needs to be detected (localised) and recognised. Keywords in the transcription of a video frame can then be extracted and employed for indexing and subsequent retrieval. The present study focuses on the latter of these, i.e. recognition of video text formally known as video optical character recognition (V-OCR). In contrast to printed text, recognition of text appearing in video frames is marked by many challenges. Typical problems include low resolution of text, complex and non-homogeneous backgrounds and, different font styles, sizes and colours etc. Another set of challenges is also introduced by the complexity of the script to be recognised. Thanks to more than five decades of extensive research [1–4], mature recognition systems have been developed for recognition of text in non-cursive scripts (languages based on Roman script for instance). Recognition of cursive scripts (like Arabic, Persian, Urdu etc.) is much more challenging and the research attention of the pattern recognition community in this problem is relatively recent (especially for caption text). To highlight the recognition challenges of cursive script, it is important to mention the complex word formation in such scripts. Typically, a word in a cursive language like Arabic or Urdu is a combination of one or more ligatures where a ligature represents one or more characters joined together through joiner rules. Joiner rules determine which characters are joined and which appear in isolated form. The shape of characters within a ligature is a function of its position (initial, middle, end, isolated etc.). Ligatures can therefore be considered as partial words. Ligatures are further categorised into primary and secondary components, the primary component being the main body of the ligature while the secondary components represent dots and diacritics. It is also worth mentioning that many ligatures share the same primary component and differ only in the number and/or position of dots (leading to high inter-class similarity). Furthermore, the non-uniform intra- and inter-word spacing in such scripts makes segmentation of lines into words highly complex hence ligatures or characters are mostly employed as units of recognition. Cursive text is printed (or rendered) in one of the standard scripts, Naskh and Nastaliq being two popular scripts (for Arabic and Urdu, respectively). Naskh script follows a horizontal baseline, i.e. characters are joined along a horizontal line. In Nastaliq, on the other hand, characters are joined diagonally making it highly cursive. This diagonal style also results in overlapping of neighbouring characters both horizontally as well as vertically making segmentation of characters much more challenging as compared to Naskh. Fig. 2 illustrates an Urdu text line printed in both Naskh and Nastaliq scripts. Among these, Nastaliq being the more common script for Urdu text makes the subject of our current study. More specifically, we target recognition of cursive Nastaliq text appearing in video frames using Urdu as a case study though the findings can be generalised to other cursive scripts as well. An example cursive (Urdu) text line illustrating various recognition challenges is presented in Fig. 3. Fig. 2Open in figure viewerPowerPoint Sample text line in (a) Nastqliq, (b) Naskh script Fig. 3Open in figure viewerPowerPoint Example of cursive text line highlighting recognition challenges Recognition of cursive text is typically carried out using holistic or analytical methods. Holistic techniques employ ligatures (or words) as recognition units while analytical methods rely on recognition of individual characters either through explicit or implicit segmentation. Holistic techniques avoid the complex character segmentation step but have to deal with a large number of unique ligature classes. On the other hand, in case of analytical techniques, the number of unique classes to recognise is relatively smaller and equals the number of characters in the alphabet (and their various context-dependent shapes). Segmentation of cursive text into characters, however, is a highly challenging problem in itself. The recent advancements in deep neural networks (DNNs) have led to the development of implicit segmentation-based recognition systems where the learning algorithm is provided with images of text lines and the corresponding transcription. The algorithm not only learns character shapes but also the segmentation points hence avoiding the complex explicit segmentation of ligatures into characters. This paper presents a detailed study on the recognition of cursive caption text. Text line images extracted from video frames are pre-processed and fed to a convolutional neural network (CNN) that extracts a sequence of features using sliding windows. The feature sequences are then used to train a long short-term memory (LSTM) network with a connectionist temporal classification (CTC) layer for sequence alignment. A series of experiments on text lines extracted from News channel videos reports high recognition rates. The key contributions of this study are listed as follows: • Development of a large data set of video frames along with ground truth information to support the evaluation of text recognition systems. • Investigation of various pre-processing (binarisation) techniques to segment text from background enhancing the recognition performance. • A comprehensive recognition framework for cursive caption text using CNN with different variants of recurrent neural networks (RNNs). • Validation of proposed techniques through an extensive series of experiments. This paper is organised as follows. In the next section, we present an overview of the current state-of-the-art on recognition of caption text. Section 3 introduces the developed data set along with the labelling tool and ground truth information. The proposed recognition technique is detailed in Section 4 while Section 5 presents the experimental protocol, results and discussion. Finally, we conclude the paper in Section 6 and discuss open challenges and potential research directions on the problem. 2 Related work OCR is one the most classical pattern recognition problems that have been investigated in images, documents, natural scenes and videos for more than five decades [3]. From the recognition of isolated characters and digits to complex end-to-end systems, the domain has matured significantly over the years. Formally, the task of an OCR system is to take a set of pixel data which contains textual information as input and convert it into a corresponding string as output. Thanks to the extensive research endeavours, mature recognition systems like Google Tessearact [5] and Abbyy FineReader etc. have been developed reporting near to 100% recognition rates on the text in multiple scripts. However, as discussed earlier, recognition of text in cursive scripts still remains challenging especially when it comes to caption text [6]. For text recognition in document images, a number of techniques have been presented both at character (analytical) and word (holistic) levels. Among methods employing characters as units of recognition graph-based models [7–10], Bayesian classifiers [11–13] and hidden Markov models (HMMs) [14–17] etc. have been typically investigated. Among holistic or word level recognition techniques, a wide variety of features as well as classifiers have been studied [18–21] reporting high recognition rates. Deep learning has also been employed for feature extraction and classification at character and word levels [22–24]. Contrary to document images, recognition of text from scene images is much more challenging due to camera perspective, varying lighting conditions and unconstrained backgrounds. Among well-known studies on this problem, histogram of oriented gradients (HOG) [25–27], Strokelets [28, 29], and SIFT descriptors [30, 31] have been proposed as popular features. For classification ANNs [32–34] and SVMs [35, 36] have been widely employed. A special case of scene text recognition is the recognition of text on road signs and has also been investigated in a number of studies [37–41]. Word spotting-based approaches have also been employed on scene text images [42–44] in the literature. In a number of relatively recent studies, the combination of CNNs with RNN (and variants) have ben effectively applied to text recognition [45–49]. Further advancements in deep learning techniques led to the development of more sophisticated architectures. Notable of these include binary convolutional encoder–decoder network (B-CEDNet) using bidirectional-RNN [50], character-aware neural network (Char-Net) using LSTM [51], character attention fully convolutional network (CA-FCN) [52] and double supervised network (DSAN) [53]. These DNN-based models claim to be robust and efficient for scene text recognition in challenging scenarios. While the challenges related to camera perspective and non-homogeneous backgrounds are not encountered in case of artificial video text, a major recognition challenge is the possible low resolution of the text. Multiple frame integration technique [54–56] has been widely employed to improve the quality of video image text by generating high resolution images prior to recognition. In [57], a holistic and component-based approach is used for text recognition in low-resolution videos reporting high recognition rates. A new classifier called FCNN (fuzzy-clustering neural network) [58] is proposed for recognition of Chinese text appearing videos. A step-wise language model is identified with ANN to develop new V-OCR in [59]. In another work [60], structural features of characters are employed for recognition of text in videos. Recognition of video text for indexing and retrieval applications has also been explored in a number of studies [61–63]. Among recent studies on this problem, Zayene et al. [64] presented a segmentation-free technique based on multidimensional LSTM (MDLSTM) for recognition of Arabic video text. The proposed technique reports high recognition rates on two benchmark data sets, ACTiV [65] and ALIF [66, 67]. Using the same data sets, another study is reported by Jain et al. [68] with CNN–LSTM hybrid DNN. Another notable method for video text recognition is presented in [69] where an SVM is employed to select an appropriate colour channel and the selected channel is exploited to recognise text using HMMs. Lu et al. [70] employ transfer learning with pre-trained CNNs for video text recognition. State-of-the-art models including InceptionV3, VGG16 and Resnet50 are considered in this study. A similar study for recognition of East Asian languages is presented in [71] where Chinese characters are recognised using CNNs. Regarding recognition of cursive scripts and more specifically Urdu text, significant research efforts have been made in the last few years to develop Urdu text recognition systems. These systems primarily target scanned document images of printed text (in Nastaliq script). Due to challenges already discussed, implicit segmentation-based techniques have remained a popular choice of researchers [72–75]. Likewise, in the case of holistic approaches, ligatures have been typically employed as recognition units [76]. The initial research endeavours on recognition of Urdu text mainly targeted isolated characters [77, 78] or already segmented ligatures [79]. Among significant holistic approaches, HMMs have been widely employed for recognition of ligatures [80–83]. These techniques use the sliding windows to extract features from ligature images which are projected in the quantised feature space hence representing each ligature image as a sequence. In some cases, the main body and dots are separately recognised [76] to reduce the total number of unique classes which can be very high in case of Urdu text (Urdu has more than 26,000 unique ligatures [84]). A number of holistic techniques are based on word spotting [85, 86] rather than recognition, to retrieve documents containing words similar to those provided as a query. Among implicit segmentation-based techniques, Hassan et al. [87] employ bidirectional LSTM networks with CTC output layer to recognise text from Urdu text lines. In a similar study, Naz et al. [72, 88] use MDLSTM with CTC layer and report more than 98% character recognition rate. Most of the studies reported on recognition of Urdu text are evaluated on one of the two publicly available databases, Urdu Printed Text Image (UPTI) [89] and Center for Language Engineering (CLE) [90] data set. UPTI is a synthetically generated data set containing more than 10,000 text lines with ground truth transcription while CLE database contains two parts; the first part contains clusters of high-frequency ligatures while the second part contains scanned images of Urdu books printed in Nastaliq font. A few efforts have also been made towards recognition of Urdu handwritten text [91] and a labelled data set of Urdu handwriting [92] is developed. The literature is relatively limited when it comes to recognition of Urdu caption text. A holistic approach is presented by Hayat et al. [93] where a number of pre-trained CNNs are employed to recognise a small set of 290 unique ligatures. While high recognition rates are reported, the number of ligatures considered is very small. In another recent study [94], bi-directional LSTMs are employed for recognition of Urdu News tickers. The technique is evaluated on a custom developed data set and the performance is compared with a commercial recognition engine. The discussion on recognition systems presented above summarises the major developments on this problem. The advent of deep learning solutions represents a key milestone which allowed the development of robust recognition systems significantly reducing the error rates. The solution we adopt for recognition of cursive video text is also inspired by the success of different DNNs on similar problems. Prior to presenting the details of the proposed solution, in the next section, we introduce the data set that has been developed as a part of our study. 3 Data set For algorithmic development and evaluation of the proposed system, we developed a benchmark data set of video frames that are labelled from two perspectives, detection of text regions and recognition of text. Detection performance refers to how effective the system is in localising the textual content while recognition performance evaluates the effectiveness of converting images into text. From the viewpoint of localisation, the bounding box of each textual region in the frame is identified and stored. Similarly, for recognition performance evaluation, transcription of each textual region needs to be stored in ground truth information. In the present study, we focus on the recognition task hence the textual regions are extracted from video frames using the ground truth information. The extracted text regions are then employed to train and evaluate the system. We have collected 46 videos from different News channels, all videos are recorded at a frame rate of 25 fps with a resolution of . Due to redundant information in successive frames, one frame every 2 s is extracted and labelled. Samples of the frames from different News channels are presented in Fig. 4 while important statistics on collected videos, frames, text lines, words and characters are summarised in Table 1. Fig. 4Open in figure viewerPowerPoint Sample video frames from different News channels Table 1. Statistics of labelled video frames S# Channel Videos Labelled images Urdu lines English lines 1 Ary news 7 3206 10,250 3605 2 Samaa news 13 2503 10,961 4411 3 Dunya news 16 3059 10,723 8,861 4 Express news 10 2424 8536 6755 total 46 11,192 40,470 23,632 In order to facilitate the ground truth labelling process, a labelling tool (Fig. 5) was developed that allows storing the location and ground truth transcription of each textual region in a video frame. The ground truth information associated with a frame is stored in the XML format. The ground truth file consists of two parts, frame meta-data and information of the textual content. Meta-data of each frame contains information on video, channel ID and a unique code for identifying the frames. The second part of the XML file has information of each text region in the frame along with a unique identity for each and, type of text (scene text or caption text). Localisation information of a text region is stored with its x and y coordinates and the width and height of the bounding rectangle. Transcription of text is stored using Unicodes of each character. Fig. 6 illustrates the ground truth information of an example frame stored in XML file. Fig. 5Open in figure viewerPowerPoint Ground truth labelling tool Fig. 6Open in figure viewerPowerPoint Labelled frame information in an XML file containing bounded-boxes and transcriptions of text As mentioned earlier, text lines extracted from video frames using the ground truth information are used to train the learning algorithm. Few examples of text lines are illustrated in Fig. 7. In an attempt to enhance the size of training data (to ensure maximum representation of various character shapes and their combinations), we also generated a set of 50,000 synthetic text lines. The lines are generated using the text taken from different books and News portals. To ensure a close resemblance with the actual data, various backgrounds are extracted from actual News channel videos and the synthetically generated text is superimposed on these backgrounds (Fig. 8). Samples of such synthetic text lines are presented in Fig. 9 where it can be seen that the generated text line images look very similar to the actual text lines extracted from video frames. Fig. 7Open in figure viewerPowerPoint Sample text lines extracted from video frames Fig. 8Open in figure viewerPowerPoint Generation of synthetic text lines Fig. 9Open in figure viewerPowerPoint Examples of synthetically generated text lines 4 Methods This section presents the details of the proposed recognition technique. The text line images are first pre-processed to segment text from the background. The binarised images of text lines are then fed to a CNN for feature extraction. The generated feature map is then fed to an RNN using sliding windows. Finally, being a sequence-to-sequence mapping problem, we need a CTC layer for sequence alignment. An overview of these steps is presented in Fig. 10 while each of these is detailed in the following. Fig. 10Open in figure viewerPowerPoint Overview of the key processing steps 4.1 Pre-processing While the recognition engine can be fed with coloured or greyscale images, removing the background information and binarising the image allows the learning algorithm better learn character shapes and boundaries. For images with simple and homogenous backgrounds, global thresholding suffices. Video frames, however, often contain text on multiple, non-homogeneous backgrounds. Furthermore, there are two scenarios in which text may appear; dark text on a bright background or bright text on dark background. Once the image is binarised, we need all text lines to follow one of the two conventions. In our study, we assume dark text on a bright background and if this is not the case, we invert the polarity of the greyscale image prior to binarisation. As a first step, we need to detect the polarity of the text. The canny edge detector is applied to the greyscale text line image and blobs are identified. These blobs correspond to text regions in the image. Region filling is applied to these blobs and the generated binary image is used as a mask on the greyscale image to extract potential text regions (characters or ligatures). We then compute the median grey value () of the extracted blobs as well as the median grey value of the background (all pixels which do not belong to any blob), . If we have dark text on bright background and the polarity agrees with our assumed convention. On the other hand, if , this corresponds to bright text on dark background. In such cases, the polarity of the image is reversed prior to any further processing. The process is summarised in Fig. 11 while more details can be found in our previous work [95]. Fig. 11Open in figure viewerPowerPoint Identification of polarity of text (a) Original image, (b) Grey scale image, (c) Text blobs, (d) Filled text blobs serving as a mask to extract corresponding blobs from the grey image, (e) Final image (Image on the right is inverted while the one on left remains unchanged) Once all text lines contain text in the same polarity, we binarise the images to contain only textual information. For binarisation, we investigated a number of thresholding techniques. These include Otsu's global thresholding method [96] as well as a number of local thresholding algorithms. The local thresholding algorithms are adaptive techniques where the threshold value of each pixel is computed as a function of the neighbouring pixels (rather than empirically fixing the threshold). Most of these algorithms are inspired by the classical Niblack thresholding [97] where the threshold is computed as a function of the mean and standard deviation of the grey values in the neighbourhood of a reference pixel. Other algorithms investigated in our study include Sauvola [98], Feng's [99] and Wolf's thresholding algorithm [100]. Prior to binarising the images, we also apply a smoothing (median) filter on each text line to remove/suppress any noisy patterns in the image. Binarisation results of applying various thresholding techniques to a sample text line image are illustrated in Fig. 12. From the subjective analysis of these results, Wolf's algorithm that was specifically proposed for low-resolution video text seems to outperform other techniques. Nevertheless, it is hard to generalise from visual inspection of a few sample images and the recognition rates on images generated by each of these techniques could be a better indicator of the effectiveness of the method. Following binarisation, we normalise the height of each text line to a fixed size (90 pixels in our case) while the width of the line is a function of the textual content it contains. Fig. 12Open in figure viewerPowerPoint Binarisation results on a sample text line (a) Greyscale image, (b) Niblack, (c) Ostu's global thresholding, (d) Feng's algorithm, (e) Sauvola, (f) Wolf's algorithm 4.2 Feature extraction Once the text lines are pre-processed, we proceed to the next step of feature extraction. As mentioned earlier, the text line image contains a sequence of characters which needs to be mapped to the corresponding sequence of characters in the ground truth transcription. The problem is hence formulated in a sequence-to-sequence mapping framework. The input sequences to the model can be raw pixel values or features (hand-crafted or machine-learned) extracted using a sliding window protocol. A number of recent studies [101–103], validated the superiority of machine-learned features over hand-engineered features (and raw pixel values). We, therefore, employ a CNN as a feature extractor. The architecture of CNN is a function of many hyper-parameters. In our study, we have employed seven convolutional layers (with max-pooling). The output of CNN is a dimensional feature vector. The architecture of the employed CNN can be seen in Fig. 13 while Fig. 14 demonstrates the correspondence between features and the respective regions in the image. Fig. 13Open in figure viewerPowerPoint Architecture of the CNN employed for feature extraction Fig. 14Open in figure viewerPowerPoint Feature extraction form text line image; each vector is associated with the respective region of the input image 4.3 Sequence predictor Feature sequences computed using CNN are fed to a sequence predictor for recognition. More specifically, we employed a bidirectional (RNN) and its different variants for sequence recognition. A simple RNN cell works by multiplying current input and previous output and employs the tanh activation function. Such vanilla RNNs are known to suffer from the vanishing (exploding) gradient problem during training hence different variants of simple RNNs were introduced. Common of these include the LSTM networks and gated recurrent units (GRUs). These sophisticated variants of RNNs are able to model long term dependencies in the input using gates. An LSTM has three (input, forget and output) while a GRU has two (reset and update) gates. Fig. 15 illustrates the architectures of simple RNN, GRU and LSTM cells. The proposed RNN architecture contains 512 time-steps as input and 2 stacks of hidden layers. Each stack contains a forward and a backward layer with 256 hidden units. Fig. 16 shows the employed model which produces a s