# Scene Text Recognition Based on Bidirectional LSTM and Deep Neural Network

Authors: MVV Prasad Kantipudi; Sandeep Kumar; Ashish Kumar Jha
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Deep learning is a subfield of artificial intelligence that allows the computer to adopt and learn some new rules. Deep learning algorithms can identify images, objects, observations, texts, and other structures. In recent years, scene text recognition has inspired many researchers from the computer vision community, and still, it needs improvement because of the poor performance of existing scene recognition algorithms. This research paper proposed a novel approach for scene text recognition that integrates bidirectional LSTM and deep convolution neural networks. In the proposed method, first, the contour of the image is identified and then it is fed into the CNN. CNN is used to generate the ordered sequence of the features from the contoured image. The sequence of features is now coded using the Bi-LSTM. Bi-LSTM is a handy tool for extracting the features from the sequence of words. Hence, this paper combines the two powerful mechanisms for extracting the features from the image, and contour-based input image makes the recognition process faster, which makes this technique better compared to existing methods. The results of the proposed methodology are evaluated on MSRATD 50 dataset, SVHN dataset, vehicle number plate dataset, SVT dataset, and random datasets, and the accuracy is 95.22%, 92.25%, 96.69%, 94.58%, and 98.12%, respectively. According to quantitative and qualitative analysis, this approach is more promising in terms of accuracy and precision rate.


---

## Body

## 1. Introduction

Understanding the visual scene is an active research area for the computer vision community. It needs enormous research in the field of computer vision and its subfields. Visual scene understanding includes the processing of both image and text, and it is always a difficult task to understand the scene and read the text written in the image. This research area is increasing gradually because it is helpful for many applications such as content-based image retrieval systems, assistance for blind people, automatic navigation systems in vehicles, and digitization of textbooks. OCR [1] is a traditional technique of recognizing the text from the documents, and the accuracy of this technique is good in the scanned documents; but when the same technique is applied to scene images, the performance of this method was not up to the mark [2]. The recognition of text from the scene needs special features because the character present in the scene may differ in size, shape, color, writing style, orientation, aspect ratio, quality of the image due to different lighting conditions, and blurred and complex background. These are the various challenges of text detection and text recognition. Generally, text detection identifies the location where exactly text is present in the image and creates a bounding box for each word or letter or line of text, and it also improves the accuracy of text recognition. The sample example of text detection is shown in the Figure 1.Figure 1
Sample image of text-involved scenes of SVT dataset.Text recognition allows the computer to understand and predict the text in the given input scene image and convert it into the computer’s understandable format. Text recognition is the most popular method for converting old printed documents into digitized forms. This process looks simple and more accessible because most of the image proceeding techniques follow the same. According to the literature survey, most of the text detection and recognition techniques are influenced by machine learning and deep learning techniques. Given the problems mentioned above and their solution, the main goal of this paper is to give a novel approach to text detection and text recognition based on deep neural networks and bidirectional LSTM. We proposed a scene text recognition method, and the proposed system is divided into three steps: in the first step, adaptive binarization technique is applied to the image so that the noise can be removed from the image, and it helps to extract the features from the blurred and complex background. In the second step, the contour detection technique is applied to the image, and it detects the meaningful area of the image, which makes the detection process easier and faster. In the third step, CNN-based architecture is designed in such a way that it can locate the text region and create a bounding box on each letter and also predict the characters. Here, CNN is combined with Bi-LSTM to make the classifier more powerful, and it is a handy tool for extracting the features from the sequence of words. This paper combines the two powerful mechanisms for extracting the features from the image and contour-based input image making the recognition process faster, which makes this technique better compared to existing methods. The complete detail of the proposed method is discussed in Section3. The rest of the paper is structured as follows: Section 2 describes the related work. Section 3 discusses the proposed work. Section 4 presents the experimental results and their comparison with existing methods. Section 5 shows the conclusion and future scope.


## 2. Literature Work

Many researchers have worked on the various techniques of detection of the text in images. Some researchers explored the texture-based approach for locating the text information in the image and used the sliding window concept to analyze the unique texture present in the input image [3–7]. Some researchers focused on sparse-based text detection methods used for computer vision applications [8–12]. It is proposed by Zhao et al. [8]. These methods work to transform the image into edge maps. A further sliding window is used to extract the text patches present in the image, and then classification has been performed. Most researchers focus on deep learning-based methods for scene text detection and recognition, and a detailed comparative analysis has been done in Table 1.Table 1
Literature study on existing methodology.
S. no.Author & yearMethodologyDatasetPerformance1S. Yasser Arafat et al. [12], 2020Faster RCNN + two stream deep neural network (TSDNN)UPTI datasetAvg. precision = 98%R. R. = 95.20%2Asghar Ali Chandio et al. [13], 2020Multiscale and multilevel featuresChars74 K and ICDAR03 datasetsPrecision = 90%Recall = 91%F-score = 91%3Yao Qin et al. [14], 2020Faster RCNN + BLSTMICDAR 2015 datasetsPrecision = 89.8%Recall = 84.3%F-score = 86.9%4Jheng-Long Wu et al. [15], 2020BLSTM + CNNCorpus datasetMacro-F1 = 72%Micro-F1 = 71%5S. Yasser Arafat et al. [16], 2020(AlexNet and Vgg16) + BLSTMUPTI datasetAccuracy = 97%6Sardar Jaf et al. [17], 2019Recurrent neural network (RNN) + BLSTMEnglish web treebank universal dependencies datasetPrecision = 91.43%Recall = 94.52%F-score = 92.20%7M. A. Panhwar et al. [18], 2019ANNSelf-datasetAccuracy = 85%8Yen-Min Su et al. [19], 2019Contour + morphological operation + ROIICDAR datasetsAccuracy = 93.44%Recall = 79.16%F-score = 85.71%9Ling-Qun Zuo Su et al. [20], 2019CNN + BLSTMSVT dataset, IIIT5K dataset, ICDAR 2003 and 2015 datasetAccuracy = 95.96%Accuracy = 98%Accuracy = 98.2%Accuracy = 91%10Baoguang Shi et al. [21], 2017CRNNSVT dataset, IIIT5K dataset, ICDAR datasetAccuracy = 97.5%Accuracy = 97.8%Accuracy = 98.7%Accuracy = 89.6%11Xiaohang Ren et al. [22], 2017Text structure component detector (TSCD)Ren’s dataset, Zhou’s dataset, Pan’s datasetPrecision = 82%Recall = 72%F-score = 77%12Xiang Bai et al. [23], 2016Bag of strokelets + HOGSVT dataset, IIIT5K dataset, ICDAR 2003 datasetAccuracy = 80.99%Accuracy = 85.6%Accuracy = 82.64%13Mingkun Yang et al. [24], 2021CAPTCHA systemIIIT5K, SVT, IC03 IC13, IC15, SVTP CUTEAccuracy = 92.9%, 89.6%, 92.5%, 92.2%, 76.8%, 80%, 77.1%14Anna Zhu et al. [25], 2021Anchor selection-based region proposal networkICDAR2013, ICDAR2015, and MSRA-TD500Precision = 90.18%, 83.34%, 84.67%Recall = 91.16%, 79.99%, 80.37%F-score = 90.62%, 81.63%, 82.49%15ZiLing Hu et al. [26], 2021Text contour attention text detectorICDAR2015, CTW1500Precision = 88.9%, 86.5%Recall = 85.2%, 80%F-score = 87%, 83.1%A tremendous amount of work has been done in scene text recognition, and results are also satisfactory [27–32]. However, these algorithms cannot give better results if the background is complex, blurry, and has different lighting conditions. The computation cost is very high when the algorithms are applied to the real dataset. Therefore, it remains a challenge.


## 3. Proposed Work

The proposed method is divided into three steps: firstly, finding the contour of the image; secondly, the text detection is using CNN; and thirdly, the text recognition using combined RNN and Bi-LSTM. The detailed description is discussed further, and the flow chart of the proposed method is shown in Figure2.Figure 2
Block diagram of proposed work.


### 3.1. Contour Detection

The scene text recognition-based method is essential to identify the region where exactly text is present in the image. Rather than working on the whole image, only the object’s boundary is sufficient for further processing. Considering the same, in the proposed method, the first contour of the image is identified [32]. Contour is used to find the boundary of the objects which is present in the image. These boundaries can be identified in different ways, such as finding the edges of the objects and finding the intensities of objects which are present in the image.In the proposed method, we used the wireframe-based boundary detection method in which the whole image is traced using structuring elements, and the first pixel of the object is identified. This first pixel represents the component of the object. Identifying the first pixel in the image always depends on how the tracing has to begin in the input image.The preferred direction of the tracing is the left most corner of the image and then towards the right direction of the image. The tracing of the image is continued until it will not find the contour of the whole image. Finally, all the boundaries of the objects are integrated, and the algorithm displays the contour of the image. The results of the contour detection are shown in Figure3.Figure 3
Original image and contour of the image on SVT dataset.


### 3.2. Text Detection Using CNN

The performance of any model always depends on the ability to discriminate the various features. An image-text can be arranged as a sequence of letters. A sequence of convolution and max-pooling layers are used to detect the text in an image. In the proposed method, four layers of CNN classify that the image’s patch contains a character. The configuration of CNN is represented in Table2 and Figure 4. First, the CNN classifier is trained with 62 classes in which 26 classes are used for uppercase letters, 26 classes are used for lower case letters, 10 classes are used for digits (0–9), and 1 for spacing. The image patches are directly classified as letters or digits; therefore, here, binary classifier is not required. The learned features are more specific and easy to discriminate from each other, making the learning process more accurate and speedy. The bounding box needs to be generated for each text present to detect the text in the image. The input image of this step is the contour image. There is a possibility that the input image can differ in size; therefore, to make it uniform, the size of the input image is 24 × 24, and each image is a greyscale image. First, the input image is padded from each side because if any character is near to the boundary of the image, then it can be detected through a sliding window. Using the sliding window, each row of the image is traced, NMS is performed to noise if it is present in the image, and the mean deviation and standard deviation of spacing are calculated. If the spacing value is lower than the threshold value, then it is considered that neighbour pixels are connected. Now, finally, the bounding box is identified for each character using a connected component analysis algorithm.Table 2
Hyperparameters used in the proposed work.
ParameterValueEpochs50Validation_split = 0.10.1Drop out0.2Filters16Batch_size64 × 64Learning rate0.00001Figure 4
Block diagram of RCNN combined with Bi-LSTM.


### 3.3. Scene Text Recognition Using Combined RNN and Bi-LSTM

This step is used to recognize the characters that are present in the image. Generally, the recognition system’s performance depends on the segmentation techniques, but sometimes good segmentation will also lead to poor recognition because of noise, different lighting conditions, different sizes of text, rotation and illumination, etc. Deep learning-based methods are used, and in this paper, to overcome these problems, we combined RNN and LSTM to improve the recognition rate. The first features are extracted from the image. The CNN classifier is used for the sequential feature extraction from the image, and training is done for all 63 classes mentioned in Section3.2. The feature extraction is done through the sliding window concept. The images which are already detected are the input for this step. The first padding of 12 pixels is done on the image, and the new image’s size is now 24 × 94. For the partition of the padded image, a subwindow is used with size 24 × 24. Each portioned patch of the image is fed into the trained CNN, and this trained CNN extracts the features from the image with size 4 × 4 × 256 and 1000 features which are the output of the 4th convolution layer and the first FC layer. These two feature vectors are combined, and it forms a one-dimensional feature vector of size 5096. PCA and normalization technique is applied to reduce the size of the feature vector. Now, the new feature vector is the size of 256-d, and these are the local and global features of the image. After extraction of local and global features from the image, the next step is feature labeling. For labeling of the feature, RNN is used in the proposed method. RNN is a unique neural network that can make use of past feature information, and it can also process the sequential inputs. To make the RNN more powerful, LSTM is combined here. LSTM has the capability of memorizing contextual information for a long time. LSTM consists of the memory cell and connection to itself and three gates that control the flow of information. The pictorial representation of the LSTM is shown in Figure 5.Figure 5
Block diagram of LSTM.it is the gate; Ct−1 is the status at the last cell, and it is hidden; ft is the forget gate; Ht is the final state of the latest Ct; W is the weight of each connection; and Ot is the output gate. The following equations compute the values of the previous parameters:(1)it=σWxi∗Xt+WhiHt−1+Wci∘Ct−1+bi,ft=σWxf∗Xt+WhfHt−1+Wcf∘Ct−1+bf,Ct=ft∘Ct−1+it∘tanhWxc∗Xt+WhcHt−1+bc,Ot=σWxo∗Xt+WhoHt−1+Wco∘Ct+bo,Ht=Ot∘tanhCt.It is better to access the past and future contextual information to recognize the text string properly. Bi-LSTM consists of two hidden layers in which one hidden layer is used to process the features in the forward direction and the other is used to process the features in the backward direction. Both the hidden layers have produced the output using the same output layer. Bi-LSTM is applied recursively for each feature present in the feature sequence in the sequence labeling process. According to the computation (mentioned in the above equation), it takes input as the current state and neighborhood state; every time,Ht is updated. After that, a softmax layer is used to distribute the state of Bi-LSTM into a probability distribution for 62 classes. The one extra class is used for finding the space between two words. Finally, the feature sequence is transformed into the sequence of probability P.Now, finally, the sequence of probabilityP must be transferred into a text string. In the proposed method, a CTC-based decoder is used for this purpose, and it is used for the classification of sequential text. For each time t, CTC calculates the probability distribution over the alphabet of possible characters, and it gives the characters which have the highest probability as output. The CTC object function is defined as follows:(2)O=−∑is,tsεSlnPts|is.It is the negative log probability function of the network which correctly labels the training dataset. Here,D represents the training dataset which consists of input and target sequence, and it is represented by (is, ts). Conditional probability is represented by Pts|is. The target O must be minimized, and it is equivalent to maximize the Pts|is. The object function is directly connected to the output of the Bi-LSTM layer, and it is defined as(3)Pis|ts=∑π:Bπ=tsPπ|p.The model is trained using gradient descent and backpropagation. In the above equation,B is used for removing the repeated and space labels. Suppose the sequence is B (c–c–f-), then the final output will be B(ccf). Once the model training is done, sequencing labeling aims to find the optimal path with max probability using Bi-LSTM.


## 3.1. Contour Detection

The scene text recognition-based method is essential to identify the region where exactly text is present in the image. Rather than working on the whole image, only the object’s boundary is sufficient for further processing. Considering the same, in the proposed method, the first contour of the image is identified [32]. Contour is used to find the boundary of the objects which is present in the image. These boundaries can be identified in different ways, such as finding the edges of the objects and finding the intensities of objects which are present in the image.In the proposed method, we used the wireframe-based boundary detection method in which the whole image is traced using structuring elements, and the first pixel of the object is identified. This first pixel represents the component of the object. Identifying the first pixel in the image always depends on how the tracing has to begin in the input image.The preferred direction of the tracing is the left most corner of the image and then towards the right direction of the image. The tracing of the image is continued until it will not find the contour of the whole image. Finally, all the boundaries of the objects are integrated, and the algorithm displays the contour of the image. The results of the contour detection are shown in Figure3.Figure 3
Original image and contour of the image on SVT dataset.


## 3.2. Text Detection Using CNN

The performance of any model always depends on the ability to discriminate the various features. An image-text can be arranged as a sequence of letters. A sequence of convolution and max-pooling layers are used to detect the text in an image. In the proposed method, four layers of CNN classify that the image’s patch contains a character. The configuration of CNN is represented in Table2 and Figure 4. First, the CNN classifier is trained with 62 classes in which 26 classes are used for uppercase letters, 26 classes are used for lower case letters, 10 classes are used for digits (0–9), and 1 for spacing. The image patches are directly classified as letters or digits; therefore, here, binary classifier is not required. The learned features are more specific and easy to discriminate from each other, making the learning process more accurate and speedy. The bounding box needs to be generated for each text present to detect the text in the image. The input image of this step is the contour image. There is a possibility that the input image can differ in size; therefore, to make it uniform, the size of the input image is 24 × 24, and each image is a greyscale image. First, the input image is padded from each side because if any character is near to the boundary of the image, then it can be detected through a sliding window. Using the sliding window, each row of the image is traced, NMS is performed to noise if it is present in the image, and the mean deviation and standard deviation of spacing are calculated. If the spacing value is lower than the threshold value, then it is considered that neighbour pixels are connected. Now, finally, the bounding box is identified for each character using a connected component analysis algorithm.Table 2
Hyperparameters used in the proposed work.
ParameterValueEpochs50Validation_split = 0.10.1Drop out0.2Filters16Batch_size64 × 64Learning rate0.00001Figure 4
Block diagram of RCNN combined with Bi-LSTM.


## 3.3. Scene Text Recognition Using Combined RNN and Bi-LSTM

This step is used to recognize the characters that are present in the image. Generally, the recognition system’s performance depends on the segmentation techniques, but sometimes good segmentation will also lead to poor recognition because of noise, different lighting conditions, different sizes of text, rotation and illumination, etc. Deep learning-based methods are used, and in this paper, to overcome these problems, we combined RNN and LSTM to improve the recognition rate. The first features are extracted from the image. The CNN classifier is used for the sequential feature extraction from the image, and training is done for all 63 classes mentioned in Section3.2. The feature extraction is done through the sliding window concept. The images which are already detected are the input for this step. The first padding of 12 pixels is done on the image, and the new image’s size is now 24 × 94. For the partition of the padded image, a subwindow is used with size 24 × 24. Each portioned patch of the image is fed into the trained CNN, and this trained CNN extracts the features from the image with size 4 × 4 × 256 and 1000 features which are the output of the 4th convolution layer and the first FC layer. These two feature vectors are combined, and it forms a one-dimensional feature vector of size 5096. PCA and normalization technique is applied to reduce the size of the feature vector. Now, the new feature vector is the size of 256-d, and these are the local and global features of the image. After extraction of local and global features from the image, the next step is feature labeling. For labeling of the feature, RNN is used in the proposed method. RNN is a unique neural network that can make use of past feature information, and it can also process the sequential inputs. To make the RNN more powerful, LSTM is combined here. LSTM has the capability of memorizing contextual information for a long time. LSTM consists of the memory cell and connection to itself and three gates that control the flow of information. The pictorial representation of the LSTM is shown in Figure 5.Figure 5
Block diagram of LSTM.it is the gate; Ct−1 is the status at the last cell, and it is hidden; ft is the forget gate; Ht is the final state of the latest Ct; W is the weight of each connection; and Ot is the output gate. The following equations compute the values of the previous parameters:(1)it=σWxi∗Xt+WhiHt−1+Wci∘Ct−1+bi,ft=σWxf∗Xt+WhfHt−1+Wcf∘Ct−1+bf,Ct=ft∘Ct−1+it∘tanhWxc∗Xt+WhcHt−1+bc,Ot=σWxo∗Xt+WhoHt−1+Wco∘Ct+bo,Ht=Ot∘tanhCt.It is better to access the past and future contextual information to recognize the text string properly. Bi-LSTM consists of two hidden layers in which one hidden layer is used to process the features in the forward direction and the other is used to process the features in the backward direction. Both the hidden layers have produced the output using the same output layer. Bi-LSTM is applied recursively for each feature present in the feature sequence in the sequence labeling process. According to the computation (mentioned in the above equation), it takes input as the current state and neighborhood state; every time,Ht is updated. After that, a softmax layer is used to distribute the state of Bi-LSTM into a probability distribution for 62 classes. The one extra class is used for finding the space between two words. Finally, the feature sequence is transformed into the sequence of probability P.Now, finally, the sequence of probabilityP must be transferred into a text string. In the proposed method, a CTC-based decoder is used for this purpose, and it is used for the classification of sequential text. For each time t, CTC calculates the probability distribution over the alphabet of possible characters, and it gives the characters which have the highest probability as output. The CTC object function is defined as follows:(2)O=−∑is,tsεSlnPts|is.It is the negative log probability function of the network which correctly labels the training dataset. Here,D represents the training dataset which consists of input and target sequence, and it is represented by (is, ts). Conditional probability is represented by Pts|is. The target O must be minimized, and it is equivalent to maximize the Pts|is. The object function is directly connected to the output of the Bi-LSTM layer, and it is defined as(3)Pis|ts=∑π:Bπ=tsPπ|p.The model is trained using gradient descent and backpropagation. In the above equation,B is used for removing the repeated and space labels. Suppose the sequence is B (c–c–f-), then the final output will be B(ccf). Once the model training is done, sequencing labeling aims to find the optimal path with max probability using Bi-LSTM.


## 4. Results

The experiments are performed on MSRATD 50 dataset, SVHN dataset, vehicle number plate dataset, SVT dataset, and random datasets to verify the performance of the proposed methodology. The experiments are performed on NVIDIA GTX 1650/60 Hz, 16 GB RAM, core-i7 10th generation processor. 80% of the dataset images are used for training purposes to train and test the model, and 20% of the images are used for testing purposes. Hyperparameters used in the architecture are described in Table 2.Existing methods are compared with the proposed method’s accuracy. We used accuracy, recall, precision, and F1-score in evaluating the proposed method. The accuracy is defined as the percentage of correctly classified instances. It is used to calculate the proportion of true positive and true negative for multiclass classification problems. The formula for calculating accuracy, precision, recall, and F1-score is given as follows:(4)accuracy=TP+TNTP+TN+FP+FN,precision=TPTP+FP,recall=TPTP+FN,F1Score=2.precision.recallprecision+recall.Here, TP = true positive, TN = true negative, FP = false positive, and FN = false negative.


### 4.1. MSRATD 50 Dataset

MSRA TD dataset is one of the benchmark datasets for text recognition, and this dataset contains 3000 images of 32 × 32 sizes. The dataset is challenging and noisy, and it contains English and Chinese text. The images in the dataset have blur and noisy background. The sample input images and the recognized text and its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 6. The proposed system had shown the accuracy of 95.22% and recall of 85.73%. The precision is 94.15%, and F-score is 87.09%. The metrics of the SVHN dataset are shown in Table 3.Figure 6
Text recognition on MSRA dataset.Table 3
Metrics of MSRATD 50 dataset.
Sr. No.ParametersOutput1Precision94.152Recall85.733F-score87.094Accuracy95.22


### 4.2. SVHN Dataset

SVHN dataset (Street View House Numbers) is the dataset that contains 600,000 digital numbers captured from various angles from various houses of Google street view. All images are of size 32 × 32. The images are blurred and have images captured from a different angle. The obtained accuracy is 92.25%, and recall, precision, and F-score are, 79.03%, 92.49%, and 89.80%, respectively. The sample input images and the recognized text along with its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 7. The metrics of the SVHN dataset are shown in Table 4.Figure 7
Text recognition on SVHN dataset.Table 4
Metrics of SVHN dataset.
S. no.ParametersOutput1Precision92.492Recall79.033F-score89.804Accuracy92.25


### 4.3. Vehicle Number Plate Dataset

We collected sample images from UFPR-ALPR dataset and tested them on our proposed method. The proposed method has shown an accuracy of 96.69%. The recall, precision, and F-score values are 93.11%, 86.77%, and 90.01%, respectively. The sample input images and the recognized text along with its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 8. The metrics of the SVHN dataset are shown in Table 5.Figure 8
Text recognition on vehicle number plates dataset.Table 5
Metrics of UFPR-ALPR dataset.
S. no.ParametersOutput1Precision93.112Recall86.773F-score90.014Accuracy96.69


### 4.4. SVT Dataset

SVT dataset is one of the challenging datasets where the images were taken from Google street view. The images in the dataset are high variability and meager resolution. The proposed method has shown an accuracy of 94.58%. The recall, precision, and F-score values are 84027%, 91.86%, and 88.49%, respectively. The sample input images and the recognized text along with its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 9. The metrics of the SVT dataset are shown in Table 6. The dataset results are partially accurate as the images are the 3D projections of original image.Figure 9
Text recognition on SVT dataset.Table 6
Metrics of SVT dataset.
S. no.ParametersOutput1Precision91.862Recall84.273F-score88.494Accuracy94.58


### 4.5. Random Dataset

We collected random text from the Internet to check the proposed method accuracy. The images were collected with a plain background and colored background. The accuracy of the proposed work is 98.12% as the samples contained the text and numbers without any noise in them. The obtained recall, precision, and F-score are 98.19%, 90.18%, and 97.07%, respectively. The sample input images and the recognized text along with its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 10. The metrics of the random dataset are shown in Table 7.Figure 10
Text recognition on random/self-dataset.Table 7
Metrics of random/self-dataset.
S. no.ParametersOutput1Precision90.182Recall98.193F-score97.074Accuracy98.12


### 4.6. Comparison Analysis of the Proposed Work

We analyzed the proposed method on four benchmark datasets MSRA TD dataset, SVHN dataset, UFPR-ALPR dataset, SVT dataset, and random text collected from the Internet and phone camera. The datasets are challenging datasets in various aspects. MSRA TD dataset is a tiny dataset of street view door numbers dataset that contains only images of house numbers. The dataset contains a blurred dataset; the proposed system can recognize the number with 95.22% accuracy. The SVHN dataset is another challenging blurred dataset containing both text and numbers with different backgrounds and fonts, and the proposed system has shown an accuracy of 92.25%. The UFPR-ALPR is a vehicle number plate dataset with different backgrounds. The proposed system has shown an accuracy of 96.69%. We considered the SVT dataset, a Google street view dataset with heavy background fluctuations and unclear text with various fonts and 3D reflections. The proposed system has shown accurate results with 94.58% of accuracy. We collected random datasets from the Internet and few images captured from Samsung mobile phones with minimal resolution. The proposed system has shown an accuracy of 98.12%. The analysis is given in Table8, and the corresponding graphs are plotted in Figure 11.Table 8
Metrics of various datasets used in the proposed system.
S. no.ParametersMSRATD 50UFPR-ALPRSVHNSVTRandom/self1Precision94.1593.1192.4991.8690.182Recall85.7386.7779.0384.2798.193F-score87.0990.0189.8088.4997.074Accuracy95.2296.6992.2594.5898.12Figure 11
Overall text recognition on all dataset.For some of the images, results are partially accurate. As we can see in Figure12, the images which were captured from the long distance or the orientation of the image are different. In those cases, model is able to detect the text partially. Proposed work is compared with the existing state of art methods, and according to the analysis, precision and accuracy is improved. The average recognition rate of proposed methodology and the comparison with state of art methods are shown in the Table 9 and Figure 13.Figure 12
Incorrect text recognition on all dataset.Table 9
Metrics of various datasets used in the proposed system.
S. no.ParametersRef. [22]Ref. [19]Ref. [18]Ref. [14]Proposed work (average)1Precision82—91.439092.152Recall7279.1694.529183.503F-score7785.7192.209188.564Accuracy—93.44——93.83Figure 13
Comparative analysis of the proposed work with existing techniques.


## 4.1. MSRATD 50 Dataset

MSRA TD dataset is one of the benchmark datasets for text recognition, and this dataset contains 3000 images of 32 × 32 sizes. The dataset is challenging and noisy, and it contains English and Chinese text. The images in the dataset have blur and noisy background. The sample input images and the recognized text and its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 6. The proposed system had shown the accuracy of 95.22% and recall of 85.73%. The precision is 94.15%, and F-score is 87.09%. The metrics of the SVHN dataset are shown in Table 3.Figure 6
Text recognition on MSRA dataset.Table 3
Metrics of MSRATD 50 dataset.
Sr. No.ParametersOutput1Precision94.152Recall85.733F-score87.094Accuracy95.22


## 4.2. SVHN Dataset

SVHN dataset (Street View House Numbers) is the dataset that contains 600,000 digital numbers captured from various angles from various houses of Google street view. All images are of size 32 × 32. The images are blurred and have images captured from a different angle. The obtained accuracy is 92.25%, and recall, precision, and F-score are, 79.03%, 92.49%, and 89.80%, respectively. The sample input images and the recognized text along with its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 7. The metrics of the SVHN dataset are shown in Table 4.Figure 7
Text recognition on SVHN dataset.Table 4
Metrics of SVHN dataset.
S. no.ParametersOutput1Precision92.492Recall79.033F-score89.804Accuracy92.25


## 4.3. Vehicle Number Plate Dataset

We collected sample images from UFPR-ALPR dataset and tested them on our proposed method. The proposed method has shown an accuracy of 96.69%. The recall, precision, and F-score values are 93.11%, 86.77%, and 90.01%, respectively. The sample input images and the recognized text along with its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 8. The metrics of the SVHN dataset are shown in Table 5.Figure 8
Text recognition on vehicle number plates dataset.Table 5
Metrics of UFPR-ALPR dataset.
S. no.ParametersOutput1Precision93.112Recall86.773F-score90.014Accuracy96.69


## 4.4. SVT Dataset

SVT dataset is one of the challenging datasets where the images were taken from Google street view. The images in the dataset are high variability and meager resolution. The proposed method has shown an accuracy of 94.58%. The recall, precision, and F-score values are 84027%, 91.86%, and 88.49%, respectively. The sample input images and the recognized text along with its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 9. The metrics of the SVT dataset are shown in Table 6. The dataset results are partially accurate as the images are the 3D projections of original image.Figure 9
Text recognition on SVT dataset.Table 6
Metrics of SVT dataset.
S. no.ParametersOutput1Precision91.862Recall84.273F-score88.494Accuracy94.58


## 4.5. Random Dataset

We collected random text from the Internet to check the proposed method accuracy. The images were collected with a plain background and colored background. The accuracy of the proposed work is 98.12% as the samples contained the text and numbers without any noise in them. The obtained recall, precision, and F-score are 98.19%, 90.18%, and 97.07%, respectively. The sample input images and the recognized text along with its bounded boxx-coordinate, y-coordinate, width, and height are listed in Figure 10. The metrics of the random dataset are shown in Table 7.Figure 10
Text recognition on random/self-dataset.Table 7
Metrics of random/self-dataset.
S. no.ParametersOutput1Precision90.182Recall98.193F-score97.074Accuracy98.12


## 4.6. Comparison Analysis of the Proposed Work

We analyzed the proposed method on four benchmark datasets MSRA TD dataset, SVHN dataset, UFPR-ALPR dataset, SVT dataset, and random text collected from the Internet and phone camera. The datasets are challenging datasets in various aspects. MSRA TD dataset is a tiny dataset of street view door numbers dataset that contains only images of house numbers. The dataset contains a blurred dataset; the proposed system can recognize the number with 95.22% accuracy. The SVHN dataset is another challenging blurred dataset containing both text and numbers with different backgrounds and fonts, and the proposed system has shown an accuracy of 92.25%. The UFPR-ALPR is a vehicle number plate dataset with different backgrounds. The proposed system has shown an accuracy of 96.69%. We considered the SVT dataset, a Google street view dataset with heavy background fluctuations and unclear text with various fonts and 3D reflections. The proposed system has shown accurate results with 94.58% of accuracy. We collected random datasets from the Internet and few images captured from Samsung mobile phones with minimal resolution. The proposed system has shown an accuracy of 98.12%. The analysis is given in Table8, and the corresponding graphs are plotted in Figure 11.Table 8
Metrics of various datasets used in the proposed system.
S. no.ParametersMSRATD 50UFPR-ALPRSVHNSVTRandom/self1Precision94.1593.1192.4991.8690.182Recall85.7386.7779.0384.2798.193F-score87.0990.0189.8088.4997.074Accuracy95.2296.6992.2594.5898.12Figure 11
Overall text recognition on all dataset.For some of the images, results are partially accurate. As we can see in Figure12, the images which were captured from the long distance or the orientation of the image are different. In those cases, model is able to detect the text partially. Proposed work is compared with the existing state of art methods, and according to the analysis, precision and accuracy is improved. The average recognition rate of proposed methodology and the comparison with state of art methods are shown in the Table 9 and Figure 13.Figure 12
Incorrect text recognition on all dataset.Table 9
Metrics of various datasets used in the proposed system.
S. no.ParametersRef. [22]Ref. [19]Ref. [18]Ref. [14]Proposed work (average)1Precision82—91.439092.152Recall7279.1694.529183.503F-score7785.7192.209188.564Accuracy—93.44——93.83Figure 13
Comparative analysis of the proposed work with existing techniques.


## 5. Conclusion and Future Scope

This research paper proposed a novel approach for scene text recognition that integrates bidirectional LSTM and deep convolution neural networks. In the proposed method, first, the contour of the image is identified, and then, it is feed into the CNN. CNN is used to generate the ordered sequence of the features from the contoured image. The sequence of features now coded using the Bi-LSTM. Bi-LSTM is a handy tool for extracting the features from the sequence of words. Thus, this paper combines the two powerful mechanisms for extracting the features from the image and contour-based input image making the recognition process faster, which makes this technique better compared to existing methods. The proposed method is evaluated on four benchmark datasets MSRA TD dataset, SVHN dataset, UFPR-ALPR dataset, SVT dataset, and random text collected from the Internet and phone camera. According to the quantitative and qualitative analysis, this approach is more promising in terms of accuracy and precision rate. The datasets are challenging datasets in various aspects. The proposed method can able to detect the text from the different backgrounds, unclear text, blurred images, different font size, and different orientation. In future, a better approach can be introduced which can deal with heavy background fluctuations and different 3D reflections.



---

*Source: 2676780-2021-11-23.xml*