Detected text‐based image retrieval approach for textual images

IET Image ProcessingVolume 13, Issue 3 p. 515-521 Research ArticleFree Access Detected text-based image retrieval approach for textual images Salahuddin Unar, Salahuddin Unar Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, 116024 People's Republic of ChinaSearch for more papers by this authorXingyuan Wang, Corresponding Author Xingyuan Wang wangxy@dlut.edu.cn Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, 116024 People's Republic of China School of Information Science and Technology, Dalian Maritime University, Dalian, 116026 People's Republic of ChinaSearch for more papers by this authorChuan Zhang, Chuan Zhang Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, 116024 People's Republic of ChinaSearch for more papers by this authorChunpeng Wang, Chunpeng Wang School of Information, Qilu University of Technology, Shandong, 250353 People's Republic of ChinaSearch for more papers by this author Salahuddin Unar, Salahuddin Unar Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, 116024 People's Republic of ChinaSearch for more papers by this authorXingyuan Wang, Corresponding Author Xingyuan Wang wangxy@dlut.edu.cn Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, 116024 People's Republic of China School of Information Science and Technology, Dalian Maritime University, Dalian, 116026 People's Republic of ChinaSearch for more papers by this authorChuan Zhang, Chuan Zhang Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, 116024 People's Republic of ChinaSearch for more papers by this authorChunpeng Wang, Chunpeng Wang School of Information, Qilu University of Technology, Shandong, 250353 People's Republic of ChinaSearch for more papers by this author First published: 31 January 2019 https://doi.org/10.1049/iet-ipr.2018.5277Citations: 12AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract This work addresses the problem of searching and retrieving similar textual images based on the detected text and opens the new directions for textual image retrieval. For image retrieval, several methods have been proposed to extract visual features and social tags; however, to extract embedded and scene text within images and use that text as automatic keywords/tags is still a young research field for text-based and content-based image retrieval applications. The automatic text detection retrieval is an emerging technology for robotics and artificial intelligence. In this study, the authors have proposed a novel approach to detect the text in an image and exploit it as keywords and tags for automatic text-based image retrieval. First, text regions are detected using maximally stable extremal region algorithm. Second, unwanted false positive text regions are eliminated based on geometric properties and stroke width transform. Next, the true text regions are proceeded into optical character recognition for recognition. Third, keywords are formed using a neural probabilistic language model. Finally, the textual images are indexed and retrieved based on the detected keywords. The experimental results on two benchmark datasets show the dominancy of text is efficient and valuable for image retrieval specifically for textual images. 1 Introduction Recent advancement in information technology and digital media, capturing, and sharing the information (i.e. image, video, and audio) has significantly increased. It needs an efficient method to retrieve such information existing in excessive amount. For this purpose, content-based image retrieval (CBIR) acting as a backbone in multimedia and computer vision communities since the last two decades [1-3]. In CBIR, for a given query image, the system will retrieve a number of similar images to the user from the database. The resulting images can be similar to the query image in sense of colour, shape, and texture of objects within the image under varying conditions and complex background. Image retrieval is a wide research field that includes several methods from information retrieval, machine learning, multimedia research, computer vision, and human–computer interaction. Image retrieval methods can be classified into two groups: text-based image retrieval (TBIR) and CBIR. TBIR methods need some heuristic information in the textual form (i.e. image descriptions and tags) for each image, and then the indexing and retrieval are performed by the textual queries. Such methods are sufficient for the limited number of database images with precise tags and description. However, the limitation with these methods is they need a huge number of human labour to manually annotate each image. Nowadays, the images are existing in millions of number, and it is almost impossible to annotate each image manually. To overcome such limitations, CBIR methods have been introduced. CBIR methods describe the images by visual contents (i.e. colour, shape, and texture) and depend heavily on analysing image descriptors and similarity measurement. For a robust CBIR system, the main purpose is to achieve higher accuracy with minimum computation time. To boost the retrieval performance, several methods have been introduced to retrieve similar images from the database [4-6]. However, researchers have not yet standardised any ideal approach and it remains a challenging problem. Owing to increment in image data, simple features such as colour, shape, and texture are not enough sufficient to construe the image efficiently. The existing methods mostly focused on extracting visual features such as colour, texture, shape, and fusing multiple visual descriptors [7-9]. Indexing of similar images is achieved based on these visual features. However, no standard method has been proposed yet for retrieving the textual images. Day by day, the increasing usage of social media sites (e.g. Instagram, Flickr, and Facebook), millions of people share their pictures. Mostly, these pictures may contain textual information that is an additional and clearer clue to perceive the image. It is a common practise among the people to edit the pictures by writing inspirational and motivational quotes, as shown in Fig. 1b. Sometimes, the pictures captured in natural scene environment also contain textual data under complex background, as shown in Fig. 1a. Therefore, the text embedded in images might be useful information for automatic tagging, annotation, and indexing. Exploiting such information can be used to retrieve similar textual images to the query image. Consequently, to improve retrieval accuracy of TBIR and CBIR for the textual images, the detected text can be an enormous asset to perceive the image more intensely. Fig. 1Open in figure viewerPowerPoint Sample textual images from datasets (a) ICDAR 2003 dataset, (b) Sindh dataset The automatic extraction of textual contents is really a challenging yet efficient task for several computer vision-based applications. For example, to help a blind person to read the contents within an image or to help a tourist to translate the contents of an image. Retrieving textual contents can be greatly efficient for robots to perform their specific actions. In recent years, the problem of text detection and localisation from the images has gained much attention [10-14]. Several methods have been proposed to detect the text from the images. However, their core objective is to detect and localise the text only. They do not consider the detected text for retrieving the similar images. We have highlighted some of the well known methods for text detection given in Table 1. Table 1. Art methods for text detection and localisation from natural scene images Method Precision Recall F value Features Determination Datasets Ezaki et al. [15] 60 64 62 connected component based text detection in natural scene images ICDAR 2003 Zhou et al. [16] 37 88 53 texture based text localisation and classification ICDAR 2003 Zhang and Kasturi [17] 67 46 — edge based text edges detection and extraction ICDAR 2003 Epshtein et al. [18] 73 60 66 stroke based SWTs ICDAR 2003ICDAR 2005 Ma et al. [19] 67 72 — edge and CC based component analysis and edge detection ICDAR 2003 Neumann and Matas [20] 59 55 57 texture and edge based text localisation using MSER ICDAR 2003Chars75K Yi and Tian [21] 71 62 62 connected components based text detection in natural scene images ICDAR 2003OSTD International conference on document analysis and recognition (ICDAR) Oriented scene text dataset (OSTD) Most state-of-the-art CBIR methods proposed to explore visual features. Sometimes the visual features are fused together to achieve high accuracy. Image indexing is performed based on these visual features. In [7], Liu et al. proposed colour information feature (CIF) by adding it with local binary pattern (LBP)-based feature, as LBP-based feature is not good at capturing rich colour information sometimes. CIF is capable to describe colour information, image brightness, and colour distribution. Walia and Pal [8] proposed a fusion framework that combines all the low-level features by employing colour difference histogram and angular radial transform. Yang et al. [22] presented a unique approach based on salient point detector and salient point expansion using local visual features. These salient points of image are obtained by using speeded-up robust features detector. To cope with large size of visual vocabulary, Wang et al. [23] proposed the hierarchy of medium-sized vocabularies. Sparse representation is adapted to select specific vocabulary. In [24], Dimitrovski et al. employed predictive clustering trees to construct indexing structure for codebook construction. The codebooks can efficiently increase the specific power of the dictionary. So far, these methods are developed for visual images (i.e. images containing the colourful objects) and cannot perform well for textual images. Moreover, several methods proposed to retrieve the images based on social tags and keywords. In [25], Li et al. proposed a model to extract visual, contextual, and semantic features to identify object and predicting the scene tag importance. However, sometimes these tags are not true and reflect the emotion only. Wu et al. [26] introduced a new method for incomplete and missing social tags. A tag matrix is used for image–tag relation that searches observed tags and visual similarity. Liu et al. [27] proposed a novel approach for improving improper social tags. Their approach represents the consistency of visual and semantic similarities along with social tags before and after improvements. Some authors introduced hybrid visual–textual relevance learning methods. Cui et al. [28] proposed a method based on textual–visual relevance learning. The method extracts the text from the image tags and associates the text with visual features. So far, these methods used visual features and social tags for indexing and retrieving the similar images and no standard method proposed yet to detect the text automatically and retrieve the images based on the detected text. In this paper, we have proposed a novel approach to retrieve the similar textual images by detecting embedded and scene text in textual images. In particular, we use text detection technique and employ the detected text as keywords and tags for indexing and retrieving the textual images. The key contributions of this work are as follows: To our best knowledge, there has been no standard method for textual images retrieval. This work is one of the few innovative investigations on indexing and retrieving textual images effectively. The proposed method is innovative in dealing with the textual images consisting the text within images (e.g. quote images, scene images, individual video frames). A fully automatic TBIR method is proposed to retrieve similar textual images based on the detected text from the complex background images. The detected text is employed as keywords/tags for indexing and retrieval. The method is robust and efficient to retrieve similar textual images for the given textual image query, and is based on easy-to-use framework. A new dataset of 1000 images consisting 20 categories is introduced. The dataset includes quote images, natural scene images, twitter snapshots, TV channel video frames, and other textual images. The rest of this paper is organised as follows: in Section 2, the proposed method is described briefly. Section 3 presents the different similarity distance measurements. In Section 4, a new dataset is introduced and experimental results are evaluated. Section 5 concludes the proposed approach and states future directions. 2 Proposed method In this section, we introduce a novel approach to detect the text, and employ that text to index and retrieve similar textual images. First, the candidate text regions are detected using the maximally stable extremal region (MSER) algorithm. After applying MSER, several non-text regions may still exist. To remove these non-text regions, we apply some geometric properties. Further filtration of non-text regions is carried out using stroke width transform (SWT). After obtaining positive text regions, we apply bounding boxes for forming text lines on each textual component. Once the text is localised and detected, it is faded into optical character recognition (OCR) for recognition. A neural probabilistic language model (LM) is employed for forming individual keywords from recognised text. Four different distance similarity measures including Euclidean distance, Canberra distance, Manhattan distance, and Cosine similarity are used to compute the similarity between the query image and the database images. Finally, top-rank images are retrieved based on the distance computation. The schematic illustration of the proposed approach is shown in Fig. 2. Fig. 2Open in figure viewerPowerPoint Schematic illustration of the proposed method 2.1 Character candidate extraction MSER has been identified as one of the best region detector due to its robustness against scale, viewpoint, and light changes. Several methods have adapted MSER to extract character candidate and achieved satisfactory results [29-31]. The main pros of MSER algorithm over other traditional methods is that the MSER can detect most of the textual components even with the low-quality images. Generally, the text has distinct contrast and appearance to its complex background and comparatively uniform colour intensity; hence, MSER is the best choice. The proposed method employs MSER to extract character candidate regions [32]. Let be the sequence of possible extremal regions, that is, , is an MSER if (1)If has a local minimum at i, then is one MSER, where is a parameter. After applying MSER filter, the obtained text regions are shown in Fig. 3b. Fig. 3Open in figure viewerPowerPoint Textual regions extraction (a) Original image, (b) Detected MSER regions 2.2 Non-text objects filtering After applying MSER, many non-text objects may still exist. We apply simple geometric properties such as width, height, aspect ratio etc. to filter out obvious non-text objects. The objects having maximum and minimum variations are eliminated first. There are numerous geometric properties that are best to distinguish text and non-text objects. The proposed method observes some of the geometric properties described in [33, 34] to eliminate non-text objects. Aspect ratio: The aspect ratio can be given as (2)We set the limit of character candidates' aspect ratio between 0.1 and 10. As some characters are very similar such as '0' and 'O', 'i' and 'l'; hence, we merge them into one category. Eccentricity: It is the distance ratio between the concentrations of the ellipse and its major axis length. It returns a scalar value that specifies the eccentricity of ellipse that has an equivalent second moment as the region value. An ellipse having 0 eccentricity is a circle and 1 eccentricity is a line segment. We set its value >0.995 to form a line segment. Extent: It returns a scalar that represents the ratio of pixels in the region to pixels in the total bounding boxes. It can be computed as total area divided by area of bounding box. We set its value between 0.2 and 0.9. Solidity: It returns a scalar that specifies the amount of the pixels in the convex hull that are also in the region. It can be computed as the area divided by convex area. We limit its value <0.3. Euler number: It returns a scalar that specifies the number of objects in a region minus the number of holes in the objects. It uses 8-connectivity to compute Euler number. We set its limit <−4. Size: Character candidates having the size of <5 px are regretted. As it may contain very limited information which can lead to time consumption. Most of the obvious non-text objects are removed after applying the above geometric properties, as shown in Fig. 4a. Once these conditions are satisfied, a character candidate can be further processed to the next step. Fig. 4Open in figure viewerPowerPoint Text detection and localisation (a) Geometric-based non-text objects filtering, (b) SWT-based non-text objects filtering 2.3 Stroke width filtering Geometric properties may not fully eliminate the non-text objects. Another common method used to distinguish text and non-text objects is stroke width. Stroke width can be defined as the length of a straight line from a text pixel to another pixel toward its gradient direction [35]. Several methods adapted stroke width for false positives elimination as it computes the width of curves and lines which can form a character [18, 36]. Text regions can have less stroke width variation, while non-text regions can have more variations. The proposed method follow up SWT to further eliminate false positives [18]. SWT is a local image operator that measures width per pixel of most prospective stroke consisting of the pixel. The size of output image is equal to the size of input image while each element containing the width of the stroke associated to the pixel. First, the initial value for each component of SWT is set to . A gradient direction of each pixel p is measured. If p covers stroke boundary, then also is perpendicular to the direction of the stroke. The ray , n > 0 is observed until another edge pixel q is found and the gradient direction is considered at pixel q. If is not obtained according to , then each component of SWT output image consisting the segment is allocated the width of until it has low value. The ray is rejected if q is not found or if is not obtained according to . We filter out the connected components based on criteria ratio that is stroke width standard deviation (std) to their stroke width mean. We set the ratio std/mean >0.5 obtained from ICDAR benchmark [37]. Once the false positives are removed, the true components are faded to the next step to form the text lines and word grouping. The attained positive text regions are shown in Fig. 4b. 2.4 Text line formation The adjacent character components are grouped together to form a straight line. To detect these lines, distinct textual components need to be merged into meaningful words. Character candidates that belong to the same text line are supposed to have similar properties (i.e. stroke width, height, size, and intensity). First, the midpoint of connected components is measured by applying Euclidean distance D and orientation angle between each connected component pair. In resultant, two maps are obtained, namely distance map and orientation map. If , then the two connected components are supposed to be adjacent characters, where is the maximum Euclidean distance from each connected component. By assuming that the text generally found in the horizontal orientation, we set value of between −45° and 45°. Each component pair satisfying the rule is checked by similarity criteria described in [35]. The components that satisfy the following criteria are processed further: where denotes width, height, stroke width mean value, and intensity of the bounding box, respectively. The threshold values can be adjusted according to the experimentation. If a line contains minimum three textual objects, it is declared to be a text line. The process ends when no more components can be merged further. The connected components satisfying the above conditions are grouped together and the rest of the components are supposed to be false positives and eliminated. The formed text lines are shown in Fig. 5a. Fig. 5Open in figure viewerPowerPoint Text formation (a) Text line formation, (b) Keywords formation Furthermore, the formed text lines are split into individual words for recognition purpose. We compute the overlap ratio between all the bounding box pairs by measuring the distance between the textual component pairs. The proposed method finds non-zero overlap ratio to locate group of neighbouring text regions. A threshold T is given as (3)where D is the distance vector that specifies the horizontal distance between the components. If the distance exceeds the threshold, it considers the two components belong to different words and they will be separated. We limit the value of to 1.5 and bounding boxes are applied to each word individually. The applied bounding boxes are shown in Fig. 5b. 2.5 Text recognition and keywords formation The true text regions detected in Section 2.4 are faded into OCR engine. There are several commercial and open source OCR tools freely available. We adapted Google's open source Tesseract OCR engine [https://opensource.google.com/projects/tesseract] for text recognition purpose. The proposed method employs recognised text words as tags and keywords for indexing the images. The natural phenomenon is to retrieve similar images based on its text confidence score and having maximum string match. The text words having high confidence score will be retrieved first. If no text is detected in an image, it becomes bit difficult to retrieve the image. Hence, we set the high recall ratio by increasing the false positives to get maximum keywords. For keywords formation, the proposed method employed a neural probabilistic LM that relies on character-level inputs and its predictions are still realised at word level [38]. The model is based on a convolutional neural network and its output from a single layer is used as an input at time t to recurrent neural network LM. Given as the size of vocabulary of recognised keywords, C is the vocabulary of characters, d is the dimensionality of each character, and is the character matrix. Suppose if word containing the combination of characters , where l is the length of word k. Then, the character-level representation for word k can be given by the matrix , where the jth column is corresponding to character cj. A narrow convolution between Ck and a filter kernel of width w is applied. Then, a bias is added and a non-linearity is applied to attained feature map . The ith component of feature map can be given as (4)where is the i to (i + w − 1)th column of and is Frobenius inner product. Given max-over-time(5)as feature corresponding to filter H for the word k. For a given filter, the basic approach is to acquire string having maximum score. The network exploits several filters with varying width w to obtain feature vector for each word k. The input of k is given as for total H filters . 3 Similarity measure For an accurate image retrieval system, features extraction and similarity measurement both plays an important role. Sometimes, feature extraction is realised smoothly but the similarity measurement is not chosen perfectly, hence the noisy result is achieved. The proposed method supports two approaches of operation: exact substring match and approximate substring match. Exact substring match retrieves the images having exact matching words compared with recognised keywords and approximate substring match retrieves the images with closest matching string. Generally, the exact substring match mode has the highest priority over approximate match and will be retrieved first. To compute the similarity distance between the detected text, we use Euclidean distance, Canberra distance, Manhattan distance, and Cosine similarity. The feature vector of each image in database is given as , where N is the number of recognised keywords in an image of database. Feature vector of query image q is given as , where N is the number of recognised keywords in q. The main idea is to select the similar images from database having maximum matching strings with the query image. The distance measures are given as follows. Euclidean distance (6)Canberra distance (7)Manhattan distance (8)Cosine similarity (9)where is the feature vector of an image in database and is the feature vector of query image. 4 Experimental results and discussion In this section, we will briefly present the experimental results and performance evaluation. All experiments are implemented and executed on a computer with 8 GB random access memory and 3.10 GHz central processing unit Intel Core-i5-2100. 4.1 Datasets The experiments are conducted on two benchmark datasets to ensure the accuracy and robustness of the proposed approach. Both the datasets containing the textual images. The datasets are given as follows: ICDAR 2003: The dataset [37] contains 500 natural scene images with the varying resolution from to . Out of which 251 images belongs to TrialTrain set and 249 images belongs to TrialTest set. The images are captured indoor and outdoor under the varying conditions (i.e. text size, font, colour, illumination, and position). The text appeared on signboards, banners, posters, and other objects. Sindh: We propose a new dataset, namely Sindh that contains total 1000 images including quotation images, twitter snapshots, natural scene images, TV news channel video frames, and other textual images. The resolution of each image varies from to , and collected randomly from Google, Instagram, and twitter. We divided these images into 20 different groups. 4.2 Retrieval performance protocol The performance accuracy of image retrieval can be computed using the mean average precision (mAP) that is the average of all image queries. The AP of top-ranked images is given as (10)where Rk is the top retrieved images and we set k = 10 since the users are more concerned with top-ranked results. The AP value for a single query is the average of precision value obtained for set of k images. The AP values are then averaged for all the queries. Given the set of relevant images for a query as , where Q is the set of all the queries, the mAP can be given as (11) 4.3 Implementation detail The proposed method detects the embedded text within images and uses it as keywords/tags for retrieving the textual images. To evaluate the performance and efficiency of proposed approach, first we will perform the experiments for text detection and recognition, and compare with the art methods. Next, we will evaluate the proposed method for detected TBIR. 4.3.1 Text detection and recognition In this section, we conducted the two experiments: (i) text detection and (ii) end-to-end text recognition. Experiment I: For an accurate and robust system, text detection is the most important task. For this purpose, we perform text detection evaluation on the benchmark datasets defined in Section 4.1 and compared the results with state-of-the-art methods. We follow up the standard evaluation protocols of precision p and recall r stated in [37]. The precision p, recall r, and frequency measure f are given as (12) (13) (14)where E is the number of total estimated words, T is the ground truth targets. The frequency measure f is used to combine precision and recall. The relative weights of precision and recall are controlled by . All the performance measures are computed for each image and then an average result is set for the performance of the proposed approach. For ICDAR 2003 dataset, the proposed approach achieved 74% precision and 68% recall values. For Sindh dataset, the method achieved 75% precision and 70% recall values. The results demonstrate that the proposed approach outperformed state-of-the-art methods for precision and f values on ICDAR'03 dataset. For Sindh dataset, the results have low accuracy due to the high complexity of different categories of images. The obtained results are given in Tables 2 and 3 for ICDAR 2003 and Sindh dataset, respectively. Table 2. Performance comparison of text detection on ICDAR 2003 dataset Method Precision Recall f Neumann and Matas [20] 0.59 0.55 0.57 Li and Lu [35] 0.59 0.59 0.59 Pan et al. [39] 0.66 0.70 0.68 Chen et al. [34] 0.73 0.60 0.66 proposed method 0.74 0.68 0.70 Table 3. Performance evaluation of text detection on Sindh dataset Method Precision Recall f proposed method 0.75 0.70 0.72 Experiment II: