Segmentation‐based recognition system for handwritten Bangla and Devanagari words using conventional classification and transfer learning

IET Image ProcessingVolume 14, Issue 5 p. 959-972 Research ArticleFree Access Segmentation-based recognition system for handwritten Bangla and Devanagari words using conventional classification and transfer learning Rahul Pramanik, Corresponding Author Rahul Pramanik rahul.wbsu@gmail.com Department of Computer Science and Engineering, Indian Institute of Technology (ISM) Dhanbad, Dhanbad, IndiaSearch for more papers by this authorSoumen Bag, Soumen Bag Department of Computer Science and Engineering, Indian Institute of Technology (ISM) Dhanbad, Dhanbad, IndiaSearch for more papers by this author Rahul Pramanik, Corresponding Author Rahul Pramanik rahul.wbsu@gmail.com Department of Computer Science and Engineering, Indian Institute of Technology (ISM) Dhanbad, Dhanbad, IndiaSearch for more papers by this authorSoumen Bag, Soumen Bag Department of Computer Science and Engineering, Indian Institute of Technology (ISM) Dhanbad, Dhanbad, IndiaSearch for more papers by this author First published: 16 March 2020 https://doi.org/10.1049/iet-ipr.2019.0208Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Offline recognition of handwritten text in Indian regional scripts is a major area of research as nearly 910 million people use such scripts in India. Most of the reported research works on Indian script-based optical character recognition (OCR) system have focused on a single script only. Research for developing methodologies that are capable of handling more than one Indian script is yet to be focused. As such, this has motivated us to study and experiment on creating a recognition system that can handle two most popular Indian scripts, namely Bangla and Devanagari. The authors propose a system that first detects and corrects skew present in Bangla and Devanagari handwritten words, estimates the headline, and further segments the words into meaningful pseudo-characters. This is followed by extraction of three different statistical features and combination of these features with off-the-shelf classifiers to study and identify the exemplary combination. Moreover, they employ state-of-the-art convolutional neural network-based transfer learning architectures and delineate a comparison with the extracted hand-crafted features. Finally, they amalgamate the identified pseudo-characters to provide the final result. On experimentation, the proposed segmentation methodology is discerned to provide good accuracy when compared with existing methods. 1 Introduction Automatic handwriting recognition has practical applications in libraries, post offices, banks, and so on. Such organisations process a large number of handwritten documents everyday. Automatic recognition system can save a lot of time even if it is possible to recognise half of such documents [1]. Automatic recognition of handwritten text is divided into two sub-domains, namely online and offline text recognition. In online text recognition, a stroke is recognised by pen tip movement and pen up and pen down switching on the surface of a personal digital assistant [2]. While, offline text recognition involves conversion of handwritten text on a scanned document into machine-readable format. Offline text recognition is comparatively difficult because it is not possible to capture the dynamic properties like pen tip movement, and so on. Evolution in recognition of printed documents has reached a new dimension while the same cannot be said for its handwritten counterpart. Handwriting still remains the principle mode of communication. The use of pen by a student to take note in a lecture is a definitive example. Use of handwritten notices and circulars in regional languages is still preferred over printed ones in several academic institutions and organisations. People tend to use handwritten documents in developing countries, especially in India, where Internet has not penetrated most parts. There are various sub-problems that requires to be addressed when dealing with offline handwritten text recognition, namely segmentation of lines, followed by segmentation of words and characters, extraction of optimal feature vector, and so on [3-14]. Due to a large variation in handwriting, this task is very challenging. Most of the reported works on handwritten word segmentation and recognition for Indian scripts have focused on a single script only [15-20]. There are a handful of research works that have developed OCR for printed documents that are capable of handling more than one Indian script [21-23]. But, only one work is present in the literature that have presented a methodology which is capable of handling two Indian scripts Bangla and Devanagari [24]. As such, research for developing methodologies that are capable of handling more than one Indian script is yet to be focused. This has motivated us to study and experiment on creating a recognition system that can handle two most popular Indian scripts, namely Bangla and Devanagari. Keeping the aforementioned shortcoming in mind, we propose a novel methodology that provides the following solutions: • estimation and correction of skew present in Bangla and Devanagari handwritten words that works even when the mtr is mostly discontinuous; • segmentation of the skew corrected words of two of the most used writing systems in the world, namely Bangla and Devanagari script into meaningful pseudo-characters; • extraction and combination of three different statistical features with off-the-shelf classifiers to study and identify the exemplary combination; • comparison between the extracted hand-crafted features and convolutional neural network based transfer learning (CNN-TL) utilising seven state-of-the-art pre-trained architectures, namely Alexnet [25], VGG-16 [26], VGG-19, Googlenet [27], Resnet18 [28], Resnet50, and Densenet201 [29]; • amalgamation of the identified pseudo-characters to provide the final result. The rest of the paper is organised as follows. Properties of Bangla and Devanagari script are elaborated in Section 2. The proposed methodology is delineated in Section 3. The experimental analysis and results are discussed in Section 4, followed by a brief conclusion in Section 5. 2 Characteristics and significance of Bangla and Devanagari scripts The Bangla and Devanagari scripts are derived from ancient Brahmi script. The Devanagari script is a highly adopted writing system in the world and is used for over 120 languages [30]. Few well-known languages using the Devanagari script include Awadhi, Bhojpuri, Bodo, Chhattisgarhi, Dogri, Garhwali, Haryanvi, Hindi, Kashmiri, Konkani, Magahi, Maithili, Marathi, Mundari, Nepali, Pali, Rajasthani, Sanskrit, Santali, and Sindhi. On the other hand, Bangla script is used for Assamese, Bangla, and Manipuri language [31]. There are 11 vowels and 39 consonants in Bangla script, while 11 vowels and 33 consonants in Devanagari script. Both scripts have certain similar syntactic and structural characteristics: • The scripts are written from left to right; • Both scripts have a strong inclination towards symmetrical round shapes within squared outlines; • In both scripts, a horizontal line runs along the top of characters. This horizontal line is called mtr in Bangla and shirorekh in Devanagari. Bangla and Hindi are the seventh and fourth most spoken language in the world [32], respectively. Bangla is also the de facto and official language of Bangladesh. Over 400 million people speaks Hindi while over 83 million people speaks Bangla language encompassing nearly 50% of the entire Indian population [33]. Digitisation at various departments of Indian administration has led to increased research interest in recognition of Bangla and Devanagari handwritten texts. It can be acknowledged from the aforementioned facts that Bangla and Devanagari scripts have a considerable significance. Keeping the above points in mind, we propose a methodology that is capable of handling both the scripts. 3 Proposed methodology We propose a strategy that first detects and corrects skew present in Bangla and Devanagari handwritten words, estimates the headline utilising [34], and further segments the words into meaningful constituent characters and modifiers. This is followed by extraction of three different statistical features and combination of these features with off-the-shelf classifiers to study and identify which combination provides the best result. CNN-TL is employed utilising seven pre-trained architectures, namely Alexnet, VGG-16, VGG-19, Googlenet, Resnet18, Resnet50, and Densenet201 and a comparison with the extracted hand-crafted features is delineated. Finally, the classified pseudo-characters are amalgamated to provide the recognised result. The proposed system is divided into five phases: preprocessing, headline estimation, segmentation, feature extraction, and classification and recognition. A complete working model of the entire approach is shown in Fig. 1. Fig. 1Open in figure viewerPowerPoint Workflow of the entire proposed methodology 3.1 Preprocessing The benchmark datasets that we use in our experimentation comprises of fairly non-complex-background documents. So, applying a local or a global binarisation methodology will provide very similar results. Initially, we use a global binarisation method [35] to binarise each handwritten word image , . Otsu [35] assumes that comprises two classes of pixels (foreground and background) and computes a threshold to separate the two classes such that their intra-class variance gets minimised [Algorithm 1 (see Fig. 2)]. The algorithm further uses the computed threshold to binarise . The binarised image is denoted as . We perform component labelling, followed by noise cleaning on each binarised word image [34]. Next, every is cropped to their boundaries. Fig. 2Open in figure viewerPowerPoint Algorithm 1: binarisation Each is traversed column-wise at a pre-defined distance [34]. The coordinates of the first foreground pixel () in each column are stored in a set . If a foreground pixel is not found in a chosen column, then coordinates of any pixel from that column are not stored in . Since, we store the coordinates of pixels in , we store the x-coordinates in and y-coordinates in . So, is denoted as while is denoted as where r: total # first pixel encountered in chosen columns. If we compute a regression line based on the pixels currently stored in , we may obtain incorrect angle for skew correction due to the presence of peaks and rifts in a Bangla or Devanagari handwritten word. An example of such peaks and rifts is shown in Fig. 3a. Keeping the above limitation in mind, we utilise the method delineated in [34] to obtain the correct skew angle of each . We utilise three different sets namely, , , and to determine the eligibility of each pixel stored in . A pixel is put in if it is determined to be further used for skew correction. Any pixel put in is finally removed from the set . We put a pair of consecutive pixel {, } in if we are unsure which one of them is ineligible. After further processing, one pixel is put in while the other in . For every three sequential pixels , , and in , is calculated. Fig. 3Open in figure viewerPowerPoint Stepwise illustration of inadmissible pixel removal (a) Peaks and rifts present in a Bangla word, (b) Condition to determine if any ineligible pixel exists among the consecutive three pixels, (c) Difference is computed for ineligible pixel determination between p4 and p5, (d) p5 is determined as ineligible and removed from , (e) Regression line is estimated, (f) Word image is rotated accordingly If [34] (Fig. 3b), then: • If , then , is stored in . • If , then {, } is stored in . Considering a special case, if a pixel in appears twice in two sequential pairs in in a single iteration, then is sent to , while the other two pixels from the two sequential pairs are removed from . After the completion of an iteration, the eligible pixels in are determined as = –(). For each remaining pixel pair in , the difference between y-coordinates and are computed between and with every pixel in (Fig. 3c). The pixel with maximum number of differences is sent to while the other pixel in the pair is removed from . Once all the pixel pairs in are traversed, the ineligible pixels are removed from as = – (Fig. 3d). Iterations are carried out until no pixel gets represented in or . values are predicted employing Algorithm 2 (see Fig. 4) utilising the eligible pixels in . The and values (where ) are used to fit a linear curve (Fig. 3e). The skewed angle is determined from the computed linear curve. Rotating directly by using the skewed angle may introduce zigzag or staircase error. To minimise this error we use nearest neighbour interpolation to rotate (Fig. 3f). Fig. 4Open in figure viewerPowerPoint Algorithm 2: linear curve fit pixel estimation 3.2 Headline and baseline estimation Same methodology is used for estimation of the headline. Once the admissible pixels are determined and stored in , predictive values are computed using Algorithm 2 (Fig. 4) for computing a linear curve. The computed linear curve is considered as the headline of the word image. This estimated headline is denoted as . For computing the baseline (lower part of the word where the vowel modifier is attached to a character), the lower half of the word image is considered. We compute the sum of all the transitions (from background to foreground and vice versa) and divide the sum with the height of the lower half of the image using the following equation: (1)where represents the average number of transition for each row in the lower half of the image. We traverse the lower half of the image starting from the bottom row m, checking the total number of transition in each row with . The first row whose total number of transitions exceed is the row that represents our baseline (). 3.3 Segmentation area identification The proposed model is based on the idea that water always flow downwards and fills cavities. If we consider as a terrain, and perform rainfall on from the bottom, the cavities where water gets stored are considered as lower rifts. Due to the presence of mtr in Bangla words and correspondingly shirorekh in Devanagari words, the space between characters form rifts. In the current work, rainfall in terrain model is used for detecting potential segmentation points in the word image as well as to detect and segment upper modifiers. Lower rifts in the word image are detected using Algorithm 3 (see Fig. 5). First, the number of closed area present in the word image is computed (Fig. 6a). Starting from the bottom row, the word image is traversed to the top. For each row, if a transition is encountered from foreground to background, the coordinates of the transition pixel () are stored. Traversing is continued on the same row until another transition () from background to foreground is found (Fig. 6b). The coordinates of this transition pixel are also stored and the background pixel values between and are changed from background to foreground. This step turns a rift () into a closed area (Fig. 6b). The foreground pixels from to in the next row, just below the transition pixels are turned to background pixels. This step ensures that no upper rifts [upper rifts: rifts that are formed when rainfall is performed from top of ] gets marked (Fig. 6c). Now, the number of closed area in the image is computed. If the area is indeed a lower rift, it should have changed into a closed area and as a result, the number of closed area should be one greater than the previous count. In that case, we mark all the pixels that constitutes the rift with a unique number and change back the pixels between and from foreground to background and the below row from background to foreground (Fig. 6d). Fig. 5Open in figure viewerPowerPoint Algorithm 3: lower rift identification Fig. 6Open in figure viewerPowerPoint Illustration of closed area and bottom rift identification (a) Closed area identified, (b) Closed area created in order to identify lower rift, (c) Upper rift eliminated from further computation, (d) Lower rift identified Due to the variation in handwriting, there are certain words where the upper modifiers namely, and get attached at both ends (Fig. 7). Ideally, for the upper part of the modifier should only be attached to its left counterpart while for the upper part of the modifier should only be attached to its right counterpart. If such variation of the modifiers are not taken care of, then the constituent component of the word may not segment correctly. Due to the presence of this type of abnormality, upper modifiers are handled in the present method using the rainfall in terrain concept. We segment the upper modifiers at first, before performing segmentation in the middle zone. Fig. 7Open in figure viewerPowerPoint Segmentation of upper modifiers for better segmentation output At first, potential segmentation points for upper modifiers are identified. For each , the area above is checked to find all closed area and lower rifts . If a closed area or a lower rift found exists below , then that closed area or lower rift is considered ineligible and discarded from or . All those closed area and lower rifts are also removed from and , respectively, whose masses fall below a certain threshold. For every closed area and lower rift in and , the left and right extreme points are determined and segmented (Fig. 7). Once the upper modifiers are segmented, the segmentation of constituent characters in the middle zone is considered. The lower rifts () occurring between and are computed and put in (Fig. 8a). The rifts whose mass falls below a certain threshold are removed from . Next, the eligibility of all the rifts in is determined for finding possible segmentation points using following three criteria: (2) (3) (4) Fig. 8Open in figure viewerPowerPoint Step-wise illustration of segmentation methodology (a) Lower rifts are detected (marked with magenta colour), (b) Eligible rifts are computed (marked with blue colour), (c) Potential segmentation points are identified, (d) Segmented area is marked with red colour, (e) Alternate segmented pseudo-characters are marked with three different colours A validation for choosing 65 and 40 as height percentage between and , respectively, is provided in Section 4.2. The rifts that do not qualify at most one of the above criteria are discarded from . Each of the remaining rift in is considered to contain a segmentation point (Fig. 8b). The centre of gravity (cog) of each of the eligible rift in is computed. The pixel closest to the segmentation point for each of the rift is computed as (5) The top most pixel of the rift is provided more weightage than the centre of gravity, as for the determination of the pixel closest to a segmentation point will be more inclined towards the headline of the word, thereby providing better segmentation result. Once a pixel close to the segmentation point is identified (Fig. 8c), the word image is traversed in upward direction until a transition from background to foreground is encountered or until is reached. The pixels from the encountered transition are changed from foreground to background up to or until a background pixel is reached as a segmentation procedure (Figs. 8d and e). is emptied and the lower rifts occurring between and are computed again and put in and the entire procedure of finding segmentation point between characters is repeated until no eligible lower rift exist for finding segmentation point. This ensures that the word has been successfully segmented and no under-segmented parts of it remains. We have made a list of the pseudo-characters that we successfully extract while performing segmentation on Bangla and Devanagari words. The entire list is provided in Fig. 9. We send these pseudo-characters for feature extraction and subsequently for classification. Fig. 9Open in figure viewerPowerPoint List of pseudo-characters that are successfully extracted while performing segmentation and to be used further for classification (a) Pseudo-characters extracted from Bangla words, (b) Pseudo-characters extracted from Devanagari words 3.4 Feature extraction Each pseudo-character image is normalised to pixels. We extract three different categories of statistical features from each of the images as described below: Type I feature: For extracting the first category of features, we divide the image based on the centre of gravity of the image. This gives us four unequal rectangular components. We again divide each of these rectangular component based on the centre of gravity of each of the component. This provides us with 16 unequal rectangular frames (Fig. 10a). Now, we have a total of 21 components, considering the image itself as a component and its four splitted unequal constituents. Four features viz., adjacent horizontal spread, adjacent vertical spread, and adjacent diagonal spread along the two diagonals are computed from each of the components. The adjacent horizontal spread denotes the sum of the length of the largest spread of black pixels along each row in each component. Similarly, vertical spread is the sum of the length of the largest spread of black pixels along each column in a particular component. Similarly, the largest spread along each major diagonal is denoted by the diagonal spread features. So, we extract 84 features from all the 21 components. Fig. 10Open in figure viewerPowerPoint Different types of features extracted (a) Type I feature, (b) Type II feature, (c) Type III feature Type II feature: For the second category of features, we divide the entire image into eight equal components. Each component has three sides (Fig. 10b). Again, we divide the entire image into four equal components and divide each of the component into eight equal constituent. Each of these constituents have three sides. So, we have a total of 40 three-sided components. If we pour water towards a side of a particular component and if part of foreground is encountered in the component, then water will slide on either direction, keeping the portion of the side of the component below it dry. We compute the length of the dry area as a feature. We pour water from top of the three sides of each of the components and compute the dry area length. So, we compute 120 features from all the 40 components. Type III feature: For the third category of features, we scan the entire image in a zigzag mode. For this purpose we designed four different strokes. The different strokes are marked with different colours (Fig. 10c). For each stroke, we store the number of foreground pixels encountered in an one-dimensional array, . We traverse the entire image from the bottom in a zigzag mode as well. Next, we divide the image into four equal non-overlapping components and compute the different counts of foreground pixels through the zigzag movement. We obtain another four one-dimensional arrays with foreground pixel count. We compute the mean, median, standard deviation, and skewness of each of the arrays using Algorithm 4 (see Fig. 11) and equations (6)–(8), respectively. Fig. 11Open in figure viewerPowerPoint Algorithm 4: median computation This provides us with features. So, we extract a total of features from each of the pseudo-character (6) (7) (8)where n denotes number of strokes while traversing in zigzag mode, denotes number of foreground pixels encountered during each stroke, while , , , and M denote the standard deviation, mean, skewness, and mode of , respectively. 3.5 Classification Majority of classifiers are differentiated based on their learning approach. Most popular learning approach includes function based learning, probabilistic learning, lazy learning, decision based learning, and so on. We have experimented on different learning based classifiers to identify the most effective classifier that provides the best result. Multi-layer perceptron (MLP) and support vector machine (SVM) is associated with function based learning. K-nearest neighbour (KNN) and KStar classifiers are associated with lazy learning while Random Forest is associated with decision based learning. We have briefly elaborated the classifiers namely MLP and Random Forest in this section, as they provide the highest accuracy during experimentation with the aforementioned features. MLP neural network constitutes of a set of origin nodes that forms the input layer, a set of computation nodes constituting one or more hidden layers, and a set of nodes composing the output layer. Each layer is fully and completely connected to the next layer. The network trains by procuring suitable weights for each and every connection such that for every input an appropriate output gets generated. The network adjusts weights in such a way that the calculated and expected outputs are as close as possible. Back-propagation (BP) algorithm performs parallel training for improving the efficiency of MLP network. BP utilises iterative gradient and reduces the error between the calculated and expected outputs of each and every node in the network for training the MLP. In the current work, we have employed single-hidden layer MLP network, so the input layer constitutes of 244 nodes and the output layer constitutes of 49 nodes for Bangla and 44 nodes for Devanagari script. Random Forest classifier uses decision tree model for classification. It creates several decision trees by randomly selecting subsets of the training set. At the final stage, the classifier merges the output of the decision trees to provide the final output. This procedure is called ‘bagging’. Random Forest employs its own hyper-parameters while growing the trees. The hyper-parameters help in making the modelling faster. Apart from MLP and Random Forest, we have also used SVM, KNN, and K-Star classifiers to provide a comparative performance analysis of the classifiers for the same set of features. We have used early-stopping in MLP to avoid overfitting. For this, we have used 20% of the training set as validation set. The training terminates when error increases for validation set or when accuracy continues to decrease consecutively for a certain amount of training iterations. For other classifiers, this is the case with various parameters when a large vector of attributes is used for training. Through exhaustive experimentation, we fine-tuned which value for each hyper-parameter of each classifier best fits the current dataset. Apart from this, we have also used crossvalidation for reducing overfitting problem. We have provided a detailed report of the various chosen parameter values in Section 4.3. 3.5.1 CNN-based transfer learning Convolutional neural networks (CNNs) are changing the way we look at technology. When a CNN is created from scratch for performing a particular task, it requires huge amount of training data for achieving the best of result. For tasks where much lesser data is available, CNN-TL provides better result. CNN-TL is a method that recycles the pre-trained CNN model on large databases for learning a new task (Fig. 12). Fine tuning a model employing CNN-TL is much faster than training a model from scratch. This is done by re-factoring a section of the weights from the pre-trained architecture and re-initialising the weights at final layers. We have used seven pre-trained architectures, namely Alexnet [25], VGG-16 [26], VGG-19, Googlenet [27], Resnet18 [28], Resnet50, and Densenet201 [29] for this purpose. Fig. 12Open in figure viewerPowerPoint Basic example of CNN-TL • Alexnet: It consists of 5 convolution and five max-pooling layers, followed by three fully connected layers including dropouts and ReLU. It won the ILSVRC 2012 contest [36]. • VGG-16 and VGG-19: VGGNet is a deeper configuration of AlexNet. This architecture uses small filters of size 3 3 for all layers. VGG-16 has 13 convolution layers followed by pooling, rectification layers, and three fully connected layers. VGG-19 is similar to VGG-16, except it contains 16 convolution layers. • Googlenet: The number of units in each layer in this architecture is increased by employing parallel filters named as inception module of size 1 1, 3 3, and 5 5 in each convolution layer. Although, Googlenet comprises of 22 layers, the number of parameters employed in this architecture is 12 times lesser than AlexNet. • Resnet18 and Resnet50: Resnets are deep CNNs that skip blocks of convolution layers by using shortcut connections. Resnet18 comprises of 18 layers and each resnet block is 2 layers deep, while Resnet50 comprises of 50 layers and each resnet block is 3 layers deep. • Densenet201: Densenet comprises of dense blocks in conventional CNN. It benefits by employing feature reuse for parameter efficiency and reduces vanishing gradient problem. This architecture is 201 layers deep. Since, the training data size is much smaller, the CNN architectures can be heavily biased towards the data they have seen in the training set, and therefore can be difficult for them to generalise the learned model to any other samples. The