Patent Publication Number: US-11037010-B2

Title: Compositional model for text recognition

Description:
BACKGROUND 
     Automated visual text recognition is a fundamental problem of computer vision. Over the years numerous approaches have been used. More recently, machine learning techniques have been applied to solve this problem. While machine learning techniques have been effective at recognition, they do have shortcomings. The typical approach for machine learning recognition is to have a detector that detects a region in an image that contains text. The detector then passes the region to a recognizer, which attempts to recognize the text in the region. Detectors lack the sophistication of recognizers and often detect inaccurate regions. Parts of letters are often truncated, and spatial transforms and distortions may compound the inaccuracy of detection. And yet recognizers have had no ability to adjust the detection region; omitted or distorted imagery may remain out of the recognizer&#39;s purview. There remains a need to improve detection of text with respect to systems for machine-learning based text recognition. 
     SUMMARY 
     The following summary is included only to introduce some concepts discussed in the Detailed Description below. This summary is not comprehensive and is not intended to delineate the scope of the claimed subject matter, which is set forth by the claims presented at the end. 
     Embodiments described herein relate to a two-stage end-to-end text recognition system. The text recognition system includes a text detection stage and a text recognition stage. Images inputted to the text recognition system are provided to both the text detection stage and to the text recognition stage. The text detection stage detects text regions in the images and provides the detected regions to the text recognition stage. The text recognition stage is trained to perform geometric rectification on the text regions using the images. There is end-to-end alignment between the text detection stage and the text recognition stage. Additionally, the text detection stage and text recognition stage are each trained independent of the other. 
     Many of the attendant features will be explained below with reference to the following detailed description considered in connection with the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present description will be better understood from the following detailed description read in light of the accompanying drawings, wherein like reference numerals are used to designate like parts in the accompanying description. 
         FIG. 1  shows a neural network. 
         FIG. 2  shows a compositional model. 
         FIG. 3  shows interaction between a geometric rectification module and a convolutional encoding module. 
         FIG. 4  shows a long short-term memory module. 
         FIG. 5  shows example text images and template skeletons. 
         FIG. 6  shows test scores. 
         FIG. 7  shows a graph of results for different models. 
         FIG. 8  shows examples. 
         FIG. 9  shows comparative results on recognition tasks. 
         FIG. 10  shows end-to-end comparative performance results. 
         FIG. 11  shows example input images, rectified images, and predicted templates. 
         FIG. 12  shows details of a computing device. 
     
    
    
     DETAILED DESCRIPTION 
     Overview 
     Core to the problem of text recognition are the degrees of variation within the word image construction process itself: what kind of font is used, with what kerning (spacing), on what surface, etc. These considerations have not been explored specifically for the problem of text or word image recognition. 
     Embodiments described herein model inverse inference for the word image construction process using deep neural networks. A compositional model for formation of a word image is set forth below. The compositional model is a composition of components that are involved in respective steps of word image formation. Discussion of this model is followed by description of modules to invert respective components in the compositional model. While the model and its inversion are new, compositional modeling in general has not previously been applied to the new regime of recognition algorithms that are based on connectionist temporal classification (CTC). Although, it should be noted that the techniques described herein are not focused on generative modeling but rather on the backward inference task of recognizing the text in an image. 
     The most successful recognition techniques stem from formulation of inference loss by means of CTC. CTC solves the many-to-many problem of sequence modeling by decoding predictions with dynamic programming. Although alternatives such as edit probability decoding have arisen recently, embodiments are described herein with reference to CTC-based decoding. Nonetheless, the principles described herein can be readily adapted to most segmentation-free decoding strategies. 
     With the recent adaptation of the segmentation-free framework of CTC, the variation of kerning or character segmentation within word images has not previously been appreciated or explored. Moreover, modeling of geometric nuisances has only recently been considered for deep learning frameworks. In the current trend of text recognition, few have explored canonical alignment of word images. Furthermore, character-based alignment suffers from the segmentation problem, which the CTC approach is designed to avoid. Thus, the techniques described herein for addressing the character alignment problem are intuitively unappealing, since failure to segment correctly will limit the registration and thus the recognition. The inventors are the first to model both the geometric noise of the text and the noise of the detector, which is well suited for 2-stage end-to-end text recognition systems. 
     It should be noted that the embodiments described herein are focused on a second stage (recognition) that works well independent of the first stage (detection). The text recognition embodiments disclosed herein provide a more robust 2-stage pipeline, as shown by empirical test results that are discussed below. 
     The embodiments described herein stem from a new model for text recognition that enables end-to-end alignment between the recognizer and a detector that is trained independently. The embodiments are the first to show that a template reconstruction loss can be used to stabilize the training of spatial transformer networks for text recognition, thus enabling a single joint training. 
     The embodiments described herein are based on the first complete and concise compositional model for word image formation applied to the text recognition problem. The embodiments can be used for end-to-end ground truth that has different levels of precision (axis aligned, oriented bounding box, or quadrilateral) all in the same data set. Furthermore, the embodiments are competitive with state-of-the-art recognizers, can be plugged into detectors trained independently, and yield top-notch end-to-end performance. 
       FIG. 1  shows a neural network  80  configured in accordance with embodiments described herein for text recognition. Initially, the neural network  80  is trained with training data  82 . The training data  82  comprises word images and their respective ground truth text equivalents. The training algorithm need not receive rectified boxes, and the the boxes/boundaries may be of heterogeneous types (e.g. some are rectangles, and some are quadrilaterals), but the boxes are to have the correct word transcriptions corresponding to them. The neural network  80  includes implicit layers or modules  84 . The parameters of the nodes of the neural network and the weights of their connections are set by the training data  82 . The trained neural network  80  is then used for text recognition. An input image  86  is passed to a detector  88  as well as to the neural network  80 . The detector  88  detects a word image  90  in the input image and outputs a boundary of the word image (e.g., bounding box coordinates) to the neural network  80 . The input image  86  is also passed to the neural network  80 . The neural network  80  processes the word image from the detector using the input image  86 . As the word image passes through the layers or modules of the neural network the boundary of the word image is adjusted (e.g. expanded) according to the trained state of the neural network  80 . That is, the neural network  80  both rectifies (spatially transforms) the word image and recognizes the text in the rectified word image. The rectification may remove perspective distortion from the inputted (detected) word image, enlarge the word image, etc. The neural network  80  outputs recognized text  92  and optionally the word image as rectified by the neural network  80 . 
     Forward Compositional Text Model 
     As noted above, rectification and recognition can leverage an explicit decomposition of the text formation process. One approach involves five distinct steps, not all of which are necessary: transcription and skeletonization; kerning; typesetting (font); lighting, shading, and background distortion (appearance); and geometric distortion.  FIG. 2  shows a compositional model of the text generation process that an inference network can be correspondingly modeled on. Starting with a word or label  100 , e.g., “GENE”, the label is first passed to a skeletonization function  102  that skelotonizes the label. A kerning function  104  is applied to the skeletonization of the label, and the kerned version is then passed to a font function  106  that applies a font. An appearance function  108  may render the output from the font module  106 , and a geometric distortion function  110  applies some geometric transforms (e.g., affine) to produce a word image  112 . The word image  112  conceptually corresponds to a digital image of a physical rendering of the label word. 
     Below, each step in this text formation process is modeled. Moreover, further below is discussion of performed ablation studies showing improved performance over the CRNN (convolutional recurrent neural network) baseline. As discussion proceeds, it should be considered that the compositional model is inverted, thus inverting the text rendering process in a way that is suitable for neural networks. Although each component in the model is described in its forward form, the task is ultimately formalized in terms of the inversion of these components. 
     Transcription and Skeletonization 
     Together, the transcription, skeletonization, font, and kerning parameters of a word image are the intrinsic parameters of that word image. Whereas the geometric distortion and appearance of the word image are extrinsic parameters of the word image. This section discusses transcription and skeletonization. 
     Transcription is the process of converting an atomic word symbol into a sequence of character symbols that will be rendered. Formally, transcription can be said to convert a word symbol from a dictionary, w∈D, to a sequence of characters from an alphabet, s w ∈Σ*, by a function T:D→Σ*. 
     The skeletonization function  102  converts a sequence of character symbols into a canonical word image. This step of rendering a canonical word image is referred to herein as skeletonization, which reflects the choice to use character skeletons to represent visual character symbols. Thus, skeletonization converts a sequence of characters from an alphabet, s w ∈Σ*, to an image, s:Σ*→B(I) where B(I) is the set of bounded functions on the spatial image domain I. Below, the term s∘T is replaced with s, assuming that the input will be in the form of a character sequence. 
     Kerning 
     The kerning function  104  imposes spacing between characters, and it does so independent of the whichever font is used. The function s does not directly model kerning. Instead, the kerning operation is modeled as a function of the image produced by s(w). The kerning function  104  converts an image with one spacing to an image with another spacing, which may be nonuniform between characters. The kerning can be different for identical word inputs. This variability is modeled with free parameters, θ∈Θ_k, k:(B,Θ_k)→B, where k(⋅,θ) is an endomorphism of the image function domain for each θ. Therefore, k(⋅;θ)∘s:D→B returns a template image with a unique spacing encoded in θ. 
     Font and Appearance 
     The canonical word image obtained by skeletonization and spacing of the kerning function is then subjected to purely local transformations, namely, the font function  106  and the appearance function  108 , which produces the rendered text on a flat surface. 
     Font is the intrinsic shape of rendered text. For purposes herein, the appearance function  108  models lighting variations, shadows, and imaging artifacts such as blur. Appearance is an extrinsic feature while font is an intrinsic feature. The font is viewed as acting locally, widening curves, or elongating the skeleton characters at each point in the domain I. The f function encodes local deformations mapping the skeleton text to the font shape. 
     For purposes herein, “appearance” refers to the effects of extrinsic features of the text location and environment. This includes background appearance and clutter, cast shadows, image noise, rendering artifacts, and the like. However, for the compositional model described herein, appearance does not include geometric context or shape of the rendering surface in the model. 
     Font and appearance are independent of the word chosen, and the free parameters of font and appearance are modeled with a hidden variable domain Θ_f. Font can be modeled by a function f:(B(I),Θ_f)→B(I). To reflect the fact that naively disentangling appearance and font is ill-posed, the appearance function a is modeled on the parameter space Θf. 
     Geometric Distortion 
     The geometric distortion function  110  models distortion of text that arises primarily due to perspective distortion of planar geometry. The vast majority of text is rendered linearly onto planar surfaces, and so a homographic model is adapted for the geometric distortion function  110 . Although homogeneous linear models are discussed herein, other models of geometric distortion may be used. 
     The rendering domain may be fixed to R=[0, 1, . . . , 31]×[0, 1, . . . , 255] to enable batching for training and to fit the dimensions of the input to the model. Note that the rendered image domain I may or may not be aligned with R. The geometric distortion acts on I, so it may be modelled as parametrically pulling back B(I) to B(g(I)): g(H)*(a∘f∘k∘s)(w)|x=[(a∘f∘k∘s)(w)]|H(x)—where H is a homography, and subscript indicates evaluation of the function at a point. 
     Although STNs (spatial transformer networks) have been used to model the geometries of the scenes in images, STNs have not previously been used to model geometric distortion due to mismatches between the prediction space of a text detector and the geometry of the scene. Furthermore, by specifically employing the relatively new IC-STN (inverse-compositional STN) that recursively computes distortion, larger distortions can be handled. 
     Complete Compositional Model 
     The complete compositional model models the text construction process for a given word using the five steps discussed above. The final image function is given by
 
 i =( g ( H )*(( a∘f )(⋅,θ_ f ) ∘k (⋅,θ_ k )∘ s ))( w ).  (1)
 
     Having set forth a compositional model for text construction, the next section describes how the compositional model can be inverted. 
     Inverse Inference 
     Equation (1) can be inverted with an entangled combination of a CNN (convolutional neural network) and an RNN (recurrent NN). Note that with this approach, it can be difficult to quantify or theoretically conclude which components model which aspects of the text. 
     Next, four of the five functions are decomposed into distinct functions and a neural architecture design is set forth for estimating these inversion functions. 
     Geometric Rectification 
     Rectification can be implemented in a number of ways, including extracting and aligning keypoints or regions, unsupervised aligning to a corpus, and image-to-image alignment. However, because the text recognition problem involves a particularly constrained scenario, it is difficult to establish a canonical pose for keypoints without formulating the skeletonization process to account for keypoint variation, which would introduce additional free parameters and corresponding complexity. Image-to-image matching based on, for example, mutual information, is also difficult at this stage because the appearance features have not yet been normalized. Finally, alignment to a mean image or a corpus of images involves painstaking congealing optimization, which can make the solution slow and which is difficult to train end-to-end. Thus, the region-based alignment methods and an STN are practical for this work. 
     Incidentally, the affine patch-based MSER (maximally stable extremal regions) was compared with an STN solution. Specifically, the MSER was estimated within a contextual region around each word detection, and then rectified to a minimal enclosing oriented bounding box for the MSER. It was found that the STN provides a more reliable rectification module (test results are discussed below). 
     Geometric distortion is modeled by estimating a homography directly from the input pattern and from coordinates of a given input region of an image. In the general case, there is no additional observation beyond the word pattern supervision, so the parameters can be trained to the rectifier in a weakly supervised manner as outlined below. Thus, an inverse-composition STN (IC-STN) is a sound fit. Unlike prior approaches, this form of geometric modeling allows for recapturing missing parts of the image and does not depend on fiducial point estimates. Note that an IC-STN can be used to both correct distortion to text, correct noise in detections, and to canonicalize disparate geometric regions of training and testing data for cropping. 
       FIG. 3  shows interaction between a geometric rectification module  120  and a convolutional encoding module  122 . As shown in  FIG. 3 , the input to the geometric rectification module is an input image  124  and the coordinates of a text region  126 . As output, the rectification module  120  outputs to the convolutional encoding module  122  a rectified crop  128  of the input image  124 . The convolutional encoding module provides outputs  130  that can be decoded, as discussed below. 
     Appearance Featurization 
     As mentioned above, disentangling a∘f without additional supervision is ill-posed. The a and f functions may be grouped together and instead estimate (a∘f)−1. This can be done in many ways, but for most tasks, deep CNN features provide optimal performance. Thus, the font and appearance may be modeled with a state-of-the-art CNN such as Resnet-6.  FIG. 3  depicts a CNN module. 
     Kerning Module 
     While the receptive field of each of the kernels in the relevant CNN feature-layers contain an appreciable context, modeling kerning with the feature CNN likely leads to entangled modeling and thus reduced performance on this task. 
     With that in mind, there are three main challenges to inverse inference of the kerning parameters: 
     (1) spacing may require a variable amount of context within each image (due to distortion), 
     (2) spacing may require a variable amount of context for different images, and 
     (3) features may not reflect the original spacing. 
     Item (1) may be addressed by the use of the geometric module in the front of whichever neural network is employed. Thus, the features returned from the Resnet-6 module, for instance, are in a canonical frame of reference. Item (2) is resolved by establishing an auxiliary loss function that encourages locality of the features. Item (3) may be resolved by explicitly modeling the context around each spacing element using a bidirectional long short-term memory (LSTM). An LSTM may be preferred because it robustly models contextual relationships in ordered data. The auxiliary mean squared error (MSE) loss, explained in more detail below, may be used to impose consistency on the output of the kerning-LSTM (kLSTM). Implementation of the LSTM is discussed below. 
     To employ an LSTM on the encoded text features, the text features are segmented along the x-axis with a sliding window having a width of 2 and a stride of 1. The windowed features (collections of feature vectors in the sliding moving window) are ordered from left to right in increasing x order. This corresponds to the time index. The LSTM takes the vectorized features as input and computes a new internal state and then operates on the next window. After a pass over all of the feature windows, the bidirectional LSTM operates in reverse order and takes as input the output of a previous cell. Finally, the reverse cells output next feature vectors that are concatenated to form a new feature encoding. See  FIG. 4 , discussed below. The output is fed into the recognition and reconstruction modules. 
       FIG. 4  shows the kLSTM module  150 . The inputs  152  to the kLSTM module  150  come from the outputs of the convolutional encoder module  122 . The inputs  152  are windowed as described above and the LSTM is propagated forwards and backwards. The outputs  154  of the bidirectional kLSTM are concatenated and fed into the convolutional decoder  156  and the MSE loss  158  is computed against the ground truth skeleton  160 . 
     Skeleton Reconstruction 
     To rebuild the skeleton from the rectified and de-kerned features, a deconvolutional architecture of three residual layers is employed, with spatial upsampling between each set of convolve, convolve, and add layers. This is designed to mirror the features in the encoding stage. Finally, a convolutional layer predicts the skeleton template. The ground truth template is compared with the prediction elementwise and the MSE loss is computed as outlined below. 
     Templates for the given transcription of ground truth word images are rendered with the following process. A font is chosen for the template; this will create a slight bias for the reconstruction task, so a standard font might be preferred. In one implementation, the Sans font in GNU GIMP (GNU image manipulation program) is used for template images. Given a font, the skeleton S k  of the character images is computed for each character c k . Finally, the function T k :I→R:x→exp{−d(x,S k ) 2 /2σ 2 } is computed over a fixed image grid for each k. The term σ is fixed to 1 pixel. Since there is some variation in the ligature between characters, as well as ascenders and descenders for each character, to fix kerning, all characters may be resized to consume the same amount of space in each template image (see  FIG. 5 ). 
       FIG. 5  shows example text images  170  and template skeletons  172 . Using a template skeleton  172  provides several advantages for inverse inference on the components above. Registration can typically have trivial local minima without templates (normally MSE has trivial minima for joint registration, and a correlation-like loss must be employed). In addition, the kerning width is in terms of a fixed unit (the template kerning width). Finally, since the template skeletons are in terms of a fixed and clear font, legibility of the reconstruction implies that the features contain the information required to encode the characters for decoding. This provides another point of inspection for understanding the neural network. 
     Text Recognition 
     The text recognition module for the full compositional model takes its input from the kerning module. A residual layer and a convolutional layer are applied to adapt to the recognition problem. Finally, an LSTM is used to convert the convolutional spatial features into character probabilities. Specifically, a bidirectional LSTM is used without peephole connections, clipping, or projection with stride 8 (1) in pixel space (feature space). The bidirectional LSTM enables contextual information to propagate along with the hidden state from a symmetric horizontal window along the columnar features. 
     The next section discusses the functions used to drive the learning of the parameters of the inverse compositional modules. 
     Loss Functions 
     Two objectives are provided for the network: recognition and reconstruction. What follows is a review of the CTC recognition loss function and the reconstruction loss. 
     The CTC loss function operates on the output of the recognition LSTM. The CTC loss function produces a sequence of score vectors, or probability mass functions, across the codec labels. The conditional probability of the label given the prediction is the sum over all sequences leading to the label, given the mapping function B: 
     
       
         
           
             
               p 
               ⁡ 
               
                 ( 
                 
                   l 
                   ❘ 
                   y 
                 
                 ) 
               
             
             = 
             
               
                 ∑ 
                 
                   
                     π 
                     : 
                     
                       B 
                       ⁡ 
                       
                         ( 
                         π 
                         ) 
                       
                     
                   
                   = 
                   l 
                 
               
               ⁢ 
               
                 
                   p 
                   ⁡ 
                   
                     ( 
                     
                       π 
                       ❘ 
                       y 
                     
                     ) 
                   
                 
                 . 
               
             
           
         
       
     
     This probability can be computed using dynamic programming. 
     CTC loss allows the inverted transcription process to have many-to-one correspondences. By marginalizing over the paths through the sequence prediction matrix that output the target, using dynamic programming, the probability of the target sequence under the model parameters is obtained. 
     Mean Squared-Error Reconstruction Loss 
     The MSE reconstruction loss takes as input (i) the output of the decoder, (ii) the predicted template S:I→R, and (iii) the ground truth template image T:I→R associated with the given word during training. See  FIG. 4 . The objective function for this task is 
                       L   mse     ⁡     (     S   ,   T     )       =       1        I          ⁢              S   -   T              L   2     ⁡     (     I   ;   R     )       2     .               (   2   )               
Emperical Studies
 
     The performance of trained recognition embodiments described above were studied from the recognition standpoint and the end-to-end standpoint. Because the recognition stage is the main thrust of the embodiments described herein, the detector design is not evaluated. A faster RCNN-based multiscale detector was used. For comparison with previous end-to-end systems, the detector obtains recall of 90% and an F1-score of 92% on the ICDAR (International Conference on Document Analysis and Recognition) Focused test set. 
     In the experiments discussed below, the detector model was frozen and was not used to train the recognizer. 
     Ablation Experiments 
     The embodiments described herein feature several new components, such as the template prediction loss and IC-STN. Therefore, the performance of the model was tested under several architectures and loss balancing parameters. This validated the significance of each component. 
     End-to-End Evaluation 
     For an experiment to evaluate end-to-end performance, the base learning rate was set to 1×10−4 using exponential decay of the learning rate with a factor of 0.9 every 5000 iterations. The ADAM optimizer was used with β 1 =0.5. The MLT Training dataset and the ICDAR Focused Training dataset were used for training the recognizer. Data augmentation was performed by the method described below, randomly perturbing the input coordinates, with a fixed σ p =0.025. 
       FIG. 6  shows end-to-end F-scores  190  for the ICDAR 2013 Focused test set. The results of the ablation experiment show the significance of template reconstruction loss and spatial transformer and LTSM-based kerning modules. The models were trained under identical settings. 
     Robustness to Coordinate Perturbation 
     Although data augmentation is often used for training text recognition algorithms, the robustness of such algorithms in the face of perspective/geometric distortion is not known to have been studied. To that end, presented next is a comprehensive evaluation of the robustness of the compositional model under several ablations. 
     For each input coordinate pair X i  in each bounding box B j , a vector η i ˜N(0,Σ p ) is drawn. Then, for each box B j  a translation vector t j ˜N(0,Σ t ) is drawn. Finally, for evaluation, the corrupted coordinates {circumflex over (X)} i   j =X i +n i +t j  replace the original coordinates in the input to the algorithm. Also, Σ t , Σ p  are chosen to be a diagonal matrix Σ t =σ t Σ,Σ p =σ p Σ, with Σ 11 ∝w, Σ 22 ∝h, and with the constants of proportionality σ t =σ p . 
       FIG. 7  shows a graph  210  of results for different models. Specifically, ten experiments were performed at each noise setting and the mean and standard deviation of the accuracy were computed for each model on each perturbed test set. Graph  190  shows that the full model improves significantly on the baseline, without-STN, and without-SSD models for higher amounts of perturbation. This is further borne out in comparative end-to-end experiments. 
     As seen in  FIG. 7 , the accuracy of the various models from the ablation study are shown for various degrees of perturbation from the ground truth coordinates. In the graph  190 , there is a large gap between models with and without an STN as the perturbation level rises. 
     Robustness to Kerning Variations 
     To evaluate the quality of kerning modeling, the ICDAR-13 Text Segmentation dataset was used. This dataset features character segments for each character in the word images. This provides an initial value for the spacing by computing the horizontal distance between the center of the enclosing bounding box of each segment. A sequence of images was then synthesized by re-parameterizing the x-axis of the image so that the number of non-character columns between each character is increased by a factor of 1+0.1*k*H, where H is the height of the cropped word image. 
     The next section discusses qualitative anecdotal evidence that the kLSTM module in the trained neural network  80  improves the quality of kerning modeling. 
     Comparative Evaluation 
     The full model was compared with state-of-the-art text recognition algorithms and end-to-end algorithms using the ICDAR Focused 2013 and Incidental 2015 datasets. The same trained network was used for all experiments, with no additional finetuning, limited to the training data provided for the end-to-end task, with the MLT 2017 training data included. For this experiment, the learning rate was initialized higher, to 5×10−4, since it was observed that the full model is more robust to swings in the early training stages. Additionally, a stagewise decay was employed by a factor of 0.1 after 10, 20, and 35 of the 50 epochs. A higher level of perturbation was incorporated, with σ p =σ t =0.05. 
       FIG. 8  shows examples  230  anecdotally demonstrating that a bidirectional LSTM can model the kerning process. The kerning of each input image increases from left to right. The top four rows are the input images. The middle four rows are the predicted templates for a model without the LSTM. The bottom four rows are the predicted templates with the LSTM. While there are still failure cases for the full model, there is significant improvement over the model that lacks the LSTM. 
     Recognition 
     The trained neural network was tested on the ICDAR Focused 2013 (IC-13) and ICDAR Incidental (IC-IST) for this task.  FIG. 9  shows comparative results on the recognition tasks. Specifically,  FIG. 9  shows accuracy results  240  as tested with the 90K dictionary for the IC-13 and IC-IST recognition tasks. 
     End-to-End 
     Even though the system is not trained end-to-end, it exhibits the best performance on the minimally assisted Generic end-to-end task. The modest improvements for the assisted tasks may be due to detector performance. However, since this work showcases the second component in the system (the neural network  80 ), this was not studied in depth.  FIG. 10  shows end-to-end comparative performance results  250  for top OCR algorithms. The ‘G’, ‘W’, and ‘S’ column headings indicate the lexicon used for each task (Generic, Weak, and Strong, respectively), and the column shows the F-score of the respective algorithm. 
       FIG. 11  shows example input images  270 , rectified images  272 , and predicted templates  274 . The first three rows show input image examples that work well. The fourth row shows shortcomings of the featurization process, which indicates that explicit background modeling may lead to improvements. In the last row, it can be seen that out-of-model geometric transformations may also cause problems. 
     Summary 
     Optical character recognition (OCR) systems are known to be constrained by the initial detector stage. Previous work dealt with this by coupling the recognizer and detector closer together and aligning the input of the second stage to the first. Embodiments described herein offer an alternative strategy that involves training the recognizer to be capable of recovering missing text and aligning the input patterns from the first stage for itself. A full formulation of this based on a textual compositional model has been explained herein. Ablation studies discussed above have shown the relative importance of the components of the model, and comparative studies place the work disclosed relative to the growing body of text recognition and end-to-end OCR systems. 
     Conclusion 
       FIG. 12  shows details of a computing device  300  on which embodiments described above may be implemented. The technical disclosures herein will suffice for programmers to write software, and/or configure reconfigurable processing hardware (e.g., field-programmable gate arrays (FPGAs)), and/or design application-specific integrated circuits (ASICs), etc., to run on the computing device or host  300  (possibly via cloud APIs) to implement the embodiments described herein. 
     The computing device or host  300  may have one or more displays  322 , a network interface  324  (or several), as well as storage hardware  326  and processing hardware  328 , which may be a combination of any one or more of: central processing units, graphics processing units, analog-to-digital converters, bus chips, FPGAs, ASICs, Application-specific Standard Products (ASSPs), or Complex Programmable Logic Devices (CPLDs), etc. The storage hardware  326  may be any combination of magnetic storage, static memory, volatile memory, non-volatile memory, optically or magnetically readable matter, etc. The meaning of the term “storage”, as used herein does not refer to signals or energy per se, but rather refers to physical apparatuses and states of matter. The hardware elements of the computing device or host  300  may cooperate in ways well understood in the art of machine computing. In addition, input devices may be integrated with or in communication with the computing device or host  300 . The computing device or host  300  may have any form-factor or may be used in any type of encompassing device. The computing device or host  300  may be in the form of a handheld device such as a smartphone, a tablet computer, a gaming device, a server, a rack-mounted or backplaned computer-on-a-board, a system-on-a-chip, or others. 
     Embodiments and features discussed above can be realized in the form of information stored in volatile or non-volatile computer or device readable media. This is deemed to include at least media such as optical storage (e.g., compact-disk read-only memory (CD-ROM)), magnetic media, flash read-only memory (ROM), or any current or future means of storing digital information. The stored information can be in the form of machine executable instructions (e.g., compiled executable binary code), source code, bytecode, or any other information that can be used to enable or configure computing devices to perform the various embodiments discussed above. This is also deemed to include at least volatile memory such as random-access memory (RAM) and/or virtual memory storing information such as central processing unit (CPU) instructions during execution of a program carrying out an embodiment, as well as non-volatile media storing information that allows a program or executable to be loaded and executed. The embodiments and features can be performed on any type of computing device, including portable devices, workstations, servers, mobile wireless devices, and so on.