Patent Description:
Optical character recognition (OCR) generally involves translating images of text into an encoding representing the actual text characters. OCR techniques for text based on a Latin script alphabet are widely available and provide very high success rates. Handwritten text generally presents different challenges for recognition than typewritten text.

Known in the art are handwriting recognition techniques that are based on Recurrent Neural Networks (RNNs) and their extensions such as Long-Short-Term-Memory (LSTM) networks, Hidden Markov Models (HMMs), and combinations thereof [<NPL>;<NPL>; <NPL>; <NPL>; <NPL>; and <NPL>].

Another method, published by Almazán et al. [<NPL>], encodes an input word image as Fisher Vectors (FV), which can be viewed as an aggregation of the gradients of a Gaussian Mixture Model (GMM) over low-level descriptors. It then trains a set of linear Support Vector Machine (SVM) classifiers, one per each binary attribute contained in a set of word properties. Canonical Correlation Analysis (CCA) is used to link the vector of predicted attributes and the binary attributes vector generated from the actual word.

An additional method, published by Jaderberg et al. [<NPL>], uses convolutional neural networks (CNNs) trained on synthetic data for Scene Text Recognition.

<NPL>, discloses a neural network, which integrates feature extraction, sequence modeling and transcription into a unified framework. The network consists of convolutional layers, recurrent layers, and a transcription layer.

<CIT> discloses a neural network-based optical character recognition (OCR) system that parses an incoming image into scaled sub-images, with each sub-image containing a character(s), locates the center of that character(s) through use of a sliding window and a positioning neural network, routes that sub-image to a character classifier, and produces an output label string containing the recognized character(s). A character rejection decision is formed from the output activation values of the neural classifier.

According to the present invention there is provided a method of recognizing image content. The method comprises: applying a neural network to the image. The neural network comprising an input layer for receiving the image, a plurality of hidden layers for processing the image, and an output layer for generating output pertaining to an estimated image content based on outputs of the hidden layers. The method further comprises applying a neural network branch to an output of at least one of the hidden layers. The neural network branch being independent of the neural network and having an output layer for generating output pertaining to an estimated error level of the estimate. The method further comprises generating a combined output indicative of the estimated image content and the estimated error level.

According to some embodiments of the invention the neural network branch comprises at least one recurrent layer generating a plurality of output values. According to some embodiments of the invention the at least one recurrent neural layer is a Long Short Term Memory (LSTM) layer. According to some embodiments of the invention the LSTM layer is a bi-directional layer.

According to an aspect of some embodiments of the present invention there is provided a method of recognizing image content. The method comprises: applying a neural network to the image. The neural network comprising an input layer for receiving the image, a plurality of hidden layers for processing the image, and an output layer for generating output pertaining to an estimated image content based on outputs of the hidden layers. The method further comprises calculating a confidence score by obtaining output probabilities that are generated by the neural network to two most probable recognitions, and assigning a confidence score that correlates with a ratio or difference between the two probabilities.

According to some embodiments of the invention the method comprises summing or averaging the plurality of output values or projections thereof, thereby providing the estimated error level.

According to some embodiments of the invention the neural network comprises a convolutional neural subnetwork generating a set of feature vectors, each corresponding to an application of the convolutional subnetwork different window within the image.

According to some embodiments of the invention the neural network comprises a recurrent neural subnetwork receiving the set of feature vectors from the convolutional neural subnetwork, and wherein the output of the least one of the hidden layers is an output of at least one layer of the recurrent neural subnetwork. According to some embodiments of the invention the recurrent neural subnetwork comprises LSTM layers. According to some embodiments of the invention at least one of the LSTM layers is a bi-directional layer.

According to some embodiments of the invention the recurrent neural subnetwork generates a sequence of vectors, and the method comprises projecting a dimension of each vector of the sequence onto a predetermined dimension corresponding to a length of an alphabet, to provide a sequence of probability distributions over the alphabet, each probability distribution corresponding to a region within the image.

According to some embodiments of the invention the neural network comprises a Connectionist temporal classification (CTC) layer receiving the sequence of probability distributions and generating the estimate.

According to some embodiments of the invention the method comprises transforming a height of the image to a predetermined height, prior to the application of the neural network.

According to some embodiments of the invention the image is an image of handwritten symbols.

According to some embodiments of the invention the image is an image of machine printed symbols.

According to some embodiments of the invention the image comprises at least one region containing handwritten symbols and at least one region containing machine printed symbols.

According to some embodiments of the invention the image is an image of a bank check.

According to some embodiments of the invention the bank check comprises a plurality of bank check fields, and the method is repeated separately for at least two of the bank check fields.

According to an aspect of some embodiments of the present invention there is provided a computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive an input image and to execute the method as delineated above and optionally and preferably as further detailed below.

According to the present invention there is provided a system for recognizing image content. The system comprises an imaging system for imaging an object to provide an image thereof. The system also comprises an image processor configured for: receiving the image; applying a neural network to the image, the neural network comprising an input layer for receiving the image, a plurality of hidden layers for processing the image, and an output layer for generating output pertaining to an estimated image content based on outputs of the hidden layers; applying a neural network branch to an output of at least one of the hidden layers, the neural network branch being independent of the neural network and having an output layer for generating output pertaining to an estimated error level of the estimate; and generating a combined output indicative of the estimated image content and the estimated error level.

According to the present invention there is provided a system for recognizing image content. The system comprises a network interface for receiving an image over a communication network. The system also comprises an image processor configured for: applying a neural network to the image, the neural network comprising an input layer for receiving the image, a plurality of hidden layers for processing the image, and an output layer for generating output pertaining to an estimated image content based on outputs of the hidden layers; applying a neural network branch to an output of at least one of the hidden layers, the neural network branch being independent of the neural network and having an output layer for generating output pertaining to an estimated error level of the estimate; and generating a combined output indicative of the estimated image content and the estimated error level.

The present invention, in some embodiments thereof, relates to image processing and, more particularly, but not exclusively, to image content recognition.

The invention is capable of other embodiments or of being practiced or carried out in various ways, which are inside the scope of the appended claims.

<FIG> is a flowchart diagram of a method suitable for recognizing image content, according to various exemplary embodiments of the present invention. It is to be understood that, unless otherwise defined, the operations described hereinbelow can be executed either contemporaneously or sequentially in many combinations or orders of execution. Specifically, the ordering of the flowchart diagrams is not to be considered as limiting. For example, two or more operations, appearing in the following description or in the flowchart diagrams in a particular order, can be executed in a different order (e.g., a reverse order) or substantially contemporaneously. Additionally, several operations described below are optional and may not be executed.

At least part of the operations described herein can be can be implemented by a data processing system, e.g., a dedicated circuitry or a general purpose computer, configured for receiving data and executing the operations described below. At least part of the operations can be implemented by a cloud-computing facility at a remote location.

Computer programs implementing the method of the present embodiments can commonly be distributed to users by a communication network or on a distribution medium such as, but not limited to, a floppy disk, a CD-ROM, a flash memory device and a portable hard drive. From the communication network or distribution medium, the computer programs can be copied to a hard disk or a similar intermediate storage medium. The computer programs can be run by loading the code instructions either from their distribution medium or their intermediate storage medium into the execution memory of the computer, configuring the computer to act in accordance with the method of this invention. During operation, the computer can store in a memory data structures or values obtained by intermediate calculations and pulls these data structures or values for use in subsequent operation. All these operations are well-known to those skilled in the art of computer systems.

Processing operations described herein may be performed by means of processer circuit, such as a DSP, microcontroller, FPGA, ASIC, etc., or any other conventional and/or dedicated computing system.

The method of the present embodiments can be embodied in many forms. For example, it can be embodied in on a tangible medium such as a computer for performing the method operations. It can be embodied on a computer readable medium, comprising computer readable instructions for carrying out the method operations. In can also be embodied in electronic device having digital computer capabilities arranged to run the computer program on the tangible medium or execute the instruction on a computer readable medium.

Referring now to <FIG>, the method begins at <NUM> and optionally and preferably continues to <NUM> at which an image is received. The image typically contains symbols belonging to an alphabet, including, without limitation, characters, accent symbols, digits and/or punctuation symbols. In some embodiments of the present invention, the image contains handwritten symbols, in which case the alphabet is a set of handwritten symbols. In some embodiments of the present invention, the image contains printed symbols, in which case the alphabet is a set of printed symbols. Also contemplated are images containing both handwritten and printed symbols. A representative example of such an image is an image of a bank check having a plurality of fields some of which may include handwritten symbols (e.g., amounts in digits and words, etc.), and some of which may include printed symbols (e.g., account No., name of account owner, etc.).

The image is preferably a digital image and can be received from an external source, such as a storage device storing the image in a computer-readable form, and/or be transmitted to a data processor executing the method operations over a communication network, such as, but not limited to, the internet. In some embodiments of the present invention the method comprises capturing the image using an imaging system, e.g., a scanner or a digital camera.

The method optionally and preferably continues to <NUM> at which the input image is resized. This operation is particularly useful when the image is received as input from an external source. The resizing can include stretching or shrinking along any of the axes of the image to a predetermined width, a predetermined length and/or a predetermined diagonal, as known in the art. The resize along each axes can be independent. For example, the method can transform only the height of the image to a predetermined height, without changing the width, or vice versa, or transform both height and width according to different transformation scenarios.

At <NUM>, a neural network is applied to the image. A representative example of a neural network <NUM> suitable for the present embodiments is illustrated in <FIG>. Neural network <NUM> typically comprises an input layer <NUM> for receiving the image <NUM>, a plurality of hidden layers <NUM> for processing the image, and an output layer <NUM> for generating output pertaining to an estimated image content based on outputs of hidden layers <NUM>. The neural network is optionally and preferably pre-trained to estimate the image content according to a specific alphabet, a subset of which is contained in the image to which the neural network is designed to be applied.

In some embodiments of the present invention the hidden layers <NUM> of neural network <NUM> comprise a convolutional neural subnetwork <NUM> (see <FIG>) generating a set <NUM> of feature vectors, each corresponding to an application of convolutional subnetwork <NUM> to a different window within the image <NUM>. The set <NUM> of feature vectors can thus represent a window moving over the image from one side of the image to the other. Optionally, but not necessarily, convolutional neural subnetwork <NUM> is a fully convolutional neural subnetwork.

Unlike fully-connected networks or subnetworks which associate a single value with each neuron of the network or subnetwork, convolutional neural networks or subnetworks operate by associating an array of values with each neuron. Conceptually, this array can be thought of as a small patch of the input image. The transformation of a neuron value for the subsequent layer is generalized from multiplication to convolution. This implies that the connection strengths are convolution kernels rather than scalar values. These more complex transformations involve more complex neural network matrices. Thus, while a matrix in a fully-connected network or subnetwork comprises an array of number values, in a convolutional neural network or subnetwork, each matrix entry is a patch of pixels.

Neural network <NUM> can optionally and preferably comprise a recurrent neural subnetwork <NUM> receiving set <NUM> of feature vectors from convolutional neural subnetwork <NUM>. Recurrent neural subnetwork <NUM> optionally and preferably employs one or more long short-term memory (LSTM) layers, more preferably one or more bidirectional LSTM layers.

Recurrent neural networks or subnetworks are designed to accept sequences of varying lengths as input. The same set of weights is used in the processing of each sequence element. Such subnetworks are constructed in a layered manner, such that every neuron of layer accepts as inputs the activations of all neurons from the previous layer, and, an addition, there are lateral connections with the activations induced at the previous step in the sequence. Bidirectional neural networks or subnetworks are employ layers going forward and layers going backwards. These two types of layers exist in parallel and the activations of both, concatenated, serve as the input to the next layer.

Long Short-Term Memory (LSTM) layers employ memory cells' in lieu of simple activations. Access to the memory cells is controlled by multiplicative factors that are called gating. At each input state, gates are used in order to decide the portion of the new input that is to be written to the memory cell, the portion of the current content of the memory cell that is to be forgotten, and the portion of the content that is to be outputted. For example, if the output gate is closed (a value of <NUM>), the neurons connected to the current neuron receive a value of <NUM>. If the output gate is partly open at a gate value of <NUM>, the neuron can output half of the current value of the stored memory.

Recurrent neural subnetwork <NUM> typically generates a sequence <NUM> of vectors. In these embodiments, the method optionally and preferably projects the dimension of each vector of sequence <NUM> onto a predetermined dimension corresponding to a length of the alphabet, thus providing a sequence of probability distributions over the alphabet, wherein each probability distribution corresponds to a region within the image. In some embodiments of the present invention output layer <NUM> comprises a Connectionist Temporal Classification (CTC) layer [<NPL>] that receives the sequence of probability distributions and generates the estimate of the image content according to the alphabet.

The CTC layer can be used both for training the network and for using the network to recognize the content of the image. For the former, a loss function is used. For the latter, the CTC decoding scheme is optionally and preferably employed to obtain the output. This can be done using greedy decoding to obtain the highest probability decoding, or using a beam-search scheme to obtain the n highest probability decoding values.

The method optionally and preferably proceeds to <NUM> at which a neural network branch <NUM> is applied to an output of at least one of hidden layers <NUM>. Network branch <NUM> is applied to an output of any of hidden layers <NUM>. One or more of the recurrent neural layer(s) of branch <NUM> can be, for example, an LSTM layer, more preferably a bi-directional LSTM layer. Typically, branch <NUM> is applied to the output of one of the layers in recurrent neural subnetwork <NUM>, e.g., the output of one of the LSTM layer, when such layers are employed. Preferably, but not necessarily, network branch <NUM> is applied to the output of the first layer in the recurrent neural subnetwork <NUM>, namely the output of the layer that is fed by the set <NUM> of feature vectors generated by convolutional neural subnetwork <NUM>.

Neural network branch <NUM> is optionally and preferably independent of neural network <NUM>. Branch <NUM> has an output layer <NUM> generating output pertaining to an estimated error level of the estimate generated by layer <NUM>. Optionally and preferably, neural network branch <NUM> comprises one or more recurrent layers <NUM> generating a plurality of output values. The method optionally and preferably sums the output values to provide the estimated error level. Typically, the output values provided by recurrent layers <NUM> are vector values. In these embodiments, each of the vector values provided by recurrent layers <NUM> is projected to a scalar, thereby providing a plurality of scalars that can optionally and preferably be subsequently summed or averaged, so that the error level generated by output layer <NUM> is also a scalar.

The method proceeds to <NUM> at which a combined output indicative of the estimated image content and the estimated error level is generated. The combined output can be displayed on a display device, printed on a print medium and/or transmitted to a remote computer over a communication network.

<FIG> is a schematic illustration of a client computer <NUM> having a hardware processor <NUM>, which typically comprises an input/output (I/O) circuit <NUM>, a hardware central processing unit (CPU) <NUM> (e.g., a hardware microprocessor), and a hardware memory <NUM> which typically includes both volatile memory and non-volatile memory. CPU <NUM> is in communication with I/O circuit <NUM> and memory <NUM>. Client computer <NUM> preferably comprises a graphical user interface (GUI) <NUM> in communication with processor <NUM>. I/O circuit <NUM> preferably communicates information in appropriately structured form to and from GUI <NUM>. Also shown is a server computer <NUM> which can similarly include a hardware processor <NUM>, an I/O circuit <NUM>, a hardware CPU <NUM>, a hardware memory <NUM>. I/O circuits <NUM> and <NUM> of client <NUM> and server <NUM> computers can operate as transceivers that communicate information with each other via a wired or wireless communication. For example, client <NUM> and server <NUM> computers can communicate via a network <NUM>, such as a local area network (LAN), a wide area network (WAN) or the Internet. Server computer <NUM> can be in some embodiments be a part of a cloud computing resource of a cloud computing facility in communication with client computer <NUM> over the network <NUM>. Further shown, is an imaging device <NUM> such as a camera or a scanner that is associated with client computer <NUM>.

GUI <NUM> and processor <NUM> can be integrated together within the same housing or they can be separate units communicating with each other. Similarly, imaging device <NUM> and processor <NUM> can be integrated together within the same housing or they can be separate units communicating with each other.

GUI <NUM> can optionally and preferably be part of a system including a dedicated CPU and I/O circuits (not shown) to allow GUI <NUM> to communicate with processor <NUM>. Processor <NUM> issues to GUI <NUM> graphical and textual output generated by CPU <NUM>. Processor <NUM> also receives from GUI <NUM> signals pertaining to control commands generated by GUI <NUM> in response to user input. GUI <NUM> can be of any type known in the art, such as, but not limited to, a keyboard and a display, a touch screen, and the like. In preferred embodiments, GUI <NUM> is a GUI of a mobile device such as a smartphone, a tablet, a smartwatch and the like. When GUI <NUM> is a GUI of a mobile device, processor <NUM>, the CPU circuit of the mobile device can serve as processor <NUM> and can execute the code instructions described herein.

Client <NUM> and server <NUM> computers can further comprise one or more computer-readable storage media <NUM>, <NUM>, respectively. Media <NUM> and <NUM> are preferably non-transitory storage media storing computer code instructions as further detailed herein, and processors <NUM> and <NUM> execute these code instructions. The code instructions can be run by loading the respective code instructions into the respective execution memories <NUM> and <NUM> of the respective processors <NUM> and <NUM>.

Each of storage media <NUM> and <NUM> can store program instructions which, when read by the respective processor, cause the processor to receive an input image and to execute the method as described herein. In some embodiments of the present invention, an input image is generated by imaging device <NUM> and is transmitted to processor <NUM> by means of I/O circuit <NUM>. Processor <NUM> estimate the content of the image and the error of the estimate as further detailed hereinabove and display the combined output, for example, on GUI <NUM>. Alternatively, processor <NUM> can transmit the image over network <NUM> to server computer <NUM>. Computer <NUM> receives the image, estimate the content of the image and the error of the estimate as further detailed hereinabove and transmits the combined output back to computer <NUM> over network <NUM>. Computer <NUM> receives the combined output and displays it on GUI <NUM>.

This Example described an Optical Character Recognition (OCR) process that was applied to handwritten and printed bank checks. The applied process included the method according to some embodiments of the present invention wherein a neural network was applied to the image to estimate the image content based on outputs of the hidden layers, and a neural network branch was applied to an output of one of the hidden layers to estimate the error level of the image content estimate.

Three different datasets containing real-world images of three different fields from bank checks have been used in this example. The datasets are named herein Account, CAR and LAR, where Account corresponds to a numeric field of account numbers, CAR corresponds to the check dollar amount, written as digits, and LAR corresponds to the check dollar amount, written in words. <FIG> shows example images from the LAR dataset (<FIG>, showing "seventy five and <NUM>"), the CAR dataset (<FIG>, showing "<NUM>") and the Account dataset (<FIG> showing "<NUM>").

For each image, the datasets contained a reference transcription of its content.

The network architecture was as follows. For the recognition part of the network, an end-to-end trainable network was used. The network took, for training, only the image and its transcription. In this example the image height was normalized to <NUM> pixels, and the image width and the transcription's length were of arbitrary length. No alignment information was used to train the network.

The input image first goes through a VGG-style convolutional neural network, composed of convolutional layers, batch normalization layers, and max-pooling layers. In this example, ReLU was used as an activation function, but other activation functions are also contemplated. In this example, the pooling layers decreased the image height from <NUM> to <NUM>, and also decreased the image width by a factor of <NUM>. This provided <NUM> max-pooling layers with dimensions <NUM>×<NUM> and <NUM> max-pooling layers with dimensions <NUM>×<NUM>.

Three convolutional layers were employed between each two pooling operations. In this example, the shape of the convolutional layers was <NUM>×<NUM> and padding <NUM>. A batch normalization layer and an activation function was employed after each convolutional layer.

The output of the convolutional stage was a sequence of feature vectors, where every feature vector was the result of applying the convolutional network to a window in the original image. The feature vectors thus corresponded to looking at the image with a sliding window from left to right.

The obtained sequence of feature vectors was fed as input to a recurrent subnetwork which in this example included two bi-directional LSTM layers. Both bi-LSTM layers were realized as a concatenation of two LSTM layers with <NUM> units. After the convolutional and LSTM layers, a variable-length sequence of vectors of length <NUM> was obtained. A linear projection was then applied from dimension <NUM> to the length of the desired alphabet, to obtain a sequence of probability distributions over the alphabet, each corresponding to a rectangular receptive field in the original image.

The final transcription was given by applying the Connectionist Temporal Classification (CTC) layer to the resulting sequence. This layer defined a probability for an output sequence given the probability vectors that come from the previous layer. The output sequence was then defined to be the sequence with the highest probability given the probability vectors. For training, the loss function was defined as the negative log-likelihood of the correct transcription given the per-window probability distributions.

The present inventors found that the CTC transcription probability performs poorly as a predictor of image mislabeling. The present inventors have therefore attached an error predicting branch to the network. The input of the error predicting branch was a layer from the middle of the network (a hidden layer), and the output of the error predicting branch was a single number, which was used either as a prediction of the number of errors in the transcription (the Edit Distance) as a regression task, or as a log-probability that the transcription is correct, as a classification task.

<FIG> illustrates the training process of the error prediction branch of the network. For training, only the source image and the correct transcription were used. The ground truth for training was calculated by running the image through the sequence prediction network, calculating the observed error between the network output and the ground truth, and training the error prediction branch based on this difference.

In the present example, the error prediction branch took as its input the output of the first bidirectional LSTM layer. Each vector of length <NUM> was projected to a vector of smaller dimension (a vector of length <NUM>, in the present example). Another bi-directional LSTM layer (with <NUM> units, in the present example) was then applied to the projected vector. Each of the outputs of the LSTM layer was projected to a scalar, and these scalars were summed, to provide a scalar output. Alternatively, the scalar output can be provided by averaging over these scalars (rather than only summing them), for example, to reduce the impact of the image width on the result. This scalar output was defined as the output of the error prediction branch, and was used as a log probability of the classification process.

Summing or averaging the projected scalars coming from the LSTM provided locality of error detection. This was advantageous both for the classification, where the method predicted whether the detected label is the same as the ground truth, and for a regression process, where the method predicted the number of errors the network makes. For classification, when the network outputs a large value at two different locations, the final sum is larger, which is consistent with the observation that there is a larger probability for error. For regression, when the network makes multiple errors, the error detection process is expected to detect errors in multiple locations, and the more locations get high values, the higher the error prediction should be.

The training of the sequence prediction network and of the error prediction branch was not done on the same dataset. The reason for that is that deep neural network models typically reach overfitting on the training set. When overfitting occurs, the loss function and the perceived labeling error reaches <NUM> or very close to it. If one attempts to train the error prediction branch jointly with the network, the outcome of the training is that the error predictor always output zeros. For this reason, training was a two-stage operation, where the network that provides the transcription was trained on a training set and evaluated on a validation set; and the error prediction branch was trained on the validation set and evaluated on the test set.

The system was trained on the three datasets (CAR, LAR, Account), achieving good accuracy. The error was measured as the overall edit distance, divided by the length of the correct string. The results are summarized in Table <NUM>, below. In Table <NUM>, the whole field accuracy results of the LAR have been measured after post-processing the transcription data. Measurements of accuracy of the dollar amount and the cent amount were provided separately.

For error prediction, an assortment of six models was trained and their results were compared. Three of the models were binary classification for predicting whether the resulting string is correct, and three were regression models that try to minimize the loss to the edit distance.

<FIG> shows ROC curves for all six models and the reference CTC score on the CAR dataset. The ROC curve shows the trade-off between the misread rate and the total system read rate. For example, a point at <NUM> misread rate and <NUM> read rate denotes that by picking a certain confidence threshold, the system rejects <NUM> % of the valid data, and reduces the number of errors by <NUM> %. To compare the effectiveness of different confidence methods, the area under curve (AUC) metric is used. As shown in <FIG>, the CTC score performs very badly as a measurement for error prediction. In contrast, the LSTM sum classification of the present embodiments performs significantly better than all other models, with an AUC of <NUM>.

Additional experiments were conducted using several benchmarks as well as on a dataset, owned by Orbograph Ltd. Yavne, Israel, including real-world cropped images of scanned dollar amounts for each method.

In the following experiments, two techniques were used to obtain a confidence measure. In a first technique, referred to below as "CTC ratio", a probability ratio was used as confidence without using neural network branch <NUM>. In this technique, probabilities that the OCR model assigns to its first and second most probable readings, given an input image, are obtained. A high confidence score is assigned where the first reading is much more probable than the second reading. If it is not, then the sample is ambiguous, and so its confidence score is lowered. In a second technique, referred to below as "ErrPred", an error prediction branch was added to the network, to directly predict whether either the OCR model will err given the input, or the sample should be rejected. The error prediction branch enacted neural network branch <NUM> (<FIG>).

The benchmarks included: ICDAR <NUM> (IC13) [<NPL>], Street View Text (SVT) [<NPL>], IAM [<NPL>], and RIMES [<NPL>].

In the IC13 dataset, the test set contains <NUM> scene images with labeled bounding boxes. The test set was restricted to contain only alpha-numeric characters with three characters or more, resulting in a test set of about <NUM> cropped text images.

The SVT dataset consists of <NUM> images collected from Google Street View. The word images have been cropped from these images, resulting in a test set of <NUM> samples.

The IAM dataset consists of scanned handwritten English texts by <NUM> writers, giving a total of <NUM>,<NUM> labeled word images. After filtering out punctuation and short words, a training set of <NUM>,<NUM> samples was obtained.

The RIMES dataset consists of scanned handwritten French texts by <NUM> people, giving a total of <NUM>,<NUM> words. Restricted to long enough words without punctuation, the training set size became <NUM>,<NUM> samples.

The Orbograph dataset consists of <NUM> million labeled images of automatically scanned and cropped fields denoting dollar amounts. This dataset was created by collecting dollar amount scanned text samples from real-world printed documents and having them manually labeled. Since this is unfiltered real-world data, it has an amount of reject samples, which were used to train the method of the present embodiments. <FIG> and <FIG> show ROC curves on the test set of the Orbograph dataset, where <FIG> shows the ROC curves for all images of the test set, and <FIG> shows the ROC curve for the non-rejected images of the test set. A misread refers to either a reject image receiving a high confidence score, or a non-reject image receiving the wrong transcription with a high confidence score.

The experiments were implemented in TensorFlow and trained on an nVidia Titan X and on an nVidia K80. CTC decoding was used a search beam of size <NUM>. Training was done using ADAM[<NPL>], with an initial learning rate of <NUM>-<NUM>.

Table <NUM> summarizes the accuracy obtained for each benchmark, as well the AUC obtained by the baselines and the method of the present embodiments, and Table <NUM> summarizes the accuracy obtained for the numeric dollar amount dataset as well the AUC obtained by the baseline and the method of the present embodiments. In Table <NUM>, the accuracy value for Orbograph (all) refers to all reject samples as misses, and as such it serves to demonstrate the need of an accurate confidence signal.

<FIG> show ROC curves for the IAM, RIMES, SVT and IC13, benchmarks, respectively, for both the baseline CTC-based confidence score as well as the method of the present embodiments.

Tables <NUM>, <NUM> and <FIG> demonstrate that the method of the present embodiments is effective in predicting and preventing OCR errors. In all datasets that have no reject samples, the probability ratio technique proved to be highly effective. When training on large datasets, both the probability ratio technique and the technique employing the error prediction branch obtain very convincing results. Without wishing to be bound to any particular theory, the relatively smaller accuracy obtained using the error prediction branch is explained by the lack of data.

Orbograph's dataset provides better benchmark conditions, as it is not synthetic and contains many samples. Additionally, this dataset mimics the real-world conditions and requirements of an OCR system by containing reject samples. As shown in <FIG> and <FIG>, when measuring error prediction accuracy without considering rejects, both techniques give very good performance, with the probability ratio method getting a slight edge. However, when considering reject samples as well, the technique employing the error prediction branch achieves exceptionally good results. The latter technique was able to reduce the error rate by as much as <NUM>%, while rejecting as few as <NUM>% of the valid samples.

<FIG> shows non-reject image examples from the Orbograph's dollar amount dataset at different levels of confidence, for qualitative assessment. It is expected to find that samples from the bottom confidence percentile are difficult to read for an OCR model trained on cropped, printed dollar amounts. Indeed, one sample is handwritten, one includes the surrounding box due to imperfect cropping, one includes unrelated text and markings, and the other two are blurry. Text in the fifth percentile is already fully legible and reasonably cropped. Note that the top percentiles are completely dominated by samples with the label "$<NUM>". The reason for that is that the dataset includes many such samples, and so the neural network of the present embodiments learned to recognize them with high confidence.

This Example demonstrates that the technique of the present embodiments vastly outperform the baselines suggested, and prove to be applicable to printed text OCR, handwritten text OCR, and scene text recognition.

Claim 1:
A method of recognizing image content, comprising:
applying a neural network to the image, said neural network being pre-trained for estimating an image content according to a specific alphabet and comprising: an input layer for receiving the image, a plurality of hidden layers for processing the image, and an output layer for generating output pertaining to an estimated image content based on outputs of said hidden layers;
applying to an output of at least one of said hidden layers, a neural network branch having an output layer for generating output pertaining to an estimated error level of said estimate; and
generating a displayed combined output comprising the estimated image content and the estimated error level;
wherein said neural network branch is trained based on a difference between an output generated at said output layer of said pre-trained neural network and a ground truth.