Optical character recognition using a combination of neural network models

Embodiments of the present disclosure describe a system and method for optical character recognition. In one embodiment, a system receives an image depicting text. The system extracts features from the image using a feature extractor. The system applies a first decoder to the features to generate a first intermediary output. The system applies a second decoder to the features to generate a second intermediary output, wherein the feature extractor is common to the first decoder and the second decoder. The system determines a first quality metric value for the first intermediary output and a second quality metric value for the second intermediary output based on a language model. Responsive to determining that the first quality metric value is greater than the second quality metric value, the system selects the first intermediary output to represent the text.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to Russian patent application No. 2020138343, filed Nov. 23, 2020, which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure is generally related to computer systems, and is more specifically related to systems and methods for optical character recognition using a combination of neural network models.

BACKGROUND

Artificial neural networks are computing systems that model biological neural networks. Such systems are trainable to solve various problems by looking at ready-made examples. They are most often used in applications that may be difficult to solve by a traditional computer algorithm.

SUMMARY OF THE DISCLOSURE

Embodiments of the present disclosure describe a system and method for optical character recognition of hand printed text using neural network models. In one embodiment, a system receives an image depicting texts. The system extracts a plurality of features from the image using a feature extractor. The system applies a first decoder to the plurality of features to generate a first intermediary output. The system applies a second decoder to the plurality of features to generate a second intermediary output, wherein the feature extractor is common to the first decoder and the second decoder. The system determines a first quality metric value for the first intermediary output and a second quality metric value for the second intermediary output based on a language model. The system determines that the first quality metric value is greater than the second quality metric value. Responsive to determining that the first quality metric value is greater than the second quality metric value, the system selects the first intermediary output to represent the texts.

DETAILED DESCRIPTION

Optical character recognition may involve processing an input image to extract a set of features that represent the input image and processing the extracted features by a decoder (represented by a trainable model) that yields a line of text depicted by the image. In some implementations, the trainable model can be represented by a neural network.

A neural network is a computational model simulating a set of connected components called artificial neurons (similar to axons in the biological brain). Each connection (synapse) between neurons can transmit a signal to another neuron. The receiving (postsynaptic) neuron can process the signal and then transmit this signal to downstream neurons. Neurons can have a state, usually expressed by real numbers between 0 and 1. Neurons and synapses may also have a weight that changes as training progresses, which may increase or decrease the strength of the signal they send to the next neurons. In addition, they can have a threshold function, so only the signal that is above (or below) this threshold will be transmitted further.

Neurons may be organized in layers. Different layers can perform different types of transformations on their inputs. Signals move from a first (input) layer to the last (output) layer through several hidden layers, such as a multilayer perceptron.

Neural networks can be applied for implementing deep learning models. Deep learning is a set of machine learning algorithms that attempt to model high level abstractions in data using architectures consisting of many nonlinear transformations, in other words, to identify “hidden features”.

Recurrent neural networks (RNN) is a type of neural network with cyclic connections between its units. These cycles create a concept of ‘internal memory’ for the network. The connections between its units form a directed sequence. The internal memory of RNN allows sequences of arbitrary length to be processed to generate corresponding variable length output sequences.

RNN can be used to process a series of events in time or sequential spatial chains. Unlike multilayer perceptrons, recurrent networks can use their internal memory to process sequences of arbitrary length. Therefore, RNN networks are applicable in tasks where something is completely divided into segments, such as handwriting recognition or speech recognition. Many different architectural solutions have been proposed for recurrent networks, from simple to complex. Recently, networks with long-term and short-term memory (LSTM) have become the most widespread.

Convolutional neural network (CNN) model may be used for pattern recognition tasks. The approach of convolutional neural networks is to alternate convolutional layers (convolution layers) and down-sampling layers (down-sampling layers or pooling layers). The network structure is unidirectional (without feedbacks) and essentially multi-layered. Standard methods are used for training, most often using a method of error back propagation and different activation functions for the neurons (transfer function) may be picked for particular tasks. The CNN architecture is distinguished by the presence of a convolution operation, the essence of which is that each image fragment is multiplied by the convolution matrix (core) element by element, and the result is summed and written to the same position of the output image.

As noted herein above, the input image can be processed to extract a set of features that represent the input image. The extracted features can then be processed by a text recognition decoder, which may be represented by a trainable model that yields a line of text depicted by the image.

LSTM and CNN can be used as building blocks for text recognition decoder designs. In some implementations, a text recognition decoder can be implemented by a connectionist temporal classification (CTC) decoder, which includes the building blocks of a CNN, a LSTM, and a CTC loss function, as further described below.

In other implementations, a text recognition decoder can be implemented by an attention based encoder-decoder, which includes the building blocks of a CNN, a LSTM (encoder), and a LSTM (decoder), as further described below. The two decoders are designed differently leading to different hypothesis (guesses) for text recognitions.

Advantageously, using both decoders simultaneously may improve an overall quality of the text recognition. The quality may be improved because these decoders may generate different results given the same inputs (because these decoders are different and have different building blocks arrangements) and the best result may be selected to improve the quality of the text recognition. Sharing different building blocks of these decoders may further reduce a processing requirement to execute the two decoders.

Various aspects of the above referenced methods and systems are described in details herein below by way of examples, rather than by way of limitation.

FIG.1depicts a high level system diagram of a system architecture in accordance with one or more aspects of the present disclosure. System architecture100includes a computing device120, and a server machine150connected to a network130. Network130may be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof.

The computing device120may perform character recognition using trainable classifiers (such as neural network) to effectively recognize texts including one or more sentences. A recognized sentence may include one or more words. A recognized word may include one or more characters (symbols).

The computing device120may be a desktop computer, a laptop computer, a smartphone, a tablet computer, a server, a scanner, or any suitable computing device capable of performing the techniques described herein. A document140including written text may be received by the computing device120. It should be noted that text printed or handwritten in any language may be received. The document140may include one or more sentences each having one or more words that each has one or more characters.

The document140may be received, for example, by scanning the document140or photographing the document140. Thus, an image141of the text including the sentences, words, and characters included in the document140may be obtained. Additionally, in instances where the computing device120is a server, a client device connected to the server via the network130may upload a digital copy of the document140to the server. In instances where the computing device120is a client device connected to a server via the network130, the client device may download the document140from the server.

The image of text141may be used to train a set of machine learning models or may be a new document for which recognition is desired. Accordingly, in the preliminary stages of processing, the image141of text included in the document140can be prepared for training the set of machine learning models or subsequent recognition. For instance, in the image141of the text, text lines may be manually or automatically selected, characters may be marked, text lines may be normalized, scaled and/or binarized.

Additionally, during preprocessing, the text in the image141obtained from the document140may be divided into fragments (e.g., words) of text. As depicted, a line is divided into fragments of text automatically on gaps having a certain color (e.g., white) that are more than threshold amount (e.g.,10) of pixels wide. Selecting text lines in an image of text may enhance the processing speed when recognizing the text by processing shorter lines of text concurrently, for example, instead of one long line of text. The preprocessed and calibrated images141of the text may be used to train a set of machine learning models or may be provided as input to a set of trainable machine learning models to determine the most probable text.

The computing device120may include a text recognition module110. Text recognition module110may be a software module executable by one or more general purpose and/or specialized hardware devices. Text recognition module110may include instructions stored on one or more tangible, machine-readable media of the computing device120and executable by one or more processing devices of the computing device120. In an implementation, the text recognition module110may use trainable decoders or machine learning models114that are trained and used to recognize one or more string of characters/symbols from the text in the image141. The text recognition module110may also preprocess any received images prior to using the images for training or inferencing based on one of the decoder(s)/models114. In some instances, the decoders/models114may be part of the text recognition module110or may be accessed on another machine (e.g., server machine150) by the text recognition module110. Based on the output of the trainable decoders/machine learning models114, the text recognition module110may extract one or more predicted sentences from text in the image141.

Server machine150may be a rackmount server, a router computer, a personal computer, a portable digital assistant, a mobile phone, a laptop computer, a tablet computer, a camera, a video camera, a netbook, a desktop computer, a media center, or any combination of the above. The server machine150may include a training module151. The decoders/machine learning models114may refer to model artifacts that are created by the training module151using the training data that includes training inputs and corresponding target outputs (correct answers for respective training inputs). The training module151may find patterns in the training data that map the training input to the target output (the answer to be predicted), and provide the decoders/machine learning models114that capture these patterns.

FIG.2depicts a block diagram of an example text recognition module in accordance with one or more aspects of the present disclosure. Text recognition module110may be a software module running on computing device120ofFIG.1. In one embodiment, text recognition module110can include image receiver module101, feature extractor module102, decode module103, quality metric determiner module104, quality metric selector module105, and output module106. One or more of modules101-106, or a combination thereof, may be implemented by one or more software modules running on one or more hardware devices.

Image receiver module101can receive a document image for optical character recognition (OCR). The image can be a document file, or a picture file with visible text. The image may be derived from various media outlets (such as printed or handwritten paper documents, banners, posters, signs, billboards, and/or other physical objects with visible texts on one or more of their surfaces. In one embodiment, image receiver module101can pre-processed the image by applying one or more image transformation to the image, e.g., binarization, size scaling, cropping, color conversions, etc. to prepare the image for OCR. In another embodiment, the pre-processing may segment the image into image fragments (images with visible words) for further processing.

Feature extractor module102can extract one or more features from an image. The features may be individual measurable properties or characteristics of a perceivable visual aspect of the image. In one embodiment, feature extractor102includes a rule-based feature extractor, or a convolutional neural network for feature extraction. Decode module103can decode the extracted features by using two or more decoders (such as a CTC or an attention decoder), the decoders to extract and decode output symbols from the features. Quality metric determiner module104can assign a quality metric value to output symbols of a particular decoder. Quality metric selector module105can compare the quality metric values among the output symbols of two or more decoders and select the output symbols with the highest quality metric value. Output module106can output the selected sets of symbols. Although modules101-106are shown separately, some of modules101-106, or functionalities thereof, may be combined.

FIG.3Aillustrates a block diagram of a hybrid decoder in accordance with one or more aspects of the present disclosure. Decoder300may be a single hybrid decoder that can decode text in an image and output symbols representing the text. Decoder300may be trained by server machine150and provided to computing device120to be used by computing device120. E.g., decoder300may be one of decoders/models114ofFIG.1. In one embodiment, decoder300includes feature extractor301, connectionist temporal classification (CTC) decoder303, and character decoder305. Feature extractor can extract features from an input image with text. The feature may be encoded and CTC decoder303may decode the encoded based on the CTC decoding, as further described below. The feature extractor301may include a CNN model used to extract features from the input image. The extracted features may be numeric vectors that represents text patterns found in the input image. Character decoder (with attention)305can decode the input image with text based on word character decoding with an attention mechanism, as further described below.

Referring toFIG.3A, in one embodiment, feature extractor301may be implemented by a convolutional neural network with a sequence of layers of different types, such as convolutional layers, pooling layers, rectified linear unit (ReLU) layers, and fully connected layers, each of which may perform a particular operation in optical character recognition. As illustrated, feature extractor301may include an input layer311, one or more convolutional layers313A-313B, ReLU layers315A-315B, pooling layers317A-317B, and a reshape (or output) layer319.

In some embodiments, an input image may be received by the input layer311and may be subsequently processed by a series of layers of feature extractor301. Each of the convolution layers may perform a convolution operation which may involve processing each pixel of an input image by one or more filters (convolution matrices) and recording the result in a corresponding position of an output array. One or more convolution filters may be designed to detect a certain image feature, by processing the input image and yielding a corresponding feature map.

The output of a convolutional layer (e.g., convolutional layer313A) may be fed to a ReLU layer (e.g., ReLU layer315A), which may apply a non-linear transformation (e.g., an activation function) to process the output of the convolutional layer. The output of the ReLU layer315A may be fed to the pooling layer317A, which may perform a down-sampling or subsampling operation to decrease the resolution and the size of the feature map. The output of the pooling layer317A may be fed to the convolutional layer313B.

Processing of the image by the feature extractor301may iteratively apply each successive layer until every layer has performed its respective operation. As schematically illustrated byFIG.3A, the feature extractor301may include alternating convolutional layers and pooling layers. These alternating layers may enable creation of multiple feature maps of various sizes. Each of the feature maps may correspond to one of a plurality of input image features, which may be used for performing optical character recognition.

In some embodiments, the pooling layer317B, which may be represented by a fully connected layer, of the feature extractor301may produce a feature vector representative of the features of the original image, which may be regarded as a representation of the image in the multi-dimensional space of image features.

The feature vector may be fed to the reshape layer319. Reshape layer319may apply a matrix transformation to the feature vector to flatten the feature vector into a feature array for CTC and/or word character decoding. Note that, various alternative implementations of feature extractor301may include any suitable numbers of convolutional layers, ReLU layers, pooling layers, and/or any other layers. In addition, batch normalization and dropout layers may be applied to improve training, and decrease a chance of overfitting, respectively.

Referring toFIG.3A, in one embodiment, CTC decoding303includes recurrent layer(s)321, fully connected layer325, and CTC layer327. The recurrent layer(s)321can include one or more gated recurrent units (GRU) or long short-term memory (LSTM) networks, or other variations of the RNN. In some embodiments, the recurrent layer(s)321may be unidirectional or bidirectional. In one embodiment, after the recurrent layer(s)321, one or more fully connected layer(s)325are applied. The fully connected layer(s)325may include a softmax layer. The fully connected layer(s)325can output to one or more CTC layer(s)327, which includes a CTC decoder or CTC loss function. CTC layer(s)327then generates a first decoded output, e.g., a first hypothesis of output symbols for the input image.

Connectionist temporal classification (CTC) is a type of neural network output and associated scoring function, for training recurrent neural networks (RNNs) such as LSTM networks to solve sequence problems where the number of spatial observations is variable. CTC refers to the outputs and scoring, and is independent of the underlying neural network structure. That is to say, the CTC loss function can be trained with any types of neural networks. The trained CTC loss function can then be used to decode feature maps to obtain the text contained in a corresponding image.

For CTC, the input data can be a sequence of observations (or spatial intervals), and the output data can be a sequence of symbols, which may include blank output data, such that one output symbol corresponds to one observation. Recognizing an individual output symbol is essentially classifying an instance of observation as a class (symbol) from a set of classes (symbols). For example, to recognize a word composed of any of the 26 alphabets (symbols), the CTC symbols can be predefined to 26 symbols for the 26 alphabets plus a blank ‘-’, which totals to 27 symbols. The blanks are removed at the time of decoding. The purpose of the blank is to solve the variability in the spatial intervals issue. That is, CTC encoding is time invariant and since a particular alphabet may be recognized in multiple spatial observations, the output may contain repetition of the particular character. The decoding process performs a CTC decoding algorithm to remove any character repetitions, e.g., “aa”→“a”, “aabbc”→“abc”, while the blanks effectively preserves an intended character repetition, e.g., “to-o”→“too”. An example of a neural network based on CTC loss function is shown inFIG.4.

Referring toFIG.3A, in one embodiment, character decoding305includes recurrent layer(s)331and character decoder layer333. In one embodiment, recurrent layer(s)331may include one or more GRU, or LSTM networks, or other variations of RNN. In one embodiment, recurrent layer(s)331may have similar components but with different trained weights than recurrent layer(s)321. Character decoder layer333can include a word decoder layer with an attention mechanism. The attention mechanism is a way to instruct the network of which observations to pay more attention to. For example, an attention layer may identify factors based on the training sample that reduce network error. Identification of these factors (or input nodes to the attention layer) can be performed during training through forward and backward propagation. An example of the structure of a neural network based on attention for recognition tasks is shown inFIG.5.

FIG.3Billustrates a block diagram of a hybrid decoder in accordance with one or more aspects of the present disclosure. Decoder310may be similar to decoder300but the recurrent (LSTM) layers321is shared or common to the CTC decoding307and character decoding309layers, e.g., the output of recurrent layers321serves as inputs for both CTC decoding307and character decoding309layers. In one embodiment, the recurrent (LSTM) layers321include LSTM layers. Here, the recurrent layers321with respect to the CTC decoder is a CTC encoder that is time invariant. The recurrent layers321can act, under the encoder-decoder word prediction scheme, as an encoder with respect to the character decoder. The combined recurrent layer321may be trained to serve dual functions, the function of a CTC encoder and the function of an encoder for the encoder-decoder word prediction. Recurrent layers321common to both decoders can reduce processing operations and memory requirements since only one set of encoding output is generated.

FIG.4illustrates an example block diagram of a connectionist temporal classification (CTC) decoder in accordance with one or more aspects of the present disclosure. Decoder350may represent a combination of feature extractor301and CTC decoding303ofFIG.3A. In one embodiment, decoder350receives an input image351with text “apple” for OCR. The input image may have dimensions of 64×128×3, e.g., 64 by 128 pixels with 3 color channels. The input image is fed to feature extractor301to extract feature maps from the input image through various convolutional, ReLU, and pooling (e.g., down sampling) layers. The output of the feature extractor301here provides a 4×8×4 feature map. The 4×8×4 feature map may be reshaped by a reshape layer (not shown) to a dimension of 16×8. The 16×8 feature map (8 spatial observations) is provided for the recurrent (LSTM) layers321(CTC encoding) to generate 8 output feature maps of dimension 16×1. The dimension of 16×1 is bounded by hyperparameters of the LSTM units, e.g., size and uni-directionality or bi-directionality. Bidirectional LSTMs are an extension of unidirectional LSTMs that can improve model performance on sequence classification problems. In this example, a current LSTM unit321receives an input spatial observation and generates an output feature map and a network state (internal memory). The network state is used as an input by a next LSTM unit321. The 8 output feature maps with dimensions 16×1 may pass through one or more fully connected layers (with softmax)325to generate 8 outputs of dimensions 6×1. Here, each of the 6×1 output represents an output symbol. An example sequence of output symbols may be “aap-plee”, where each symbol corresponding to one of the 8 spatial observations. The CTC loss layer327then decodes the symbols and converts the symbols to an output. In this example, repeated symbols “aa” and “ee” may appear in the encoding output because the first two spatial observations may have inferred an “a” and the last two spatial observations may have inferred an “e”. CTC loss layer327then applies a CTC decoding algorithm to the output sequence of layers325by first removing repeated symbols, followed by removing any blanks in the sequence, i.e., “aap-plee”→“ap-ple”→“apple”. AlthoughFIG.4illustrates an example network with 8 observations for the LSTM layer or recurrent layer321, any number of LSTM observations is possible. Note that the number of observations should be greater than a length of the predicted sequence of symbols. Note that the image/feature dimensions and the characters “apple” in this example are for illustrative purposes only and various implementations can include various image/feature dimensions and image texts.

FIG.5illustrates an example block diagram of an attention decoder in accordance with one or more aspects of the present disclosure. Decoder370may represent feature extractor301and character decoding305ofFIG.3A. Decoder370may use the encoder-decoder scheme with attention, where one or more recurrent layers (such as recurrent layers331ofFIG.3A) is used as the encoder component and one or more recurrent layers (such as character decoder layer333ofFIG.3A) is used as the decoder component of the encoder-decoder scheme. Referring toFIG.5, in one embodiment, decoder370receives an input image371with text “forum” for OCR. The input image is provided to feature extractor301to extract feature maps from the input image through various convolutional, ReLU, and pooling (e.g., down sampling) layers. The feature map outputs are provided to recurrent layer331, e.g., visual feature encoder, to encode the feature maps into an intermediary representation, e.g., sequence {tilde over (v)}. The intermediary representation is then decoded into output symbols y. Note that there are two special symbols: (SOS) start of sequence and (EOS) end of sequence in the output symbols, which denote the start and the end of the output symbols, respectively. The EOS solves the issue that the output sequence of symbols may have a different length than the observations in the recurrent layers. In one embodiment, the character decoder layer333includes one or more recurrent layers (e.g., LSTM) to perform the decoding. The character decoder layer333may be directional or bidirectional. In one embodiment, an output symbol is decoded based on an input from a corresponding intermediary representation. In another embodiment, the character decoder includes an attention mechanism. Here, an output symbol is decoded based on a context of a plurality of intermediary representation. For example, inferencing an input image may produce the intermediary representation, e.g., sequence13. At each spatial observation, the character decoding layer may choose one or more most interesting input observations based on the latest output. In one embodiment, a context vector373is generated to represent the attention mechanism. That is, context vector373represents which portion of the intermediary representation is most interesting for the decoder at an observation. In one embodiment, the context vector includes a binary vector for each observation, e.g., the decoding process determines which of the intermediary representations are to be considered. In one embodiment, the context includes a weighted vector for each observation, e.g., the decoding process determines how much weight of which of the intermediary representations are to be considered.

FIG.6illustrates an example block diagram of a hybrid CTC-attention decoder in accordance with one or more aspects of the present disclosure. Decoder400may represent decoder300ofFIG.3A. Referring toFIG.6, decoder400may be a hybrid CTC-attention decoder with CTC decoding and character decoding with attention that shares a common CNN feature extractor, e.g., common to the CTC decoding and the character decoding. In one embodiment, decoder400receives an input image401with text “Some text” for OCR. Image401may be preprocessed to dimensions H×W, where H is a height, and W is a width of image401. Image401may be passed through CNN301for features extraction to generate array of features (feature map)403of dimensions H1×W1×C, where H1 is a height, W1 is a width, and C is a channel for features403. The extracts features403is reshaped to features404with dimensions W1 by C*H1. Here, the reshape transformation can flatten a three-dimensional (3D) feature map into a 2D feature map. Next, feature map404is provided to both CTC decoding303and character decoding305. Referring to CTC decoding303, feature map404is provided to one or more LSTM layers321to generate intermediary representations409with dimensions W1×C2. Note that, the C2 value is only bound to the LSTM hyperparameters (size and directionality). For example, if a bidirectional LSTM layer321of size100is used, then C2=2*100=200. If a unidirectional LSTM layer321of size128is used, then C2=1*128=128. If all spatial observations of the input sequence are available, as in the instant example, bidirectional LSTMs include two trainable LSTM units instead of one LSTM unit. The first LSTM is trained using the initial input sequence and the second LSTM is trained using a reversed copy of the input sequence. This can provide additional context to the network and result in a faster training session.

The intermediary representations409are provided to fully-connected layer(s)325. The fully-connected layer325performs another convolution operation by multiplying the intermediary representations409by a matrix of trainable weights (not shown). In one embodiment, the output can be a softmax layer411that outputs a probability of performing one of the actions from the set of all possible actions. The disadvantage of this architecture is that it is difficult to implement the choice of several simultaneous actions at once. In another embodiment, the softmax layer411is a sequence of W1 vectors (or a matrix with W1 columns), and each column contains a set of probabilities corresponding to a set of symbols. In one embodiment, the set of probabilities in a column is normalized, i.e. the sum of the probabilities for a column is 1. The set of symbols, with a size=“#symbols”, includes all valid text recognition characters, e.g., letters, numbers, separators, etc. plus a special “-” character called blank, which will be used by the CTC decoder327. The softmax layer411, with dimensions W by #symbols, is passed through CTC decoder327to generate a first set of output characters (symbols) representative of the text in input image401. Note that CTC decoder412performs a CTC decoding algorithm that removes repeated characters followed by blanks ‘-’, if any.

Referring to character decoding305, feature maps404are provided to one or more LSTM layers331to generate intermediary representations413of dimensions W1×C2. Attention-based character decoding uses a different special character: EOS in the intermediary representations413, which indicates the end of the sequence. Thus, LSTM layers331can generate a sequence with an EOS indicator. Intermediary representations413is provided to character decoder with attention333and a character decoding algorithm is applied to generate a second set of output characters/symbols representative of the text in input image401. Here, the character decoding algorithm of character decoding333determines the location of the EOS character and removes any characters after the EOS character.

In one embodiment, the first set of output characters and the second set of characters are each passed through a language model (not shown) to determine a first and a second quality metric for the first and second set of output characters, respectively. Quality metric value refers to a ranking of how polished the sequence of output characters/symbols is. If the first quality metric value is greater than the second quality metric value, the first set of output characters is determined to be the output of the hybrid decoder400, and vice versa. In one embodiment, the language model includes a rule-based model based on dictionary, morphological model of inflection, a syntactic model, or statistics on the compatibility of letters and/or words, or a combination thereof. The quality metric value may be determined based on a counting mechanism of matching outputs for one or more of these rule-based models. For example, for a dictionary model, if the first set of output characters (a first word) matches a dictionary look up, the quality metric value for the first set of output characters increments by 1. Similarly, if the second set of output characters (a second word) matches a dictionary look up, the quality metric value for the second set of output characters increments by 1. In another example, a model based on statistics on the compatibility of letters may use a character compatibility table that indicates if some adjacent characters are compatible. Given a particular character, the character compatibility table may indicate which adjacent characters are frequently occurring, e.g., compatible and the adjacent characters. For each set of output characters, each character and its adjacencies in the set are looked up in the character compatibility table for a match. If a match exists, the quality metric value for the set increments by 1.

In another embodiment, the language model includes a trainable recurrent neural network (RNN) model (not shown). In another embodiment, the RNN model includes a self-attention mechanism as described above (not shown). Here, the RNN model may be trained to provide a quality metric value for a set of characters based on correctness of spelling of a word represented by the set of characters. For example, a RNN model may be trained for text classification to classify a sequence of characters into two classes: words with correct spelling and words with incorrect spelling. The training samples for the first class may use a dictionary and labeled the first class. The training samples for the second class can include randomly generated words that is not in the dictionary and labeled the second class. The trained RNN may infer a quality metric value based on a probability of the inferred class, e.g., the quality metric value may be the probability the word belongs to the first class. The self-attention mechanism may be similar to the attention mechanism for decoder333.

Referring toFIG.1, the above described decoders may be trained by server machine150. Training module151of server machine150may be configured to train any of the decoders inFIGS.3A-6. In one embodiment, referring toFIG.3A, the hybrid CTC—attention decoder300may be trained as a single unit. For example, module151may train the CTC decoding303using a CTC loss function (as shown inFIG.4) based on a CTC forward backward algorithm. Module151may simultaneously train character decoding305using forward and backward propagation. Here, input images and labels are prepared by an operator for training, e.g., input image with “apple” and two separate outputs “aap-plee” and “<sos>apple<eos>” for CTC decoding303and character decoding305, respectively, may be provided as ground truths to train decoder300. Here, <sos> represents start of sequence, and <eos> represents end of sequence.

For example, training module151may set up a number of convolutional layers for the feature extractor and a number of LSTM layers for the recurrent layers. Training module151may initialize random value for all parameters of the decoder network. Training module151may calculate a forward pass, e.g., pass the training instance through the network and obtain all symbol probabilities for each observation. For the CTC decoding, training module151may construct a table (for all possible paths for each possible sequence of the intermediary representation) to represent specific symbol sequences for each observation. Training module151may compute CTC forward and backward variables for the table and determine a divergence error for each symbol probability at each observation based on the CTC forward and backward variables. Based on the divergence error, training module151may update one or more weights in the LSTM and CNN layers.

In another embodiment, module151may train the CTC decoder portion of the hybrid CTC—attention decoder300separately from the attention-character decoder portion of the hybrid CTC—attention decoder300. E.g., the CTC decoder portion (feature extractor301and CTC decoding303) of300may be trained first. Once the weights are determined, module151trains the character decoding305to determine the weights for the layers in character decoding305.

Referring toFIG.3B, in one embodiment, the hybrid CTC—attention decoder310may be trained as a single unit by module151. In another embodiment, module151may train the hybrid CTC—attention decoder300separately. I.e., the CTC decoder portion (feature extractor301and CTC decoding303) of300is trained first, followed by the character decoding309. Once a decoder is trained, the trainable decoder can be provided to a text recognition module for inference, such as text recognition module110ofFIG.1.

FIG.7depicts a flow diagram of a method for optical character recognition in accordance with one or more aspects of the present disclosure. The method700can be performed by processing logic that can include hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, method700is performed by text recognition module ofFIG.1. Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, one or more processes can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

At block701, processing logic receives an image with text. At block703, processing logic extracts a plurality of features from the image using a feature extractor. The feature extractor can be a trainable CNN model trained to extract the image features. The image features may be characteristics of the writing in the image that can be used for pattern recognition or machine learning classification. At block705, processing logic applies a first decoder to the plurality of features to generate a first intermediary output. The first intermediary output may be a first hypothesis/guess for the sequence of perceivable characters in the image. At block707, processing logic applies a second decoder to the plurality of features to generate a second intermediary output, where the feature extractor is common to the first decoder and the second decoder. The second intermediary output may be a second hypothesis/guess for the sequence of perceivable characters in the image. At block709, processing logic determines a first quality metric value for the first intermediary output and a second quality metric value for the second intermediary output based on a language model. The language model can be a dictionary model, character compatibility model, etc. At block711, responsive to determining that the first quality metric value is greater than the second quality metric value, processing logic selects the first intermediary output to represent the text.

In one embodiment, the first decoder includes a connectionist temporal classification decoder and the second decoder includes a character decoder with attention. In one embodiment, the first decoder includes a first long short-term memory component and the character decoder with attention includes a second long short-term memory component. In one embodiment, the first and the second decoder includes a long short-term memory component common to the first and the second decoder. In one embodiment, the first decoder and the second decoder is trained simultaneously with a connectionist temporal classification (CTC) loss function and an attention mechanism, respectively.

In one embodiment, the language model includes a dictionary, a morphological model of inflection, a syntactic model, or a model for statistical compatibility of letters or words. In one embodiment, the language model includes a recurrent neural network model with self-attention.

FIG.8depicts an example computer system800which can perform any one or more of the methods described herein. In one example, computer system800may correspond to a computing device120capable of executing text recognition module110ofFIG.1, or server machine150capable of executing training module151ofFIG.1. The computer system may be connected (e.g., networked) to other computer systems in a LAN, an intranet, an extranet, or the Internet. The computer system may operate in the capacity of a server in a client-server network environment. The computer system may be a personal computer (PC), a tablet computer, a set-top box (STB), a personal Digital Assistant (PDA), a mobile phone, a camera, a video camera, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, while only a single computer system is illustrated, the term “computer” shall also be taken to include any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

The exemplary computer system800includes a processing device802, a main memory804(e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory806(e.g., flash memory, static random access memory (SRAM)), and a data storage device816, which communicate with each other via a bus808.

The computer system800may further include a network interface device822. The computer system800also may include a video display unit810(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device812(e.g., a keyboard), a cursor control device814(e.g., a mouse), and a signal generation device820(e.g., a speaker). In one illustrative example, the video display unit810, the alphanumeric input device812, and the cursor control device814may be combined into a single component or device (e.g., an LCD touch screen).

The data storage device816may include a computer-readable medium824on which is stored instructions826(e.g., corresponding to the method ofFIG.7, etc.) embodying any one or more of the methodologies or functions described herein. Instructions826may also reside, completely or at least partially, within the main memory804and/or within the processing device802during execution thereof by the computer system800, the main memory804and the processing device802also constituting computer-readable media. Instructions826may further be transmitted or received over a network via the network interface device822.