Patent Description:
When a computer is used to process textual information for different languages, the characters need to be first encoded in a machine-readable format. Each character of the language may be assigned a unique encoding. Typically, the encoding has no semantic meaning in that an encoding of a first character has no relationship to an encoding for a second character. This may make training an application to recognize the characters less efficient and also cause resulting output to have recognizable errors.

"<NPL>" describes multi-pooling and data augmentation with non-linear transformation that are applied to a convolutional neural network (CNN) for multi-font printed Chinese character recognition (PCCR).

"<NPL>" describes a glyph-aware embedding of Chinese characters comprising explicitly incorporating visual appearance of a character's glyph in its representation by a convolutional neural network.

"<NPL>" describes a system of classification using probabilistic neural networks, wherein training of the classifier starts with the use of distortion modeled characters from four fonts.

The claimed invention is defined by the independent claims. Specific embodiments are defined in the dependent claims.

With respect to the discussion to follow and in particular to the drawings, it is stressed that the particulars shown represent examples for purposes of illustrative discussion, and are presented in the cause of providing a description of principles and conceptual aspects of the present disclosure. In this regard, no attempt is made to show implementation details beyond what is needed for a fundamental understanding of the present disclosure. The discussion to follow, in conjunction with the drawings, makes apparent to those of skill in the art how embodiments in accordance with the present disclosure may be practiced. Similar or same reference numbers may be used to identify or otherwise refer to similar or same elements in the various drawings and supporting descriptions. In the accompanying drawings:.

A system generates encodings that represent characters in a language. The encodings are in a machine readable format (e.g., binary codes). The system generates the encodings based on a similarity between characters in a language. The encodings are based on a visual similarity of symbols in the language. The similarity is based on glyph structure, where a glyph is an elemental symbol within an agreed set of symbols that are intended to represent a readable character in the language. The symbols that are visually similar also share a semantic similarity in the language, but this may not be the case for all visually similar characters.

By generating similar encodings for characters that are considered visually similar may be advantageous when an application is processing text of the characters. For example, the encodings may offer error robustness in that if a single bit or small number of bits in an encoding is incorrectly predicted, the outputted character may still be a character that is visually similar to the correct character. Also, there is a chance that if only a single bit is incorrectly predicted, the encoding may still be used to determine the correct character. Further, even if the wrong character is selected, outputting a visually-similar character may be less confusing to a user than outputting a visually dissimilar character from the correct character. Also, the training of a process that is configured to learn a character may be trained faster if the encodings are based on visual characteristics because visually similar characters may likely share many bits in a representation. It will therefore be easier for the process to learn the bits that are similar and then focus on the more challenging bits to distinguish between the visually similar characters.

<FIG> depicts a simplified system <NUM> for generating encodings for characters according to some embodiments. The characters may be from a specific language, such as Mandarin, or other logographic-related languages. However, system <NUM> may be used to generate encodings for any language or multiple languages.

An application <NUM> may be configured to receive characters, such as a visual representation of characters, and generate encodings for the characters. The visual representation may be a textual representation of characters. The characters may be logographic in which characters are marked by a letter, symbol, or sign that is used to represent an entire word or phrase. The encodings may be machine readable encodings, such as a series of binary codes. However, the encodings may be any representation for the characters that can be read, such as a series of letters or numbers.

Application <NUM> may use the encodings to generate an output. For example, application <NUM> may be an optical character recognition (OCR) engine that receives an image and outputs the text from the image. Application <NUM> may then analyze the text to generate encodings for characters in the text. Once generating the encodings, application <NUM> may use the encodings to generate an output, which may be the translation of the characters into another language, such as English. Other output may include a representation that corresponds to the recognized characters, such as a Pinyin version of the corresponding characters or the actual visual representation of the characters. Various outputs may be appreciated.

Application <NUM> may use an encodings model <NUM> to generate the encodings. Encodings model <NUM> may include the corresponding encodings for characters in the language. Encodings model <NUM> may also include any parameters for a process that is used by application <NUM> to recognize characters and generate the corresponding encodings. For example, application <NUM> may use a prediction network that receives an image of a character, and generates an encoding based on encodings model <NUM>. Then, application <NUM> generates output for the encoding, such as a transaction of the character in another language.

As will discussed in more detail below, encodings for visually similar characters may also be similar. In some embodiments, the encodings may be binary numbers, such as a binary number of N binary numbers (e.g., <NUM>). As will be discussed in more detail below, encodings model <NUM> may include encodings that are more similar for visually similar characters. That is, a number of bits may be similar in the encodings for visually similar characters. A similar encoding may mean that a number of bits in the similar encodings are equal, such as above a threshold, out of a total number N bits. For example, the three symbols <IMG> (the English word "rain"), <IMG> (the English word "snow"), <IMG> (the English word "thunder") have a similar meaning in the Mandarin language, and they all have a common sub-structure <IMG>. Therefore, encodings model <NUM> may use a glyph-aware encoding function E in which the E(<IMG>) is more similar to E(<IMG>) than to E(<IMG>) (the English word "hit"). To generate the similar encodings for visually similar characters, server system <NUM> trains a network to generate the encodings in encodings model <NUM>.

Using the encodings that are visually aware, such as glyph aware, to encode logographic scripts have multiple advantages. For example, when an encoding is used by application <NUM>, such as to generate the output of application <NUM>, advantages may result when errors occur when an application selects the encoding. For example, error robustness is provided if application <NUM> is predicting a given glyph (e.g., symbol <IMG> (snow)) through a glyph-aware encoding of a certain amount of bits N with a slightly redundant N (e.g., <NUM>N>x, with x being the total number of admissible characters), there is a much better chance that if a single bit is incorrectly predicted, the output of the system with the error may still match the correct character. That is, not all combinations of bits may be used, but the closest pattern to the predicted bit pattern may be the correct encoding for the character. Thus, the additional bits may be used for error-robust redundancy.

Also, using the encodings may produce fewer grave errors for application <NUM>. For example, in a situation where one, or a few, bits are incorrectly predicted for application <NUM>, even if this error is enough to change the prediction to a different character, the resulting character may likely be visually similar, such as the symbol thunder <IMG> is to the symbol snow <IMG>. Seeing the thunder character in place of the snow character may be much less confusing to a user than if a completely random character was inserted in the position by application <NUM>.

Also, the encodings based on visually similar characters may also make learning the representations faster. A process that needs to learn a character as an output will learn faster if the encoding is visually aware. This is because visually similar characters will likely share many bits in the representation. It will therefore be easier for the process to learn these bits and make a focus on more challenging bits to learn allowing the process to distinguish between visually similar characters. For example, application <NUM> may include an optical character recognition application. Using the visually aware encodings, encodings model <NUM> may learn the commonalities as well as the differences of symbols faster by first learning the easier bits that can be predicted for a character. The same applies to scene text recognition (STR) that recognizes scenes in images, such as a landmark in a picture. Further, errors as discussed above, in the output may be easier to understand for a human reader.

Application <NUM> may also be used for rendering of characters, such as generation of text that simulates handwritten text. Application <NUM> may receive a representation of text as bits for input and generate images as output. The generated image may represent the same character as the input, but at the same time look handwritten in style (plus potentially other conditions, such as having a homogeneous style, etc.). In this case, using the encodings for the input bits of the representation has advantages because it facilitates the task of application <NUM> to generate the images, which can intrinsically learn that a specific bit typically corresponds to a stroke in a specific position due to the similarity of encodings to visually similar characters.

<FIG> depicts an example of training a prediction network <NUM> to generate encodings model <NUM> according to some embodiments. Server system <NUM> uses a prediction network <NUM> that can receive an input of images and output classifications for the images. The input of images may be images of characters for the language. The classifications may be a label that is predicted for the character. For example, prediction network <NUM> receives a symbol for the word rain and outputs a label for the symbol <IMG>.

<FIG> depicts an example of labels and images according to some embodiments. The images may be cropped from different documents and include an image of a character. As shown, at <NUM>-<NUM> to <NUM>-<NUM>, different symbols for a language have been cropped. Also, corresponding labels for each image crop may be provided, such as at <NUM>-<NUM> to <NUM>-<NUM>.

Referring back to <FIG>, prediction network <NUM> is trained to classify the image crops with a label. An input of images and corresponding labels is received from a number of documents that contain characters of a language in which the encodings are generated. The documents may contain all of the characters or a large amount of characters in the language. In some embodiments, the labels may be provided by a user or machine and are the correct label for the character in the image crop. For example, a set of documents may be written in the Mandarin language and a result of applying an optical character recognition engine to the document indicates the location of each character and a label for the character. Alternatively, a searchable document that is rendered may be created from arbitrary characters in the language and the renderer may automatically know the location and identification for each character.

<FIG> depicts a simplified flowchart <NUM> of a method for training prediction network <NUM> according to some embodiments. At <NUM>, server system <NUM> retrieves documents with characters. At <NUM>, server system <NUM> extracts images of the characters. At <NUM>, server system <NUM> generates labels for the characters.

At <NUM>, server system <NUM> trains prediction network <NUM> to classify the image-crops of the characters to the corresponding character label. There may be K unique characters and therefore, the output of prediction network <NUM> performs a classification task with K classes. That is, given an image crop, prediction network <NUM> may output a value, such as a vector of dimension K, having a value at an index corresponding to the classification. For example, each character in a language may be associated with a vector of a dimension K. Then, the output of prediction network <NUM> may correspond to one of the vectors for a character. This character is the classification for the image.

In some embodiments, the encoding value is different from the output value for the classification. For example, to obtain the encoding, prediction network <NUM> may include a layer that outputs the encoding in a fixed dimension N, which matches the target dimension (e.g., a length or number of bits) of the binary encoding. That is, if the encodings are each <NUM> bits, then the value of N is "<NUM>". The layer that outputs the encoding may be different from the layer that outputs the classification. For example, the layer that outputs the encoding may be an internal layer (e.g., an intermediate representation) in prediction network <NUM> and the layer that outputs the classification may be the output layer of prediction network <NUM> or the last layer. The value of N may be significantly smaller than K, which implies that the internal representation is more compact than the number of classes.

Referring back to <FIG>, at <NUM>, after training, server system <NUM> inputs images into prediction network <NUM> and computes an encoding from a layer, such as a dense layer (described below), of prediction network <NUM> for each image. Then, at <NUM>, server system <NUM> stores the encodings for the characters in encodings model <NUM>.

The encodings may be generated from prediction network <NUM> in different ways. In some embodiments, a layer, such as a bottleneck layer, in prediction network <NUM> may output a value in a fixed dimension N, which matches the target dimension of the encoding. During the training, parameters in prediction network <NUM> is adjusted to accurately predict labels for the images. For example, prediction network <NUM> may output a classification for images. Then, the classification output by prediction network <NUM> is compared to labels <NUM> and the parameters in prediction network <NUM> are adjusted during the training process such that prediction network <NUM> can accurately predict a classification for images based on labels <NUM>.

<FIG> depicts a more detailed example of prediction network <NUM> according to some embodiments. Prediction network <NUM> includes layers <NUM> that receive an input of images and process those images. For example, the input of images may be images <NUM>-<NUM> to <NUM>-<NUM>, and other images. In some embodiments, layers <NUM> may be two-dimensional (2D) convolutional layers that process characteristics in the images. Layers <NUM> may process the images to perform operations on characteristics of the images to analyze characteristics of the images. This may include layers of different dimensions or size of filters, and the operations may include pooling operations to reduce resolution, activation functions, etc. Layers <NUM> also include a flattening layer to flatten the image to a vector.

Layers <NUM> may also include a number of dense layers may also be included. A dense layer transforms the vector using a number of filters, activation functions, etc. Different variations of layers <NUM> may be appreciated.

The dense layer <NUM> has a constraint, an output of a dimension N, which is equal to the dimension of the encoding. This means that the output of dense layer <NUM> is a number of values of a dimension N that equal the number of values in an encoding. The constraint makes this dense layer a bottleneck layer because the output of the dense layer is restricted (e.g., reduced from a previous layer) to N dimensions. Additionally, dense layer <NUM> may have a function, such as an activation function, that forces its outputs to be between two values, such as <NUM> and <NUM>. In some embodiments, a sigmoidal activation function that output values between <NUM> and <NUM> may be used. Accordingly, dense layer <NUM> may output N values between zero and one for each image.

Output layer <NUM> may be a last layer that has a dimension K. The dimension K allows for a classification to be determined. For example, the K dimensions may be non-negative outputs that sum to <NUM> to represent a probability distribution. The values of the output may correspond to a single classification of a character in a language. In some embodiments, the value of K may be smaller than the value of N. The representation of the dense layer <NUM> may act as bottleneck where the data is compressed into a compact representation. The processing step(s) between layer <NUM> and <NUM> allow a conversion of the compact representation into a non-compact classification layer. Within <NUM>, the network may represent similar images (e.g. pixel-similarity) with similar embeddings (e.g. vector similarity). In the output layer <NUM>, images from different characters are equally (dis-)similar from each other.

Once prediction network <NUM> is trained, the values received from dense layer <NUM> with a dimension N may be used to generate encodings in encodings model <NUM>. In some embodiments, the values for each character may be discretized to bits to make a binary encoding. For example, server system <NUM> may map numbers below <NUM> to a binary value of <NUM> and numbers above <NUM> to a binary value of "<NUM>". Other thresholds may also be used or other methodologies, such as an average or median of values by a specific dimension for the images of a character.

Other variations may be used during training to generate the encodings. For example, server system <NUM> may gradually modify an activation function in dense layer <NUM> to make the results more and more skewed towards being exactly <NUM> or <NUM>. For example, the values may be multiplied by an increasing "temperature" value before applying a soft max function. This makes the specific value of the threshold less important as more values are skewed towards <NUM> or <NUM>. Another variation adds a supplementary loss term to stress that the same characters should have the same encoding in dense layer <NUM>. Server system <NUM> may add a penalty during training for any pair of elements inside a training batch based on how different their K dimensional representations are. Many measures of differences can be used, such as L1 or L2 distance calculations.

Accordingly, as can be seen, encodings model <NUM> may be generated from output of dense layer <NUM>, and not output layer <NUM>. Output layer <NUM> is used to validate the results of prediction network <NUM> such that prediction network <NUM> is accurately predicting the character of images. This classification task may ensure that the embeddings in layer <NUM> are discriminable.

<FIG> depicts an example of encodings according to some embodiments. The encodings shown at <NUM> to <NUM> may be <NUM>-dimensional binary encodings for six symbols. The <NUM> values of the encodings may be represented as a value of <NUM> or <NUM>. The value of <NUM> for a bit may be where there is no slash marks and the value of <NUM> for a bit is where slash marks are included. Similar patterns of bits for symbols indicate that the encodings are similar. For example, for encodings <NUM> and <NUM>, the number of bits is similar. Also, the encodings at <NUM> and <NUM> also include a number of similar bits as well as the encodings at <NUM> and <NUM>.

<FIG> depicts an example for symbols with similar binary codes according to some embodiments. At <NUM>, neighboring encodings for symbol at <NUM> are shown. In this example, seven neighboring encodings for that symbol are shown. A number of bits for each respective symbol are shown as [<NUM>]. This means, there are seven flipped bits or different bits for the first symbol that neighbors the symbol at <NUM>; there are seven flipped bits for the second symbol that neighbors symbol at <NUM>, etc. Each symbol with neighboring labels at <NUM> to <NUM> for symbols <NUM> to <NUM> may include different numbers of bits that are flipped. However, the most number of bits that are flipped may be <NUM> out of <NUM> dimensions.

Accordingly, using encodings that are based on visually similar characters may improve the performance of application <NUM>. This reduces the number of errors that can result, and also reduces the effect of errors that may result. Additionally, the training of the algorithm to generate encodings model <NUM> may be faster and easier.

<FIG> illustrates hardware of a special purpose computing machine according to one embodiment. An example computer system <NUM> is illustrated in <FIG>. Computer system <NUM> includes a bus <NUM> or other communication mechanism for communicating information, and a processor <NUM> coupled with bus <NUM> for processing information. Computer system <NUM> also includes a memory <NUM> coupled to bus <NUM> for storing information and instructions to be executed by processor <NUM>, including information and instructions for performing the techniques described above, for example. This memory may also be used for storing variables or other intermediate information during execution of instructions to be executed by processor <NUM>. Possible implementations of this memory may be, but are not limited to, random access memory (RAM), read only memory (ROM), or both. A storage device <NUM> is also provided for storing information and instructions. Common forms of storage devices include, for example, a hard drive, a magnetic disk, an optical disk, a CD-ROM, a DVD, a flash memory, a USB memory card, or any other medium from which a computer can read. Storage device <NUM> may include source code, binary code, or software files for performing the techniques above, for example. Storage device and memory are both examples of computer readable storage mediums.

Computer system <NUM> may be coupled via bus <NUM> to a display <NUM>, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. An input device <NUM> such as a keyboard and/or mouse is coupled to bus <NUM> for communicating information and command selections from the user to processor <NUM>. The combination of these components allows the user to communicate with the system. In some systems, bus <NUM> may be divided into multiple specialized buses.

Computer system <NUM> also includes a network interface <NUM> coupled with bus <NUM>. Network interface <NUM> may provide two-way data communication between computer system <NUM> and the local network <NUM>. The network interface <NUM> may be a digital subscriber line (DSL) or a modem to provide data communication connection over a telephone line, for example. Another example of the network interface is a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links are another example. In any such implementation, network interface <NUM> sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

Computer system <NUM> can send and receive information through the network interface <NUM> across a local network <NUM>, an Intranet, or the Internet <NUM>. In the Internet example, software components or services may reside on multiple different computer systems <NUM>, clients <NUM>, or servers <NUM>-<NUM> across the network. The processes described above may be implemented on one or more servers, for example. A server <NUM> may transmit actions or messages from one component, through Internet <NUM>, local network <NUM>, and network interface <NUM> to a component on computer system <NUM>. The software components and processes described above may be implemented on any computer system and send and/or receive information across a network, for example.

Some embodiments may be implemented in a non-transitory computer-readable storage medium for use by or in connection with the instruction execution system, apparatus, system, or machine. The computer-readable storage medium contains instructions for controlling a computer system to perform a method described by some embodiments. The computer system may include one or more computing devices. The instructions, when executed by one or more computer processors, may be operable to perform that which is described in some embodiments.

Claim 1:
A computer-implemented method (<NUM>) for generating encodings, the method (<NUM>) comprising:
retrieving (<NUM>), by a computing device, documents with characters;
extracting (<NUM>), by the computing device, images of the characters;
generating (<NUM>), by the computing device, labels for the characters;
training (<NUM>), by the computing device, a prediction network (<NUM>) to classify image-crops of the characters to the corresponding character label;
after training (<NUM>), inputting (<NUM>), by the computing device, the images into the prediction network (<NUM>) and computing (<NUM>), by the computing device, a set of encodings from a dense layer (<NUM>) of the prediction network (<NUM>), the dense layer (<NUM>) having an output of a dimension N, which is equal to the dimension of the encodings; and
storing (<NUM>), by the computing device, the set of encodings for the one or more characters in an encodings model (<NUM>),
wherein an encoding in the set of encodings is retrievable when a corresponding character is determined,
wherein the encodings represent the characters in a language in a machine-readable format and are generated based on a visual similarity between the characters in the language, and
wherein the visual similarity is based on glyph structure, where a glyph is an elemental symbol within an agreed set of symbols that are intended to represent a readable character in the language, and/or wherein the symbols that are visually similar share a semantic similarity in the language.