Patent Description:
Images can depict text that is written in a source language. It is often desirable to translate the text depicted in the image from the source language into the target language. To accomplish this, conventional systems generally use at least two models: an optical character recognition (OCR) model and a machine translation model. The OCR model processes the image to recognize text depicted in the image, and the recognized text is then input into a machine translation model that translates the recognized text into the target language. <CIT> is directed towards optical character recognition and machine language translation of OCR text from an image based on non-textual context information from the image.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that can include the operations of obtaining a first image that depicts first text written in a source language; inputting the first image into an image feature extractor that is trained to extract, from an input image, a set of image features that are a description of a portion of the input image in which the text is depicted; obtaining, from the feature extractor and in response to inputting the first image into the feature extractor, a first set of image features representing the first text present in the first image; inputting the first set of image features into a decoder, wherein the decoder is trained to infer text in a target language from an input set of image features, wherein the inferred text is a predicted translation of text represented by the input set of image features; and obtaining, from the decoder and in response to inputting the first set of image features into the decoder, a second text that is in a target language and is predicted to be a translation of the first text. Other embodiments of this aspect include corresponding systems, devices, apparatus, and computer programs configured to perform the actions of the methods. The computer programs (e.g., instructions) can be encoded on computer storage devices.

These and other embodiments can each optionally include one or more of the following features.

In some implementations, methods can include training the feature extractor using a set of input training images that depict training text in the source language and corresponding sets of training image features, wherein each set of training image features is a description of a portion of the input image in which the training text is depicted.

In some implementations, the decoder can include a text-to-text translation model.

In some implementations, methods can include training the decoder, which can include the operations of training the text-to-text translation model to translate text written in the source language into text in the target language, wherein the text-to-text translation model is trained using a set of input training text data in the source language and a corresponding set of output training text data that is a translation of the input training text data from the source language into the target language; and training the trained text-to-text translation model to output text data in a target language that is a predicted translation of text represented by an input set of image features that represent text in an input image, wherein the trained text-to-text translation model is trained using a set of input training images that depict training text in a source language and a corresponding set of text data that is a translation in a target language of the training text depicted in the input training images.

In some implementations, the feature extractor can be a convolution neural network (CNN) with <NUM> layers of convolution, residual, and pooling.

In some implementations, the decoder can be a <NUM>-layer multi-head transformer decoder and the text-to-text translation model is a transformer neural machine translation model.

Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages. For example, the innovations described in this specification can utilize a single end-to-end image translation model to translate text depicted in an image from a source language into a target language, as compared with conventional systems that utilize multiple models, such as an OCR model and a machine translation model. By utilizing a single end-to-end model, the techniques described in this specification are more resource efficient (i.e., they utilize less computer processing and memory requirements) in comparison to conventional systems. Relatedly, the single end-to-end image translation model is more easily distributed to end-user devices in contrast with the two or more models of conventional systems. This is because the single end-to-end image translation model is expected to have less dependencies (relative to conventional systems with two models, where one model inter-depends on another model) and fewer resource requirements (i.e., hardware specifications, dependency packages, etc.).

Moreover, the techniques described in this specification also enable improved conversion of text represented in input images to translated text than conventional systems. Conventional systems generally use two distinct and unrelated models (an OCR model and a machine translation model). The OCR model recognizes text depicted in the image, and the machine translation model translates the recognized text output by the OCR model. As such, any errors introduced by the OCR model are propagated to the machine translation. For example, the OCR model may incorrectly determine that the letter "O" in "GOOD" is a zero ("<NUM>") and thus, recognize the text as "GOOD. " In this case, the separate machine translation model may not be able to translate this word and thus, may incorrectly return "GOOD" as the translated text. In contrast, the image translation model described in this specification accommodates for error/noise that would otherwise be introduced by the conventional OCR models. Using the above example, the image translation model described in this specification would accommodate for the OCR error (i.e., it would recognize that "GOOD" refers to "GOOD" with an <NUM>% probability) and thus, generates a translation for "GOOD" instead of "GOOD. " As a result, the image translation model described in this specification achieves higher translation accuracy than conventional systems.

In addition, the image translation model described in this specification processes text in the entire image as opposed to conventional OCR models that process text on a line-by-line basis. Because the image translation model described in this specification operates on the entire image, it performs translations at a block level. In doing so, the translation is more accurate than conventional systems because the translation preserves the context of a sentence that may be split across multiple lines, which may otherwise be lost when translating on a line-by-line basis (as in conventional OCR models).

The image translation model described in this specification also operates on fewer parameters than conventional systems. For example, conventional systems first use an OCR model to generate text in a source language that is predicted to be present in an input image and then a machine translation model uses this generated text to generate a translation of this text in the target language. In contrast, the end-to-end image translation model described in this specification uses fewer parameters because it directly decodes an input image that depicts input text in a source language, into text in the target language that is a translation of the input text. This further results in reduced latency for the end-to-end image translation model described in this specification relative to the multiple models used in conventional systems.

This specification generally relates to the translation of text included in images from a source language into a target language.

<FIG> is a block diagram of an example environment <NUM> in which an end-to-end image translation model translates text depicted in an image.

The environment <NUM> includes an image source <NUM> and an image translation model <NUM>. The image source <NUM> can be a database or repository of images, or another storage device (e.g., a user device such as a computer or a mobile device) that stores one or more images. A user device is an electronic device that is capable of requesting and receiving content over a network (such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof). Example user devices include personal computers, mobile communication devices, digital assistant devices, and other devices that can send and receive data over the network. A user device typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network, but native applications executed by the user device can also facilitate the sending and receiving of content over the network. Examples of content presented at a user device <NUM> include webpages, word processing documents, portable document format (PDF) documents, images, videos, and search results pages.

The image translation model <NUM><NUM> is a model that that infers, from image features determined from a region of an image that depicts text in a source language, corresponding text in a target language (which is different from the source language). The image translation model <NUM> can be implemented as a supervised or unsupervised machine learning model (or another appropriate statistical model). In some implementations, the image translation model <NUM> of <FIG> is implemented as a convolution neural network.

Unlike conventional solutions, which utilize multiple models with their respective loss functions, the image translation model <NUM> described in this specification is an end-to-end model which utilizes only a single loss function during the image translation model <NUM>'s training. In the implementation of <FIG>, the image translation model <NUM> includes two components: a feature extractor <NUM> and a decoder <NUM>. The structure and operations of the feature extractor <NUM> and the decoder <NUM> are summarized below and are described in greater detail with reference to <FIG>.

The feature extractor <NUM> accepts an image as an input and processes the image to identify a set of image features that are a description of a portion of the input image in which the text is depicted. The feature extractor <NUM> can be implemented as a convolution neural network (CNN). In some implementations, the feature extractor <NUM> is a CNN that is referred to as darknet-<NUM> (which is a hybrid between another CNN referred to as darknet-<NUM> and a residual network). In such implementations, the feature extractor includes successive 3x3 and 1x1 convolution layers with residual connections and contains <NUM> total layers. As such, the CNN of the feature extractor includes fifty-three (<NUM>) layers of convolution, pooling, and residual layers, as depicted in <FIG>. The set of image features output by the feature extractor <NUM> are then reshaped in the form of one or more sequences to be input into the decoder <NUM>. Other dimensions for the network can also be used.

The decoder <NUM> accepts the set of image features as input and outputs/generates text data in the target language that is a translation of the text depicted in the input image. In other words, the decoder <NUM> generates target text based on an input set of image features, without having to generate text in the source language. The decoder <NUM> can be implemented as a six-layer multi-head transformer decoder, which is depicted in <FIG>. Each layer of the transformer decoder includes three sub-layers: a first sublayer that is a multi-head self-attention mechanism, a second sublayer that performs multi-head attention over the output of the feature extractor <NUM>, and a third sublayer that is a position-wise fully connected feed-forward network. Residual connections are employed around each sublayer, and each transformer layer also starts with a layer normalization operation.

In the above-described implementation, the image translation model <NUM> translates text depicted in an image in one source language into one target language. In other words, the image translation model <NUM> only translates text in a particular source language to text in a different target language. As a result, a separate image translation model <NUM> would be required for (<NUM>) translating text from the same source language into a different target language; (<NUM>) translating text from a different source language into the same target language; or (<NUM>) translating text from a different source language into a different target language. Alternatively, the image translation model <NUM> could be trained to translate text depicted in an image from one or more source languages into one or more target languages.

<FIG> is a flow diagram of an example process <NUM> for translating text depicted in an image from a source language to a target language. Operations of the process <NUM> are described below for illustration purposes only. Operations of the process <NUM> can be performed by any appropriate device or system, e.g., using the image translation model described in <FIG> or any other appropriate data processing apparatus. Operations of the process <NUM> can also be implemented as instructions stored on a non-transitory computer readable medium. Execution of the instructions cause one or more data processing apparatus to perform operations of the process <NUM>.

The process <NUM> obtains a first image that depicts a first text written in a source language (at <NUM>). In some implementations, the first image can be obtained from the image source <NUM>. In some implementations, a service executing locally on a user device or on a server (that is remote from the user device) provides a user interface for display on a user device. On this user interface, the user of the user device can input (e.g., as an attachment or as a file upload) a first image (which depicts a first text in the source language) from an image source <NUM>, such as the user device's storage (or from some other local or networked location).

In some implementations, operations <NUM> and <NUM>, which are described below, are performed by the end-to-end image translation model <NUM> described with reference to <FIG>.

The image translation model <NUM> extracts a set of image features from the first image (<NUM>). In some implementations, the process <NUM> inputs the first image (obtained at operation <NUM>) into the feature extractor <NUM> of the image translation model <NUM>. The feature extractor <NUM> is trained (as further described below) to extract, from an input image, a set of image features that are a description of a portion of the input image in which the first text is depicted. As used in this specification, an image feature is a feature of an image that generally represents the shapes, colors, and textures in the portion of the input image in which the text is depicted. The types of image features that the feature extractor <NUM> can extract include, but are not limited to, the beginning of a sentence and/or a paragraph, the end of a sentence, line, and/or a paragraph, the beginning or ending of a line, the spaces between words, the characters/letters/numbers that are identified at particular locations in the text (and/or the probability of each such identified character), and the words that are identified in a string of text (and the probability of each such word; in some implementations, multiple words may be predicted for a particular string of text with a probability representing a likelihood of each predicted word for that string of text). Thus, for the input first image, the feature extractor <NUM> extracts a set of image features that are a description of a portion of the first image in which the text is depicted.

As one example, the feature extractor <NUM> may identify the text "GOOD" included in the first image as a word and may further determine that this word is either "GOOD" (with two zeros) with a <NUM>% probability or "GOOD" with a <NUM>% probability. In such cases, the feature extractor <NUM> outputs both versions of the identified text. In this example, the feature extractor <NUM> outputs the raw image features that are extracted from the image without drawing any inferences about these features. This is different from a conventional OCR model that might infer, albeit incorrectly, that the word is "G00D. " In some implementations, after the feature extractor <NUM> has extracted the first set of image features from the first image, the feature extractor <NUM> flattens this first set of image features into one or more sequences of image features.

The image translation model <NUM> processes the first set of image features to generate a second text that is predicted to be a translation of the first text (at <NUM>). In some implementations, the decoder <NUM> of the image translation model <NUM> receives, as input, the one or more sequences of the first set of image features. The decoder <NUM> is trained (as further described below) to infer text in a target language, which is a predicted translation of text represented by the input set of image features, from an input set of image features. Thus, the decoder <NUM> uses the one or more sequences of the input first image features to generate a second text that is predicted to be a translation of first text from the source language into the target language.

The process <NUM> provides the second text for display on a user interface (at <NUM>). In some implementations, the output of the decoder <NUM> is provided by the service executing locally on the user device or on the server, for display on the user interface presented on the user device.

The following paragraphs describe the training of the image translation model <NUM>.

In some implementations, the feature extractor <NUM> and the decoder <NUM> are trained using the following set of training data that includes: (<NUM>) a set of training images in which each training image depicts input training text that is written in a source language; (<NUM>) a corresponding set of data that includes the image features that are a description of a portion of the input training images in which the input training text is depicted; and (<NUM>) a set of data that includes a set of output training texts that is each written in the target language and is a corresponding translation of an input training text.

In some implementations, at the beginning of the training, the feature extractor <NUM> can be randomly initialized, while the decoder <NUM> can be initialized from a pre-trained text-to-text translation model (e.g., a neural machine translation (NMT) model or another appropriate machine translation model). Such a text-to-translation model is pre-trained to translate text written in the source language into text in the target language. This text-to-text model is trained using a set of input training text data in the source language and a corresponding set of output training text data that is a translation of the input training text data from the source language into the target language.

The feature extractor <NUM> is trained using the set of training images and a corresponding set of data that includes the image features that are a description of a portion of each input training image in which input training text is depicted.

The decoder <NUM>, which is initialized with the pre-trained text-to-text translation model, is further trained to output text in a target language that is predicted to be a translation of text in a source language that is depicted in a set of input training images. Thus, the decoder <NUM> is trained by training the image translation model using the set of input training images that each depict input training text in a source language and a corresponding set of text data that includes a translation in a target language of the input training text included in each of the input training images.

During training, the feature extractor <NUM> and the decoder <NUM> can be trained individually. For example, the feature extractor <NUM> can be trained first for a certain (n) number of iterations (or until a loss function (e.g., the cross entropy loss <NUM>) of the image translation model is below a certain first threshold) and then the decoder <NUM> can be trained for a certain (n) number of iterations (or until a loss (e.g., the cross entropy loss <NUM>) for the image translation model <NUM> is below a certain second threshold that can be the same as or less than the first threshold). Alternatively, the feature extractor <NUM> and the decoder <NUM> can be trained in alternating fashion. In other words, the feature extractor <NUM> can be trained while the decoder <NUM> is held steady, and vice versa. The training of the feature extractor <NUM> and the decoder <NUM> can continue for a certain number (N) total iterations or until a loss function (e.g., the overall cross entropy loss <NUM>) of the image translation model <NUM> satisfies (e.g., is at or below) a pre-determined loss threshold.

Once trained, the image translation model <NUM> can be provided as a package on a user device, which then can be accessed locally by the on-device service referenced in operation <NUM>. Alternatively, the trained image translation model <NUM> can be stored on a network server (or another cloud instance), which then can be accessed by the device's locally executing service (or the service that executes on a remote server) over a network.

<FIG> is block diagram of an example computer system <NUM> that can be used to perform operations described above. The system <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, and an input/output device <NUM>. Each of the components <NUM>, <NUM>, <NUM>, and <NUM> can be interconnected, for example, using a system bus <NUM>. The processor <NUM> is capable of processing instructions for execution within the system <NUM>. In some implementations, the processor <NUM> is a single-threaded processor. In another implementation, the processor <NUM> is a multi-threaded processor. The processor <NUM> is capable of processing instructions stored in the memory <NUM> or on the storage device <NUM>.

In another implementation, the input/output device can include driver devices configured to receive input data and send output data to other input/output devices, e.g., keyboard, printer and display devices <NUM>.

Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks; and CD-ROM and DVD-ROM disks.

Claim 1:
A computer implemented method, comprising:
obtaining a first image that depicts first text written in a source language;
inputting the first image into an image feature extractor that is trained to extract, from an input image, a set of image features that are a description of a portion of the input image in which the text is depicted;
obtaining, from the feature extractor and in response to inputting the first image into the feature extractor, a first set of image features representing the first text present in the first image;
inputting the first set of image features into a decoder, wherein the decoder is trained to infer text in a target language from an input set of image features, wherein the inferred text is a predicted translation of text represented by the input set of image features; and
obtaining, from the decoder and in response to inputting the first set of image features into the decoder, a second text that is in a target language and is predicted to be a translation of the first text;
wherein the feature extractor and the decoder are trained using a single loss function.