Image text recognition

A method and system for analyzing text in an image. Classification and localization information is identified for the image at a word and character level. A detailed profile is generated that includes attributes of the words and characters identified in the image. One or more objects representing a predicted source of the text are identified in the image. In one embodiment, neural networks are employed to determine localization information and classification information associated with the identified object of interest (e.g., a text string, a character, or a text source).

BACKGROUND

Image analysis services are used to analyze an image of a scene and determine if one or more objects are present in the image. Frequently, images include text that is difficult to detect due to the positioning of the text and a lack of context associated with the words or characters of the text.

In many applications (e.g., military, law enforcement, and security applications), detailed information concerning text-based regions of an image is desired. However, due to a lack of granularity in the detection phase, certain image processing systems are limited in the level of information that can be provided regarding portions of an image that include text. Accordingly, conventional image processing systems ineffectively process text-based regions of images and produce a robust set of attributes corresponding to the text within a scene captured by the image.

DETAILED DESCRIPTION

Embodiments described herein relate to recognizing a text string within an image using an image text recognition system including deep neural networks configured to classify and localize the text string at a line, word, and character level. A deep neural network is an artificial neural network with multiple hidden layers of units between an input layer and an output layer. In one embodiment, the image text recognition system includes multiple region proposal and classification neural networks. In one embodiment, the neural networks determine classification information associated with an identified object of interest (e.g., a text string, a character, or a text source) to enable a classification of the object. In one embodiment, a text source refers to an object representing a predicted source of a text string identified within an image. Example text sources include, but are not limited to, a license plate, a road sign, a transaction receipt, an envelope, a business card, etc.)

In one embodiment, localization is performed by the respective neural networks of the image text recognition system to identify region proposals within the image. In one embodiment, a region proposal includes a portion of an image that the image text recognition system predicts includes a text-based object. In one embodiment, localization is performed to identify a position or location of the region proposal within the image. In one embodiment, the neural networks identify one or more region proposals including an object of interest, such as a text string (e.g., a word), character, or text source information).

In one embodiment, the image text recognition system performs classification of the one or more detected objects. For example, a classification or text type may be identified for the contents of a text string (e.g., one or more words or text string configurations). Example classifications may include a license plate number, an address, a phone number, a zip code, a monetary value, etc.)

In one embodiment, the neural networks of the image text recognition system optimize the localization and the classification associated with an image. In one embodiment, a joint loss function is employed to optimize both the localization (e.g., bounding box regression) of a region proposal and classification corresponding to a detected object (e.g., text string, character, or text source) in a substantially simultaneous manner.

In one embodiment, a first neural network, or text string detector, of the image text recognition system receives an image and detects and localizes one or more text strings. In one embodiment, the text string detector, detects and localizes (e.g., identifies a first region of interest predicted to include one or more words) and classifies one or more words within the image.

In one embodiment, a second neural network, or character detector, of the image text recognition system is applied to the first region of interest (e.g., as identified by the text string detector) to detect and localize one or more characters of the identified text string. For example, the character detector detects, localizes, and classifies the one or more characters (e.g., letters, numbers, symbols, punctuation, etc.) of a text string, such as a word.

In one embodiment, a third neural network, or text source detector, of the image text recognition system receives an image and detects and localizes one or more objects within the image that represent predicted sources of text (e.g., text sources). Example text sources include, but are not limited to, a license plate, road sign, receipt, envelope, etc. In one embodiment, the text source may be presented by an object within the image that identifies a potential “source” or context of the text.

In one embodiment, localization performed by the respective neural networks results in an oriented quadrilateral (e.g., a multi-sided polygon with coordinate information) identifying a position or location within the image of a proposed region of interest (e.g., a region including an object such as a text string, a character, or information identifying a text source). In one embodiment, the localization information includes coordinates provided in a clockwise order to enable the orientation of proposed region to be determined.

FIG. 1is a block diagram illustrating various components of an image text recognition system120, according to one embodiment. In one embodiment, the image text recognition system120may include multiple neural networks trained to detect one or more objects in an image110. In one embodiment, example neural networks may include, but are not limited to, region proposal and classification deep neural networks, convolutional neural networks, region-based convolutional neural networks, deep neural networks, etc., or a combination thereof.

In one embodiment, the image text recognition system120may be an online service configured to analyze Unicode text in the image. In one embodiment, the image110may be received by the image text recognition system120via a suitable network140, including, for example, the Internet, intranets, extranets, wide area networks (WANs), local area networks (LANs), wired networks, wireless networks, or other suitable networks, etc., or any combination of two or more such networks.

In one embodiment, the image text recognition system120classifies and localizes (including coordinate information and orientation information) text at a line, text string (e.g., word), and character level for the image110. In one embodiment, the image text recognition system120includes a text string detector124including a neural network trained to detect, localize, and classify one or more text strings (e.g., words) within the image110. In one embodiment, the image text recognition system120includes a character detector126including a neural network trained to detect, localize, and classify one or more characters of the one or more identified text strings. In one embodiment, the image text recognition system120includes a text source detector128including a neural network trained to detect, localize, and classify one or more text sources corresponding to the one or more identified text strings.

In one embodiment, the image text recognition system120includes a pre-processing module122. The pre-processing module122receives the image110and generates multiple resolutions of the image110. In one embodiment, the pre-processing module122resizes the image110to an optimal size and pads the re-sized image for passing to a neural network of the text string detector124for inference processing. In one embodiment, the multiple resolution versions of the image110are provided by the pre-processing module122to the text source detector128and the text string detector124to enable the production of predicted regions of interest (e.g., estimated bounding box coordinates) across multiple image resolution scales that may be combined for a more robust localization estimate by the text string detector124, the character detector126, and the text source detector.

In one embodiment, the text string detector124may be trained to detect one or more text strings in the image110. In one embodiment, the text string detector124is trained using a collection of curated scene and document image datasets, also referred to as “training datasets”. In one embodiment, the training datasets may be annotated with quadrilateral annotations representing bounding box regions including one or more text strings (e.g., words). In one embodiment, the training dataset includes synthetic data and real image data including text collected from one or networks. In one embodiment, the synthetic data of the training dataset may be annotated with the quadrilateral markings during creation of the synthetic data. In one embodiment, the real image data may be annotated via a user-input process, such as a crowdsourcing mechanism (e.g., Amazon® MTurk).

In one embodiment, the training dataset uses a scene text generator to generate synthetic text in an image under multiple photorealistic distortions. In one embodiment, the image text generator (not shown inFIG. 1) processes images at a text string (e.g., word) localization level. For example, for text string localization, multiple “words” or text strings (e.g., thousands of text strings, such as “%1rd29”, “house” johndoe@email.com, “144-41-009312”) are placed in an image. In one embodiment, the image text generator is configured to vary various attributes of the image and text strings, such as, for example, font color, font size, background image, rotation (e.g., perspective transformations), text width, text height, padding (e.g., new lines, indents). In one embodiment, the image text generator adds data augmentation “noise” filters, such as, for example, static blurring, contour embossing, smoothing, sharpening, motion blurring, random levels of compression (e.g., JPEG compression), etc.

In one embodiment, the image text generator (not shown inFIG. 1) processes images at a character localization level. In one embodiment, a set of one or more characters (e.g., 1-30 characters) are placed in an image in structured and unstructured formats, such as, for example, a “free form” format, a “license plate” format, a “phone number’ format, etc. In one embodiment, the image text generator is configured to vary various attributes of the image and characters, such as, for example, font color, font size, background image, rotation (e.g., perspective transformations), text width, text height, padding (e.g., new lines, indents), a language of the characters, etc. In one embodiment, the image text generator adds data augmentation “noise” filters, such as, for example, character borders, strikethroughs, static blurring, contour embossing, smoothing, sharpening, motion blurring, random levels of compression (e.g., JPEG compression), etc.

In one embodiment, the text string detector124receives as an input the image110as processed by the pre-processing module122, and identifies one or more regions of interest (region proposals) within the image110representing a predicted location of a text string (also referred to as a “text string prediction region”). In one embodiment, the text string detector124is configured to use default bounding boxes and aspect ratios that are customizable to support the natural elongation of text strings. In one embodiment, the text string detector124identifies the regions of interest corresponding to a text string and provides the information associated with the regions of interest to the character detector126for character annotation (e.g., letters, numbers, symbols, etc.). In one embodiment, the text string detector124provides the one or more identified text string prediction regions to the character detector126and the profile generator130.

In one embodiment, the character detector126receives as an input the one or more text string prediction regions (e.g., a region within the image predicted to have a text string or word as received from the text string detector124) and predicts a location or region of the image110of one or more characters of the one or more text strings (also referred to as “character prediction regions”). In one embodiment, the character detector126is configured to use default bounding boxes and aspect ratios that are customizable to support the identification of atypical character shapes and sizes (e.g., square-shaped characters). In one embodiment, the character detector128may be executed multiple times corresponding to each of the predicted text string regions identified by the text string detector124. In one embodiment, the character detector128provides the one or more character prediction regions to the profiler generator130.

In one embodiment, the text source generator128receives image110as in input and outputs one or more regions in the image110where one or more text-related image sources are found (also referred to as one or more “text source regions”). Example text-related image sources, or text sources, include a street sign, a license plate, a receipt, etc. In one embodiment, the one or more text source regions are provided to the profile generator130for use in determining attributes corresponding to the text source, as described in greater detail below.

In one embodiment, the profile generator130receives information from the pre-processing module122, the text string detector124, the character detector126and the text source generator128and generates a profile132including text information and attributes relating to the text identified in the image110. In one embodiment, the profile generator130receives: a set of multiple resolutions of the image110from the pre-processing module; one or more text string prediction regions from the text string detector124; and one or more character prediction regions from the character detector126. In one embodiment, the profile generator130aggregates the aforementioned outputs to generate the profile132including text information including a language of the text and an identification of the text at a line, text string (e.g., word), and character level. In one embodiment, the profile generator132determines text attributes corresponding to the identified text. Example text attributes include, but are not limited to, font size, font type, the text source (e.g., a license plate, road sign, billboard, receipt, freeform, etc.), a language of the text (characters and words), orientation information associated with the text, etc. In one embodiment, the profile generator132determines a classification or text type corresponding to the identified text. Example classifications include, but are not limited to, a license plate number, an address, a phone number, a zip code, monetary value, etc.)

In one embodiment, the profile generator130includes a knowledge-graph defining associations between a text source and a classification. For example, the knowledge-graph may include an association between a license plate (text source) and license plate number (classification), defining a relationship indicating that license plate numbers are located within license plates. In one embodiment, the multiple associations may be used to filter and identify the text attributes based on the identified text source. In one embodiment, the profile132may be any suitable file format, such as, for example, a JavaScript Object Notation (JSON) file. In one embodiment, the profile132may be provided by the profile generator130to a computing system (e.g., a computing system that submitted the image110to the image text recognition system120for analysis), stored in a data store (e.g., memory134), or a combination thereof.

In one embodiment, the image text recognition system120includes a processing device138and a memory134configured to execute and store instructions associated with the functionality of the various components, services, and modules of the image text recognition system120, as described in greater detail below in connection withFIGS. 2-6.

FIG. 2illustrates a flowchart that provides an example of a process200executed by an image text recognition system120ofFIG. 1), according to various embodiments. It is understood that the flowchart ofFIG. 2provides an example of the many different types of functional arrangements that may be employed to implement the operation of the image text recognition system120as described herein. Process200may be performed by a processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device), or a combination thereof.

In block210, an object is identified in an image. In one embodiment, the object is detected by a text source detector (e.g., text source detector128ofFIG. 1) including a neural network configured to detect one or more objects in an image. In one embodiment, the one or more objects represent a predicted source of text within the image. For example, for an image of a scene including a street with multiple vehicles, objects including a road sign of the vehicles may be identified in block210.

In block220, a first region within the image is identified. In one embodiment, the first region is a text string prediction region wherein one or more text strings are identified. For example, the first region may be a portion of the image including a text string or word of a road sign (e.g., the text string “STOP”). In one embodiment, the first region may be identified by a text string detector (e.g., the text string detector124ofFIG. 1) including a neural network trained to detect text strings (e.g., words or other text groupings). In one embodiment, the text string detector may receive multiple versions of the image having different resolutions from a pre-processing stage (e.g., processing performed by the pre-processing module122ofFIG. 1). In one embodiment, bounding box coordinates of the first region are identified in block220. In one embodiment, the first region is represented by an oriented quadrilateral (e.g. a four-sided polygon) with coordinates arranged in a clockwise order to enable the orientation of the first region to be determined. In one embodiment, the orientation information may be included in a profile associated with the image.

In block230, one or more text characters of one or more text strings (e.g., words) within the first region are identified. In one embodiment, the one or more characters may include numbers, letters, symbols, etc. that are included within the one or more text strings identified within the first region. In one embodiment, detection and localization corresponding to the one or more characters is performed by a neural network (e.g., character detector126ofFIG. 1) trained to detect and localize characters. In one embodiment, a text string candidate region (e.g., the first region) is provided by a text string generator to a character generator configured to detect and localize the one or more characters within the text string candidate region. In one embodiment, one or more character prediction regions are generated in block230. In one embodiment, the one or more text characters within the first region are localized and classified by a neural network of a character detector (e.g., character detector126ofFIG. 1).

In block240, a profile is generated that includes information corresponding to the text characters based on a stored association of a position of the text characters within the first region and the first object. In one embodiment, an association between the first object (e.g., a predicted text source) and a classification of the characters is used to determine information about the text characters including attributes such as font size, font type, a language of the text characters or words, and an orientation of the text characters or words. For example, a stored association between a road sign (e.g., the first object or text source) and the location of the identified characters within a first region of the road sign may result in the classification of the text string “STOP” and the characters “S”, “T”, “0”, and “P” along with a font type (e.g., highway gothic font of a sans-serif typeface set used in road signage) and font size (e.g., 250 mm character font height). In one embodiment, the profile may be a detailed JSON file including information corresponding to the image, the text (e.g., a language of the text, one or more words, one or more characters, etc.) and text attributes (e.g., font size, font type, text source, etc.).

FIG. 3illustrates a flowchart that provides an example of a process300executed by a neural network (e.g., image text recognition system120ofFIG. 1), according to various embodiments. It is understood that the flowchart ofFIG. 3provides an example of the many different types of functional arrangements that may be employed to implement the operation of the image text recognition system120as described herein. Process300may be performed by a processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device), or a combination thereof.

In block310, a first region within an image is detected by a first neural network of an image text recognition system. In one embodiment, the first region includes a first word (e.g., a recognized grouping or text string). In one embodiment, the first region, also referred to as a text string candidate region, is generated by a region proposal and classification deep neural network trained to detect, localize, and classify words within an image. In one embodiment, the neural network is configured to optimize the localization and classification substantially simultaneously. For example, the neural network may employ a weighted multi-loss function for optimization of a smooth least absolute deviations (L1) loss for localizing (e.g., performing bounding box regression) a quadrilateral corresponding to the first region and a soft-max loss for classification of the object (e.g., the first word). In one embodiment, a joint loss function is optimized, as represented by the following example equation:
Lall=LsmoothL1+∝Lsoftmax;

wherein classification and localization (e.g., bounding box regression) are optimized substantially simultaneously. Alternative joint loss functions and corresponding equations may be employed to optimize the localization and classification information substantially simultaneously.

In block320, one or more characters of the first word are detected by a second neural network. In one embodiment, one or more regions corresponding to predicted characters of the first word (e.g., character prediction regions) are identified. In one embodiment, a neural network trained to detect one or more characters analyzes the first region (as received from the neural network trained for word detection) and predicts a location (e.g., as represented by a bounding box) of characters of the first word. In one embodiment, a region proposal and classification deep neural network trained to detect, localize, and classify characters is applied to the first region. In one embodiment, the character detector neural network is configured to optimize the localization and classification of characters substantially simultaneously. As described above, the neural network may employ a weighted multi-loss function for optimization of a smooth L1 loss for localizing (e.g., performing bounding box regression) a quadrilateral corresponding to the first region and a soft-max loss for classification of the object (e.g., one or more characters).

In block330, an object representing a predicted source of the first word (e.g., a text source) is detected. In one embodiment, a region is identified within the image that corresponds to a text source associated with the first word. For example, the text source may be a transaction receipt and the first word may be “Sale”, “Amount”, “$138.95”, etc. In one embodiment, a region proposal and classification deep neural network (e.g., the text source generator inFIG. 1) trained to detect, localize, and classify one or more text sources is applied to the image. In one embodiment, the character detector neural network is configured to optimize the localization and classification of characters substantially simultaneously. As described above, the neural network may employ a weighted multi-loss function for optimization of a smooth L1 loss for localizing (e.g., performing bounding box regression) a quadrilateral corresponding to a text source candidate region and a soft-max loss for classification of the object (e.g., a text source).

In block340, the outputs of the text string detector (e.g., the word prediction region and text string classification information), the character detector (e.g., the character prediction regions and character classification information), and the text source detector (e.g., the text source prediction region and text source classification information) are provided to a profile generator for the generation of a profile corresponding to the image. In one embodiment, the profile generator analyzes the aforementioned information and generates the profile including text information (e.g., localization and classification information corresponding to the first word, the one or more characters, a language) and text attributes (e.g., a font size, font type, text source). In one embodiment, the profile (e.g., a detailed file) may be provided to a computing system (e.g., a computing system associated with the submission of the image for analysis) and stored in a data store. In one embodiment, a knowledge graph of associations between the text source and classifications of the first word and characters may be used to determine and refine the text attributes. In one embodiment, any suitable configuration or format of the profile may be employed, including, for example, a JSON file.

FIG. 4illustrates an example of aspects of an image text recognition system configured to process an image410to generate a corresponding profile432. As illustrated, this example relates to the processing of an image410of a scene including a vehicle. In one embodiment, the image410is received by a text string detector424and a text source detector428.

In one embodiment, the text string detector424includes a text string neural network trained to detect one or more text strings (e.g., words) within the image410. In one embodiment, the text string detector424is configured to include a localization and classification optimizer450. In one embodiment, the localization and classification optimizer450may include a set of instructions executable by a processing device to optimize text string localization451and text string classification452substantially simultaneously. As shown in the example inFIG. 4, the text string detector424may identify multiple text string prediction regions within the image410, including a first text string prediction region corresponding to a text string “A42 DRS”. In one embodiment, the results of the processing of the image410by the text string detector424(e.g., the one or more text string prediction regions, text string localization information, and text string classification information) are provided to a profile generator430.

In one embodiment, the text source generator428performs analysis on the image410using a text source neural network429(e.g., a region proposal and classification deep neural network) trained to detect and localize one or more text sources. In one embodiment, the text source neural network429includes a localization and classification optimizer460programmed to optimize text source localization461and text source classification462substantially simultaneously. As shown in the example inFIG. 4, the text source detector428may identify multiple text source prediction regions within the image410, including a first text source prediction region corresponding to a region of the image of the vehicle corresponding to the vehicle's license plate. In one embodiment, the results of the processing of the image410by the text source detector428(e.g., the one or more text source prediction regions, text source localization information, and text source classification information) are provided to the profile generator430.

In one embodiment, as shown inFIG. 4, the one or more text string prediction regions are provided to a character detector426. In one embodiment, the character detector426includes a character neural network427(e.g., a region proposal and classification deep neural network) trained to detect and localize one or more characters. In one embodiment, the text source neural network429includes a localization and classification optimizer470programmed to optimize character localization471and character classification472substantially simultaneously. As shown in the example inFIG. 4, the character detector426may identify multiple character prediction regions within the one or more text string prediction regions. In this example, the character detector426detects, localizes, and classifies characters corresponding to character prediction regions “A”, “4”, “2”, “D”, “R”, and “S” of the text string candidate region “A42 DRS”. In one embodiment, the results of the processing of the one or more text string prediction regions by the character detector426(e.g., the one or more character prediction regions, character localization information, and character classification information) are provided to the profile generator430.

In one embodiment, the profile generator430analyzes the information received from the text string detector424, text source detector428, and character detector426to generate a profile432corresponding to the image410. For example, the profile432may be a file (e.g., a JSON file) including information identifying the text string prediction regions, the text string localization451, the text string classification452, the text source prediction region(s), the text source localization461, the text source classification462, the character prediction regions, the character localization471, and the character classification472. In one embodiment, the profile generator430is a set of instructions executable by a processing device to determine text information at a line, word, and character level and text attributes (e.g., font size, font type, text source) corresponding to the text identified within the image410.

FIG. 5illustrates an example interface500of an image text recognition system, according to various embodiments. In one embodiment the interface500includes a portion wherein a user may submit an image510for analysis by the image text recognition system. In this example, instructions530are provided to assist a user in the upload or provisioning of the image510. In one embodiment, the image text recognition system receives the image510and performs operations and functions (as described in detail above with respect toFIGS. 1-5) to identify multiple text string prediction regions (e.g., text string prediction region520. In one embodiment, the image text recognition system generates an output identifying the text string “warning!”540.

FIG. 6illustrates a diagrammatic representation of a machine in the example form of a computer system600including a set of instructions executable by an image text recognition system120to cause the system to perform any one or more of the methodologies discussed herein. In one embodiment, the neural network may include instructions to enable execution of the processes and corresponding components shown and described in connection withFIGS. 1-5.

The example computer system600includes a processing device (processor)602, a main memory604(e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory606(e.g., flash memory, static random access memory (SRAM)), and a data storage device618, which communicate with each other via a bus630.

The computer system600may further include a network interface device608. The computer system800also may include a video display unit610(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device612(e.g., a keyboard), a cursor control device614(e.g., a mouse), and a signal generation device616(e.g., a speaker).

The data storage device618may include a computer-readable medium628on which is stored one or more sets of instructions of the image text recognition system120embodying any one or more of the methodologies or functions described herein. The instructions may also reside, completely or at least partially, within the main memory604and/or within processing logic626of the processing device602during execution thereof by the computer system600, the main memory604and the processing device602also constituting computer-readable media.

The preceding description sets forth numerous specific details such as examples of specific systems, components, methods, and so forth, in order to provide a good understanding of several embodiments of the present disclosure. It will be apparent to one skilled in the art, however, that at least some embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known components or methods are not described in detail or are presented in simple block diagram format in order to avoid unnecessarily obscuring the present disclosure. Thus, the specific details set forth are merely presented as examples. Particular implementations may vary from these example details and still be contemplated to be within the scope of the present disclosure. In the above description, numerous details are set forth.

It will be apparent, however, to one of ordinary skill in the art having the benefit of this disclosure, that embodiments of the disclosure may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the description.