Method and system for document data extraction

Certain aspects of the present disclosure provide techniques for extracting data from a document. An example method generally includes identifying a bounding polygon of the region from an electronic image of the document and extracting data from within the bounding polygon of the region. The method further includes generating revised extracted data based on the extracted data, and combining the revised extracted data with other data extracted from the electronic image of the document to generate input data for a data processing application.

INTRODUCTION

Aspects of the present disclosure relate to extracting data from a document, and more specifically to deep learning solutions for document data extraction in specialized applications.

BACKGROUND

Documents and forms may be used to record or reflect information or data about a person, business, event, or other matter. The document may contain fields for specific types of information. Each field may have a label allowing a user to identify it or instructing the user regarding what type of information is held within or should be entered into the field. Depending on the type of document, a single field may have multiple sections for multiple inputs related to the subject of the field. In some cases, users seek to digitize documents to make them more searchable, usable, or accessible. In many instances, this is done by uploading a photo of the document, and then utilizing a software application with a generic extractor in order to extract the information from the document (e.g., via optical character recognition (OCR) techniques).

In some instances, the generic extractor is unable to correctly extract data from a field or fields of the document. This may be due to empty fields, incorrect inputs (i.e. an incorrect code or value associated with the type of information in that field), abnormal field shapes (such as an irregular polygon or field divided into multiple subfields), or any type of additional complexity of the field. The generic extractor may not work on such fields even while it works on most fields in the document.

Accordingly, techniques are needed to extract data from a document in instances where a generic extractor cannot correctly extract the data.

BRIEF SUMMARY

Certain embodiments provide a method of extracting data from a document, wherein the document comprises a region containing optional data. The method generally includes identifying a bounding polygon of the region from an electronic image of the document and extracting data from within the bounding polygon of the region. The method further includes generating revised extracted data based on the extracted data, and combining the revised extracted data with other data extracted from the electronic image of the document to generate input data for a data processing application.

Other embodiments provide a method of document data extraction. The method generally includes identifying from an electronic image of a document including a region containing optional data, a bounding polygon of the region in the document by performing image segmentation of the document and predicting whether the region contains data. Based on the prediction, data from within the bounding polygon of the region is extracted by performing optical character recognition. The method further includes generating revised extracted data based on the extracted data and combining the revised extracted data with other data extracted from the electronic image of the document into a structured data format representing the document to generate input data for a data processing application based on the document. The input data is then provided to the data processing application.

DETAILED DESCRIPTION

Aspects of the present disclosure provide apparatuses, methods, processing systems, and computer-readable mediums for extracting data from a document, and more specifically for applying machine learning to document data extraction in specialized applications.

As noted above, users seek to digitize documents for a variety of reasons, including making the documents more searchable, usable, or otherwise accessible. Documents and forms may generally contain a variety of fields for different types of information specific to the document type. Depending on the document type, fields may take on different formats, hold different types of data, and vary in size, shape, and type within the same document. A user seeking to digitize their document may upload a photo or scan of the document, and then extract data from the document via a generic data extractor software application. However, depending on the varying fields, a generic data extractor may not be able to correctly extract data (e.g., due to empty fields, incorrect inputs, abnormal field shapes, or any type of additional complexity of the field).

The present disclosure introduces techniques that allow for data extraction in the instances where conventional data extraction techniques will not work. Once a document image is uploaded to a software application for extraction, the standard fields (or those without abnormal formatting or data inputs) can be run through a generic extractor, while a specialized extractor can be utilized for the fields or regions that are not compatible with the generic extractor. The specialized extractor can utilize machine learning models to identify a bounding polygon of the region containing the desired data, and then extract and correct the data. The extracted data from the region can be combined with the data extracted by the generic extractor, and combined such that the whole document is digitized.

A separate extractor for certain regions of the document that are difficult to digitize can more accurately extract data, allowing digitization of documents with more complex fields, or fields of varying types. This allows for a more functional application as well as a better user experience, by digitizing the document with the use of a single application, with less need to verify the extracted data for accuracy and completion. It also requires less follow-up by the user, where previously a user would be required to manually enter such un-extractable data. As such, embodiments of the present disclosure provide a better, faster, and more accurate solution for data extraction.

FIG.1illustrates an example flow chart illustrating the extraction of data from a document image for a data processing application.

A document image105may be uploaded to one or more document data extraction modules for processing. The document that is the subject of document image105may generally be any type of document or form having fields which potentially contain data. The document image105can be uploaded in any manner, such as uploading a photo of the document. The document image105may be read into both a specialized extraction module and a generic extraction module180, depending on the type of field and data that will need to be extracted from an image. The software application (comprising the two extraction modules mentioned) then extracts information from the document, for example by using optical character recognition (OCR) techniques.

As noted above, the document image105may be uploaded to generic extraction module180. However, in some instances, the generic extractor is unable to correctly extract data from a field or fields of the document. This can occur for any number of reasons, for example an abnormal field shape, improper input, or complex field data.

In cases where the generic extraction module180won't be able to extract data from a certain field, the document image105may be processed by a bounding polygon location identification module130. For example, bounding polygon location identification module130can identify a bounding polygon around one or more fields and/or a region of the document. The bounding polygon may be any regular or irregular polygon, such as a rectangle or a trapezoid. Once the module130identifies the bounding polygon, it extracts bounding polygon data135. For example, bounding polygon data135may include coordinates of the bounding polygon and/or data from inside the region bounded by the bounding polygon (e.g., text data, image data, and/or the like). Identifying the location of the region in the document can be done by segmentation of the document image and/or by OCR. In some aspects, this step helps to eliminate noise in the data that is being processed.

Bounding polygon data135is then processed by a data extraction module140. The data extraction module140is configured to extract data from within the bounding polygon of the region. The extracted data145is then output into a data revision module150. The data extraction module140may comprise a deep learning extractor. For example, the data extraction module140may be based on LayoutLMv2, which is a model that is based on multi-modal pre-training of text and layout for information extraction from a document. The data extraction module140operates on the bounding polygon data135(or segmented data) yielded by the bounding polygon location identification module130. The data extraction module140may be trained using information specific to one or more fields within the bounding polygon, in order to produce an output from those fields. In training data extraction module140, training data specific to the field or region data is being extracted from helps tune the model comprising the bounding polygon location identification module130, the data extraction module140, and the data revision module150(discussed below), in order to build a model artifact optimized specifically for that field or region.

The data revision module150eliminates any incorrect input from the extracted data. For example, if an incorrect code for the corresponding field is input, the data revision module150will identify and eliminate the incorrect code (e.g., either removing the incorrect code altogether or replacing the incorrect code with a known correct code, such as a most similar correct code with similarity being determined based on an edit distance). After the data is revised to correct for incorrect inputs, the revised extracted data155is output to a data combination module160.

Data combination module160receives extracted data from the generic extraction module180(corresponding to data from outside of the region) and the revised extracted data155. Data combination module160then combines the data in order to output combined data165corresponding to the whole document. For example, the data extracted from a first set of fields using generic extraction module180and revised extracted data155(e.g., which was extracted from a second set of fields from which data could not be extracted using the generic extractor) may be combined to form the complete set of data extracted from the document. The combined data165is then sent to a data processing application170for further processing.

In some embodiments, data may be extracted for fields in the region using both the generic and the specialized extractor. In such cases, the revised extracted data155may be used to override any corresponding data extracted using the generic extractor from the same fields. Furthermore, confidence scores from one or both extractors may be used to determine which data to incorporate into combined data165. For example, if the specialized extractor returns a higher confidence score for a field in the region than the generic extractor, if the difference between confidence scores returned by the two extractors is above a threshold, and/or if the confidence score returned by the specialized extractor is at least greater than a minimum threshold, then the revised extracted data155determined based on the specialized extractor may be used.

In some aspects, the functionality performed by bounding polygon location identification module130, data extraction module140, and data revision module150may be associated with a first machine learning model (e.g., which may be a single model or an ensemble of models). Additionally, the functionality of data combination module160, which combines extracted data from the generic extraction module180with the revised extracted data155, may be associated with a second machine learning model. In another aspect, some or all of the steps illustrated byFIG.1may be associated with separate machine learning models.

In some aspects, there may be an additional module (not pictured) that predicts whether the region from which data cannot be extracted using a generic extractor is empty. As noted above, this region is identified by a bounding polygon of the region containing the desired data for extraction. If the region is empty, then only generic extraction module180is necessary to extract data from document image105.

FIG.2Aillustrates an example of a document data extraction module that is implemented within a service separate from an application. In this example, application210sends document image205to an outside service220. The document image205corresponds to document image105ofFIG.1. In this example, the service implements the document data extraction module240, which may implement the bounding polygon location identification module130, data extraction module140, and data revision module150(or equivalent functions) ofFIG.1. The service220then sends extracted data255back to the application210, where the extracted data may be combined with other data from a generic extractor.

In another aspect, depicted inFIG.2B, the document data extraction module270is implemented with an application260. In this alternative example, the application260implements the document data extraction module270, which may implement the bounding polygon location identification module130, data extraction module140, and data revision module150(or equivalent functions) ofFIG.1.

FIG.3illustrates an example document from which data may be extracted according to techniques described herein. For example, the document300may comprise a W-2 form used for tax preparation. A user seeking to file their taxes may wish to upload the document300to a service that digitizes the filing of the W-2300. Accordingly, it is imperative that each region of the W-2300containing data is accurately digitized. A generic extractor may be able to extract data from the standard fields of the document, but a specialized extractor may be required for problematic regions. For example, one such region that may require a specialized extractor model may correspond to “box12” indicated by boundary310. In this example, box12comprises a nonstandard format that is different from the rest of the fields of W-2300, containing multiple columns with specific allowed codes. This variance in field format, size, and data entry type makes the fields within boundary310more complex as compared to the rest of the fields of the W-2. This introduces difficulty in accurately extracting data from the document. Accordingly, the present disclosure provides a method for extracting data from the region corresponding to boundary310, which is then combined with other data extracted from W-2300. As illustrated here, the bounding polygon of boundary310may comprise a rectangle which encloses “box12” in the form and its relevant data entry.

In some aspects, other boxes within the document300may require a specialized extractor. For example, Box14320allows for open, non-standardized input, which may not be accurately extracted by a generic extractor. The present invention may be applicable to any fields of exemplary document300, or any other document, which a generic extractor has difficulty reading.

Example Methods for Extracting Data from a Document

FIG.4illustrates example operations for extracting data from an electronic image of a document according to embodiments described herein.

As illustrated, operations400begin at step410with identifying a bounding polygon of a region from the image, such as by bounding polygon location identification module130ofFIG.1. In some aspects, identifying the bounding polygon includes performing image segmentation. The region may correspond, for example, to boundary310of the W-2 document300depicted and described with respect toFIG.3. The region may, for example, comprise optional data. As used herein, a region of a document that comprises “optional data” refers to a region that may or may not include data, such as because the region includes data in some cases and does not include data in other cases.

Operations400then proceed to step420with extracting data from within the bounding polygon of the region. In some aspects, this is performed by data extraction module140ofFIG.1. In some aspects, extracting data from within the bounding polygon of the region includes performing optical character recognition of the data within the bounding polygon of the region.

Operations400then proceed to step430with generating revised extracted data based on the extracted data. For example, this may be performed by data revision module150ofFIG.1. In some aspects, generating revised extracted data based on the extracted data includes eliminating incorrect output from the extracted data.

For example, incorrect output may comprise an invalid code in the case of boundary310(e.g., indicating “box12” in the form) of W-2300as illustrated inFIG.3. For example, because the set of valid “box12” codes is limited, a comparison of the extracted data to a list of valid box12codes would indicate an incorrect input. This is one example, and many other examples of incorrect input are possible. In some aspects, eliminating incorrect output from the extracted data comprises comparing the extracted data to a list of known valid data (e.g., known valid box12codes) in order to eliminate some or all codes if they are found to be invalid.

In some aspects, steps410,420, and430are performed using one or more machine learning models (e.g., each of which may be a single model or an ensemble model comprising multiple models). In one embodiment, step410is performed using a first machine learning model, step420is performed using a second machine learning model, and step430is performed using a third machine learning model. In one example, step430is performed by the same machine learning model as step420. In yet another example, step430is performed using rules, and does not involve the use of a machine learning model.

Operations400then proceed to step440with combining the revised extracted data with other data extracted from the electronic image of the document to generate input data for a data processing application. In some aspects, step440corresponds to functionality performed by data combination module160ofFIG.1. In some aspects, step440comprises combining the revised extracted data with other data extracted from the electronic image of the document into a structured data format representing the document to generate input data for a data processing application based on the document. A structured data format representing a document generally refers to a data structure that includes variables to be populated with data extracted from the document. Examples of structured data formats include extensible markup language (XML), Javascript object notation (JSON), hypertext markup language (HTML), and/or the like.

In some aspects, the other data is extracted using one or more machine learning models. For example, the other data may comprise data from regions of the document outside of the identified bounding polygon of the region. For example, in the case of W-2300as illustrated inFIG.3, the generic extractor may be utilized for regions outside of boundary310(e.g, representing “box12” in the form). For example, data from one or more of boxes1-11in the form may be extracted via a generic extractor.

In some aspects, operations400may further comprise providing the input data to the data processing application. The data processing application may correspond to data processing application170ofFIG.1. For example, using a W-2300as illustrated inFIG.3, the data processing application may be one utilized for filing taxes, such as TurboTax.

In some aspects, operations400may further comprise predicting whether the region contains data. In some embodiments, predicting is performed using a machine learning model and/or a rule-based model. Such models may use text extracted using OCR (e.g., determining whether there is something that looks like “box12” in the text from the document, or the text from the region) and/or extracted values from other fields (e.g., a rule may indicate that a W-2 with certain other fields populated, or certain amounts in those fields, typically has content in box12, or such a correlation may be indicated in training data for a machine learning model based on historical data). An affirmative prediction can be made, for example, if a confidence score output by a machine learning model is above a certain threshold and/or if a rule is satisfied.

Again utilizing the example illustrated inFIG.3, not every field in a standard W-2300will contain data. In particular, “box12” is an example of a field that more often than not, will not contain any data. In this example, operations400would include predicting whether box12contains data. This is one example, and many other examples of forms and documents in which operations400would comprise predicting whether a field contains data are possible.

FIG.5also illustrates example operations for extracting data from an electronic image of a document, wherein the document includes a region containing optional data.

Operations500begin at step510with identifying from the image a bounding polygon of the region in the document by performing image segmentation of the document, such as by bounding polygon location identification module130ofFIG.1.

Operations500then proceed to step520with predicting whether the region contains data.

Operations500then proceed to step530with extracting, based on the prediction, data from within the bounding polygon of the region by performing optical character recognition. For example, step530(and, in some embodiments, subsequent steps) may only be performed if it is predicted in step520that the region contains data. In some aspects, step530is performed by data extraction module140ofFIG.1. For example, step530may comprise extracting data from within boundary310of a W-2300, as illustrated inFIG.3.

Operations500then proceed to step540with generating revised extracted data based on the extracted data. For example, step540may be performed by data revision module150ofFIG.1, and may comprise eliminating incorrect output from the extracted data.

Operations500then proceed to step550with combining the revised extracted data with other data extracted from the electronic image of the document into a structured data format (e.g., including variables corresponding to fields of the document that are populated based on the revised extracted data and/or the other data extracted from the electronic image of the document) representing the document to generate input data for a data processing application based on the document. In some aspects, step550is performed by data combination module160ofFIG.1.

Operations500then proceed to step560with providing the input data to the data processing application. The data processing application may correspond to data processing application170ofFIG.1.

In some aspects, some or all of steps510,530, and540are performed by a first machine learning model (which implements, for example, functionality associated with bounding polygon location identification module130, data extraction module140, and/or data revision module150ofFIG.1). In these example aspects, other data extracted from the electronic image of the document is extracted by a second machine learning model (which implements, for example, functionality associated with generic extraction module180ofFIG.1).

Example System

FIG.6illustrates an example computing system with which embodiments of the present disclosure may be implemented. The system comprises a processor configured to execute executable instructions to cause the system to perform the operations illustrated inFIG.1. In one example, system600may comprise application210ofFIG.2Aand/or application260ofFIG.2B.

As shown, system600includes a central processing unit (CPU)602, one or more I/O device interfaces604that may allow for the connection of various I/O devices614(e.g., keyboards, displays, mouse devices, pen input, etc.) to the system600, network interface606through which system600is connected to network690(which may be a local network, an intranet, the internet, or any other group of computing devices communicatively connected to each other), a memory608, and an interconnect612.

CPU602may retrieve and execute programming instructions stored in the memory608. Similarly, the CPU602may retrieve and store application data residing in the memory608. The interconnect612transmits programming instructions and application data, among the CPU602, I/O device interface604, network interface604, and memory608.

CPU602is included to be representative of a single CPU, multiple CPUs, a single CPU having multiple processing cores, and the like.

Memory608is representative of a volatile memory, such as a random access memory, or a nonvolatile memory, such as nonvolatile random access memory, phase change random access memory, or the like. As shown, memory608includes modules corresponding to those illustrated inFIG.1.

Generally, the system600allowed for users to upload a document image which will be processed via document reader620. Bounding polygon location identification module630may correspond to bounding polygon location130identification module ofFIG.1. Data extraction module640may correspond to data extraction module140ofFIG.1. Data revision module650may correspond to data revision module150ofFIG.1. Data combination module660may correspond to data combination module160ofFIG.1. Data processing module670may correspond to data processing module170illustrated inFIG.1.

In some examples, as illustrated inFIG.2, one or more aspects of a document data extraction process may be implemented via an outside service, and thus bounding polygon location identification module630, data extraction module640, and data revision module650may not be run on the same device (e.g., a local device) that is capturing the image of a document and running an application intended to utilize the data extracted from the image of the document. For example, where document data extraction is performed by an external service external from a device, document reader620and data combination module660may instead be performed in an application local to the device.

Example Clauses

Clause 1: A method of extracting data from a document, wherein the document comprises a region containing optional data, the method comprising: identifying a bounding polygon of the region from an electronic image of the document, extracting data from within the bounding polygon of the region, generating revised extracted data based on the extracted data and combining the revised extracted data with other data extracted from the electronic image of the document to generate input data for a data processing application.

Clause 2: The method of Clause 1, further comprising providing the input data to the data processing application.

Clause 3: The method of Clause 1 or 2, wherein the extracting of the data from within the bounding polygon of the region is performed using a first machine learning model.

Clause 4: The method of Clause 3, wherein the other data extracted from the electronic image of the document is extracted using a second machine learning model.

Clause 5: The method of any one of Clauses 1 through 4, wherein identifying a bounding polygon of the region in the document comprises performing image segmentation on the electronic image.

Clause 6: The method of any one of Clauses 1 through 5, wherein extracting data from within the bounding polygon of the region comprises performing optical character recognition.

Clause 7: The method of any one of Clauses 1 through 6, wherein generating revised extracted data comprises eliminating incorrect output from the extracted data.

Clause 8: The method of any one of Clauses 1 through 7, further comprising predicting whether the region contains data.

Clause 9: The method of any one of Clauses 1 through 8, wherein the region corresponds to Box12of a W-2.

Clause 10: The method of any one of Clauses 1 through 9, wherein combining the revised extracted data with other data extracted from the electronic image of the document comprises constructing a structured data format representing the document.

Clause 11: A method of extracting data from a document, wherein the document comprises a region containing optional data, the method comprising: identifying a bounding polygon of the region from an electronic image of the document by performing image segmentation, predicting whether the region contains data, extracting, based on the prediction, data from within the bounding polygon of the region by performing optical character recognition, generating revised extracted data based on the extracted data, combining the revised extracted data with other data extracted from the electronic image of the document into a structured data format representing the document to generate input data for a data processing application based on the document, and providing the input data to the data processing application.

Clause 12: The method of Clause 11, wherein the extracting of the data from within the bounding polygon of the region is performed using a first machine learning model.

Clause 13: The method of Clause 12, wherein the other data extracted from the electronic image of the document is extracted using a second machine learning model.

Clause 14: The method of any one of Clauses 11 through 13, wherein generating revised extracted data comprises eliminating incorrect output from the extracted data.

Clause 15: A system, comprising: a memory having executable instructions stored thereon; and a processor configured to execute the executable instructions to cause the system to: identify a bounding polygon of a region of a document from an electronic image of the document; extract data from within the bounding polygon of the region; generate revised extracted data based on the extracted data; and combine the revised extracted data with other data extracted from the electronic image of the document to generate input data for a data processing application.

Clause 16: The system of Clause 15, wherein the processor is further configured to provide the input data to the data processing application.

Clause 17: The system of any one of Clauses 15 through 16, wherein extracting the data from within the bounding polygon of the region is performed using a first machine learning model.

Clause 18: The system of Clause 17, wherein the other data extracted from the electronic image of the document is extracted using a second machine learning model.

Clause 19: The system of any one of Clauses 15 through 18, wherein the processor is configured to perform image segmentation on the electronic image in order to identify the bounding polygon.

Clause 20: The system of any one of Clauses 15 through 19, wherein the processor is configured to perform optical character recognition in order to extract data from within the bounding polygon of the region.

Clause 21: An apparatus comprising: a memory having executable instructions stored thereon; and a processor configured to execute the executable instructions to cause the apparatus to perform the operations of any one of Clauses 1 through 20.

Clause 22: An apparatus comprising: means for performing the operations of any one of Clauses 1 through 20.

Clause 23: A computer-readable medium having instructions stored thereon which, when executed by a processor, performs the operations of any one of Clauses 1 through 20.

Additional Considerations