Automatic delineation and extraction of tabular data using machine learning

A computer-implemented method for using a machine learning model to automatically extract tabular data from an image includes receiving a set of images of tabular data and a set of markup data corresponding respectively to the images of tabular data. The method further includes training a first neural network to delineate the tabular data into cells using the markup data, and training a second neural network to determine content of the cells in the tabular data using the markup data. The method further includes, upon receiving an input image containing a first tabular data without any markup data, generating an electronic output corresponding to the first tabular data by determining the structure of the first tabular data using the first neural network and extracting content of the first tabular data using the second neural network.

BACKGROUND

The present invention relates generally to computer technology, and more particularly to a content recognition system that automatically determines a structure that is used to present information as tabular data and further determines the content in the tabular data.

A substantial amount of literature, including books, papers, articles, blogs, reports, etc., is generated in any field, such as medicine, law, engineering, science, business, etc. In addition, literature is generated as part of commerce, e.g. invoices, quotes, account statements, contracts, etc. All such literature is typically written for exchange between persons without any plan for machine understanding. The documents, for purposes of differentiation, are described as “natural language” documents as distinguished from documents or files written for machine readability and understanding. With the advent of artificial intelligence and other advances in machine learning, machines, such as computers, can analyze the corpus of information from the literature to provide insights that may not be detectable by humans. Various document parsing and analysis systems exist that facilitate deciphering contents of the literature and provide functionality such as search capability.

SUMMARY

According to one or more embodiments of the present invention, a computer-implemented method for using a machine learning model to automatically extract tabular data from an image includes receiving a set of images of tabular data and a set of markup data corresponding respectively to the images of tabular data. The method further includes training a first neural network to delineate the tabular data into cells using the markup data, and training a second neural network to determine content of the cells in the tabular data using the markup data. The method further includes, upon receiving an input image containing a first tabular data without any markup data, generating an electronic output corresponding to the first tabular data by determining the structure of the first tabular data using the first neural network and extracting content of the first tabular data using the second neural network.

According to one or more embodiments of the present invention, a system includes a memory, and a processor coupled with the memory. The processor is programmed with machine learning algorithms to perform a method that includes training a machine learning model. The training includes receiving a set of images of tabular data and a set of markup data corresponding respectively to the images of tabular data. The training further includes training a first neural network to delineate the tabular data from the set of images into cells using the markup data. The training further includes training a second neural network to determine content of the cells in the tabular data from the set of images using the markup data. The method, after the training, further includes, upon receiving an input image containing a first tabular data without markup data indicative of a structure of the first tabular data, generating an electronic output corresponding to the first tabular data by determining the structure of the first tabular data using the first neural network and extracting content of the first tabular data using the second neural network.

According to one or more embodiments of the present invention, a computer program product includes a memory storage device having computer executable instructions stored thereon. The computer executable instructions when executed by a processor cause the processor to perform a method that includes training a machine learning model. The training includes receiving a set of images of tabular data and a set of markup data corresponding respectively to the images of tabular data. The training further includes training a first neural network to delineate the tabular data from the set of images into cells using the markup data. The training further includes training a second neural network to determine content of the cells in the tabular data from the set of images using the markup data. The method, after the training, further includes, upon receiving an input image containing a first tabular data without markup data indicative of a structure of the first tabular data, generating an electronic output corresponding to the first tabular data by determining the structure of the first tabular data using the first neural network and extracting content of the first tabular data using the second neural network.

Embodiments of the present invention facilitate the use of machine learning to automatically parse and understand tabular data in digital literature and in turn to use the literature for complete analysis. According to one or more embodiments of the present invention tabular data in the literature is deciphered despite the tabular data being presented in various layouts, styles, information type and format, and without explicit description of the encoding/formatting information about the structure of the tabular data. Accordingly, embodiments of the present invention facilitate improvements to automatic content recognition systems that are currently unable to automatically decipher tabular data presented.

In the accompanying figures and following detailed description of the disclosed embodiments, the various elements illustrated in the figures are provided with two or three-digit reference numbers. With minor exceptions, the leftmost digit(s) of each reference number corresponds to the figure in which its element is first illustrated.

DETAILED DESCRIPTION

Published literature includes vital information presented in tabular format (“tables”), that is, as tabulated data included in the document(s). The literature can include information presented in the form of a table that organizes the information across multiple rows and columns. Such tabular data facilitates presenting information in a structured manner, summarizing key information, and presenting results and/or observations. Typically, the tabulated data is included in the document as an image, without any corresponding information that describes the structure that is used to tabulate the data. The structure indicates how the data is delineated, for example, into rows, columns, cells, and other such components of the table.

In many cases, understanding the information in tables is central to analyzing the various documents in the literature. Accordingly, it is necessary in machine learning to read and understand tabular data to use the literature for complete analysis. However, a technical challenge exists in deciphering tabular data in literature because of the tabular data being presented in various layouts, styles, information type and format, and without explicit description of the encoding/formatting information about the structure of the tabular data. While humans are adept at reading such varied tabular data layouts and styles, deciphering the tabular data is a technical challenge. Some examples of variability in tables may include, different heights of rows and columns, merging of first neural network memory cells, different number of columns, different number of rows in different columns, or different types of borders distinguishing the cells, etc.

For example, in the case of medical literature such as clinical studies, information regarding results across different groups is contained in such tables. It is understood that medical literature is just one example, and that tabulated data is presented in literature in various other fields, such as engineering, science, business, law, etc., and the embodiments of the present invention are not limited to using documents from any particular field.

Further, in multiple cases, the data that is presented in a document using the tabulated format is not present in the remaining text of the document. Hence, for an automatic content recognition system that parses available literature autonomously to decipher the contents of the document, a technical challenge exists to decipher such tabulated data. Currently, it is a technical challenge to parse such tabulated data because there is no standard format or structure that is used for tabulating the data. The data is typically tabulated for human understanding, and humans are generally adept at parsing such data that is formatted in tabular format using ad hoc structures, and without having to be provided a description of the formatting.

Embodiments of the present invention address such technical challenges and facilitate machines to autonomously understand the structure (or format) of tabulated data in order to extract meaning of the tabulated data. One or more embodiments of the present invention facilitate training a machine learning (ML) model to receive tabulated data without any description of the structure or delineation or labels specifying the formatting of the tabulated data, to identify discrete columns, rows, and/or cells, and to extract the content (e.g. text, images etc.) that is included in the table. In one or more embodiments of the present invention the extracted content is output in a structured and meaningful format.

In one or more embodiments of the invention, the ML model receives the tabulated data in the form of digital images of tables that do not have any delineation or labels specified. For example, the digital images can be in any digital image file format such as bitmap, Joint Photographic Experts Group (JPEG), portable network graphic (PNG), or any other format. In one or more examples, the images of tabulated data are extracted from documents, which in turn are in any digital content format, such as portable document format (PDF), DOC, and the like. The ML model can then identify discrete columns, rows, and cells, and extract the text included therein in a structured and meaningful way. In one or more embodiments of the present invention, the extracted data is output using computer readable format such as hypertext markup language (HTML), extended markup language (XML), or any other such computer readable format.

FIG. 1depicts a system100for content recognition according to one or more embodiments of the present invention. The system100can be the content recognition workload96depicted inFIG. 8. Alternatively, as noted herein, the system100can be a stand along system used for content recognition. The system100includes, among other components a content recognition device120that receives an input image112of tabulated data from an electronic document110. The input image112that is received is not accompanied with any structural information that describes the format of the tabulated data in the input image112.

The content recognition device120uses an ML model122to parse the content of the input image112and generate an electronic output130that includes the tabulated data in markup format132. The tabulated data in the markup format132includes a structure of the tabular data in delineated format, for example, describing the cells in the tabular data. For example, the delineated format describes the rows and the columns of the tabular data that divides the data into multiple cells. In one or more embodiments of the present invention, the content recognition device120can also extract text from the input image112and include it in the electronic output130. The extracted text can be presented such that the meaning can be determined based on the relationships between the cells in the tabular data.

The ML model122is trained to parse the content of the input image112using training data140. The training data140includes extracted images142of tabular data paired with the corresponding markup data144that provides structure and content of the tabular data in the extracted images142. The training data140includes multiple (e.g. thousands, millions etc.) such corresponding pairs of extracted images142of tabular data and corresponding markup data144of tabular structure and content. The training data140is used by the ML model to lean to identify table structure and content in a given image of tabular data, such as the input image112, without the corresponding description of the structure of the tabular data.

FIG. 2depicts an example input image112in an example scenario, andFIG. 3depicts an electronic output130in an example scenario. In the depicted example shown inFIG. 2, the input image112includes columns410and rows420. As shown, one or more rows can include sub-rows430. The rows420and the columns410form multiple cells440. The structure of the tabular data is delineated into markup format132in the electronic output130as shown inFIG. 3. The delineation in the markup format132describes the column delineation510corresponding to the columns410. The column delineation510indicates a number of columns410. The column delineation510also describes the headings of the columns410. The column delineation510can include other characteristics to describe the columns410in other examples where the columns410have other attributes, such as sub-columns, for example.

The markup format132shown inFIG. 3also includes row delineation520corresponding to the rows420. The row delineation520describes a number of rows420in the input image112. The row delineation520also includes the headings of the rows420. The row delineation520also includes the sub-delineation530for a row420that includes one or more sub-rows430.

Further, the electronic output130shown inFIG. 3includes the extracted text540from the cells440shown inFIG. 2. The extracted text540is included according to the structure that is delineated from the input image112shown inFIG. 2.

FIG. 4depicts a flowchart of a method600for training the ML model122of the content recognition device120to extract tabular data from the input image112according to one or more embodiments of the present invention. The method600includes training the ML model122using the training data140, at block610. The training data includes training images142with tabular data and corresponding training markup data144that describes the structure and content of the tabular data in the training images142. In one or more examples, the training markup data144can be multiple XML files that provide labels of the tabular data in corresponding image files or document files that include the training images142.

The training markup data144provides the ML model122with normalized structured representation of the tabular data in the training images142. In one or more embodiments of the present invention, the training images142are generated by concerting tabular data in multiple documents (e.g. PDF files) into image files (e.g., JPEG). Such conversion can be performed automatically in one or more examples, so that the training data140itself is generated without manual efforts. The training data140provides multiple (thousands, millions) of samples that provide the training images142accompanied by ground truth of the tabular data in markup representation144.

The ML model122, in one or more embodiments of the present invention, can have a deep learning network architecture, which can be trained with the training data140.FIG. 5depicts an example structure of the ML model122and a dataflow of training the ML model122according to one or more embodiments of the present invention.

In one or more embodiments of the present invention the ML model122is a deep neural network learning architecture that uses an encoder-decoder model with multiple layers of convolutional neural network (CNN), attention modules, and recurrent neural networks (RNN). The ML model122in one or more embodiments of the present invention includes an encoder710, and two decoding sets, a first decoding set720and a second decoding set730.

The encoder710includes one or more CNNs that analyze the training images142and use a set of convolutional filters to capture the visual features of the training images142. The parameters of the convolutional filters are updated during training via back propagation. The visual features collectively are called a feature map. Each pixel in the feature map is a high-dimensional feature vector which describes the pattern of a corresponding local patch (e.g., 16×16 pixels) in the training images142. The encoder710forwards the feature map to the first decoding set720and the second decoding set730.

The decoding sets720,730can include one or more units of RNNs and attention modules. In one or more examples, the RNNs can be implemented as long short-term memory (LSTM) or gated recurrent unit (GRU) cells. The first decoding set720can be used to determine the structure of the tabular data in the training images142(and input image112). Accordingly, the first decoding set720can also be referred to as a set of structure decoding neural networks. The second decoding set730can be used to determine the contents of one or more cells in the tabular data of the training images142(and input image112). Accordingly, the second decoding set730can also be referred to as a set of content decoding neural networks.

The set of structure decoding neural networks (720) includes a structure attention module722and a structure decoder724, which is an RNN. The structure attention module722is a neural network that learns how to assign different degrees of focus on different portions of the feature map encoded from training images142(and input image112) to decipher the structure. Higher-degree focus in a certain region of the feature map makes the structure decoder724exploit more information from that region. The degree of focus at a given location in the feature map is determined by multiplying a set of weight parameters to the feature map itself, as well as a “hidden state” of the structure decoder724. In an RNN, a “hidden state,” in the context of a recurrent layer, is a value that is shared during the recurrence to provide a representation of previous inputs. The weight parameters of the attention module722are adjusted during the training via back propagation as will be described further. The attention module722analyzes the feature map of the training images142to determine the degree of focus. The structure decoder RNN724also includes its own weight parameters that are updated via back propagation. The structure decoder RNN724and the structure attention module722are repeatedly trained in conjunction with each other to adjust the weight parameters of both networks, the structure decoder RNN724and the structure attention module722.

The weights are updated to minimize the difference between the markup representation estimate730that is generated by the set of structure decoding neural networks (720) and the known tabular structure144for the training images142. As depicted inFIG. 5, the markup representation estimate is compared with the tabular structure144to compute a loss function740. The loss function740indicates a difference in the structure that is extracted and the known structure in the training data144. The loss function740is provided as feedback to the structure decoder RNN724, the structure attention module722, and encoder710. Alternatively, a structure accuracy score is computed based on the difference and provided as a feedback. The weights of the structure decoder RNN724, the structure attention module722, and encoder710are updated based on the structure accuracy score to make the markup representation estimate730closer to the tabular data144. In one or more examples, the structure accuracy score is compared with a predetermined accuracy threshold. If the structure accuracy score exceeds the predetermined structure accuracy threshold, the set of structure decoding neural networks720are marked as being trained. Alternatively, if the structure accuracy score is different from the predetermined structure accuracy threshold by more than a certain threshold, the set of structure decoding neural networks720are continued to be trained, i.e., weights adjusted.

The set of content decoding neural networks730is trained and operates in similar manner to the set of structure decoding neural networks720. The set of content decoding neural networks730includes a content attention module732and a content decoder734, which is an RNN. The content attention module732is a neural network that learns how to assign different degrees of focus on different portions of the feature map encoded from training images142(and input image112) to decipher the content. Higher-degree focus in a certain region of the feature map makes the content decoder734exploit more information from that region. The degree of focus at a given location in the feature map is determined by multiplying a set of weight parameters to the feature map itself, as well as the hidden state of the structure decoder724and the hidden state of the content decoder734. The weight parameters of the content attention module732are adjusted during the training via back propagation as will be described further. For example, the content attention module732analyzes the feature map of the training images142that are to be used by the content decoder RNN734. The content decoder RNN734also includes its own weight parameters that are updated via back propagation. The content decoder RNN734and the content attention module732are repeatedly trained in conjunction with each other to adjust the weight parameters of both networks, the content decoder RNN734and the content attention module732.

The weights are updated to minimize the difference between the representation estimate730that is generated be the set of content decoding neural networks and the known content in tabular data for the training images142. As depicted inFIG. 5, the markup representation estimate730is compared with the tabular data144to compute the loss function740. The loss function740indicates a difference in the content that is extracted and the known content in the training data144. The loss function740is provided as feedback to the content decoder RNN734, the content attention module732, and encoder710. Alternatively, a content accuracy score is computed based on the difference and provided as a feedback. The weights of the content decoder RNN734, the content attention module732, and encoder710are updated based on the content accuracy score to make the markup representation estimate730closer to the tabular data144. In one or more examples, the content accuracy score is compared with a predetermined accuracy threshold. If the accuracy score exceeds the predetermined accuracy threshold, the set of content decoding neural networks730are marked as being trained. Alternatively, if the accuracy score is different from the predetermined accuracy threshold by more than a certain threshold, the set of content decoding neural networks730are continued to be trained, i.e., weights adjusted.

Accordingly, for delineating tabular data presented in an image format, in one or more embodiments of the present invention, the ML model120includes three neural networks—the encoder710, the structure attention module722, and the structure decoder724. The encoder710determines feature maps from the training images, which are forwarded to the structure attention module722. The structure attention module722determines degree of focus from the feature maps. The structure decoder uses the degree of focus and the feature maps to determine delineation estimate of the structure of the tabular data. Based on a difference between the estimate and the markup representation from the training data144, the weights of the encoder710, the structure attention module722, and the structure decoder724are updated/adjusted.

In addition, the ML model120includes, for extracting content from the tabular data, the content attention module732and the content decoder734, both of which work in conjunction with the encoder710. Here, the content attention module732uses the feature maps that are determined by the encoder710to determine degrees of focus for content extraction. The content decoder734subsequently determines content estimate using the feature maps and the degrees of focus. The difference between the content estimate and the actual content from the training data is used to adjust the weights of the encoder710, the content attention module730, and the content decoder734. In one or more examples, the content attention module732uses inputs from the structure decoder724. In one or more examples, all the neural networks in the ML model120are trained in conjunction.

Referring back to the flowchart of the method600inFIG. 4, in one or more embodiments of the present invention, the method600further includes receiving the input image112that includes tabular data with an unknown structure, at block620. In other words, the structure of the tabular data in the input image is unknown to the content recognition device120.

The content recognition device120determines the delineation of the structure of the tabular data in the input image112using the trained ML model122, at block630. Particularly, the structure decoding neural network(s) facilitate identifying the delineation of the tabular data based on the weights that are setup during the training. The delineation includes a markup representation of the cells440in the input image112. The delineation also includes a markup representation of the columns410and the rows420from the tabular data. The delineation further identifies the demarcation between sub-rows and sub-columns that may be present in the tabular data.

Further, the method600includes extracting content from the tabular data in input image112, at block640using the set of content decoding neural networks730.

The method600further includes merging the extracted content and the delineation of the tabular data from the input image112, at block650. The merging includes adding the extracted content in the markup representation of the delineated structure of the tabular data. Accordingly, in one or more embodiments of the present invention, a markup file is generated (seeFIG. 3) that includes tags according to the markup language being used to demarcate the tabular structure. The markup file further includes the extracted content embedded within the tags.

In this manner, the input image112is analyzed automatically using the ML model122to identify the structure of the tabular data from the input image and further to extract the content of the tabular data. Further, the tabular data is represented using a markup format. It should be noted that although a markup format is used throughout the examples described herein, in one or more embodiments of the present invention, the tabular data can be represented using any other electronic formatting or protocols, such as a comma separated volume (CSV), or any other machine readable format.

As shown inFIG. 6, the computer system800has one or more central processing units (CPU(s))801a,801b,801c, etc. (collectively or generically referred to as processor(s)801). The processors801can be a single-core processor, multi-core processor, computing cluster, or any number of other configurations. The processors801, also referred to as processing circuits, are coupled via a system bus802to a system memory803and various other components. The system memory803can include a read only memory (ROM)804and a random access memory (RAM)805. The ROM804is coupled to the system bus802and may include a basic input/output system (BIOS), which controls certain basic functions of the computer system800. The RAM is read-write memory coupled to the system bus802for use by the processors801. The system memory803provides temporary memory space for operations of said instructions during operation. The system memory803can include random access memory (RAM), read only memory, flash memory, or any other suitable memory systems.

The computer system800comprises an input/output (I/O) adapter806and a communications adapter807coupled to the system bus802. The I/O adapter806may be a small computer system interface (SCSI) adapter that communicates with a hard disk808and/or any other similar component. The I/O adapter806and the hard disk808are collectively referred to herein as a mass storage810.

Software811for execution on the computer system800may be stored in the mass storage810. The mass storage810is an example of a tangible storage medium readable by the processors801, where the software811is stored as instructions for execution by the processors801to cause the computer system800to operate, such as is described herein below with respect to the various Figures. Examples of computer program product and the execution of such instruction is discussed herein in more detail. The communications adapter807interconnects the system bus802with a network812, which may be an outside network, enabling the computer system800to communicate with other such systems. In one embodiment, a portion of the system memory803and the mass storage810collectively store an operating system, which may be any appropriate operating system, such as the z/OS or AIX operating system from IBM Corporation, to coordinate the functions of the various components shown inFIG. 6.

Additional input/output devices are shown as connected to the system bus802via a display adapter815and an interface adapter816and, in one embodiment, the adapters806,807,815, and816may be connected to one or more I/O buses that are connected to the system bus802via an intermediate bus bridge (not shown). A display819(e.g., a screen or a display monitor) is connected to the system bus802by a display adapter815, which may include a graphics controller to improve the performance of graphics intensive applications and a video controller. A keyboard821, a mouse822, a speaker823, etc. can be interconnected to the system bus802via the interface adapter816, which may include, for example, a Super I/O chip integrating multiple device adapters into a single integrated circuit. Suitable I/O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols, such as the Peripheral Component Interconnect (PCI). Thus, as configured inFIG. 6, the computer system800includes processing capability in the form of the processors801, and, storage capability including the system memory803and the mass storage810, input means such as the keyboard821and the mouse822, and output capability including the speaker823and the display819.

In some embodiments, the communications adapter807can transmit data using any suitable interface or protocol, such as the internet small computer system interface, among others. The network812may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. An external computing device may connect to the computer system800through the network812. In some examples, an external computing device may be an external webserver or a cloud computing node.

It is to be understood that the block diagram ofFIG. 6is not intended to indicate that the computer system800is to include all of the components shown inFIG. 6. Rather, the computer system800can include any appropriate fewer or additional components not illustrated inFIG. 6(e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Further, the embodiments described herein with respect to computer system800may be implemented with any appropriate logic, wherein the logic, as referred to herein, can include any suitable hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware, in various embodiments.

Embodiments of the present invention can be implemented using cloud computing technology in one or more examples. It is to be understood that although this disclosure includes a detailed description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Accordingly, one or more embodiments of the present invention facilitate a machine to autonomously understand unstructured tables from various literature that is available in electronic format. Embodiments of the present invention provide a technical solution to a technical challenge in the field of computing technology and improves the computing technology by facilitating the machine to perform such functions autonomously.

Further, embodiments of the present invention provide a practical application by facilitating systems to provide automatic such as corpus conversion and compare and comply. For example, corpus conversion service can be a system supported by machine learning that is trained to parse the tabular data. The system uses optical character recognition (OCR) to locate text in document images and uses handcrafted features to parse tables. To train the system, users need to manually annotate the location of columns, rows, and cells. As described herein, embodiments of the present invention operates without requiring explicit annotations of the elements of tables. Rather, embodiments of the present invention use structured representation as supervision. Accordingly, embodiments of the present invention provide an improved training and an improved system that is suited for end-to-end training and multi-task training. Additionally, existing compare and comply system uses a set of manually defined rules to define the table layout and extract content using OCR technology. Embodiments of the present invention facilitate such extraction by using a processing end-to-end that avoids some of the errors accumulated by the different processing steps in the compare and comply system.

Further, embodiments of the present invention improve the content recognition systems by including tabular data processing and improving performance of tabular data processing in some cases. As noted earlier, a substantial quantity of information is available in unstructured tabular format inside of multiple electronic documents. Such tabular data includes information such as health insurance coverage, information published in the scientific literature, and the like.

Embodiments of the present invention can be fed with an image representation of a table, which is user friendly to obtain. Embodiments of the present invention automatically learn how to recognize the table layout (structure and content) in an end-to-end manner and decide when text is to be extracted. Further, embodiments of the present invention parse and delineate the table structure entirely, rather than just extracting content of each cell from the table as a single unit. Additionally, embodiments of the present invention can parse and delineate multi-column/multi-row tables. Further yet, embodiments of the present invention parse the tabular data without explicit annotations of one or more elements (e.g., locations of columns, rows, and cells) in the structure of the table.

Accordingly, embodiments of the present invention only requires the extracted images of tables to intelligently identify their structures and contents into the reusable markup (HTML/XML) formats. Embodiments of the present invention, thus, provide an advantage over existing OCR based systems, in that embodiments of the present invention facilitates identifying the structure of table images and generates programmatic tables of elements, which is not achieved by the OCR based systems.