Machine learning on mixed data documents

A first natural language document is received. The document includes unstructured data and a first table structure that includes a plurality of first table entries. The first table structure is identified based on the document. The first table structure is extracted from the document in response to the identifying. A first machine learning output is generated based on a first machine learning model and from the document. A second machine learning output is generated based on a second machine learning model and from the first table structure. The first output of the document and the second output of the first table structure are combined.

BACKGROUND

The present disclosure relates to machine learning, and more specifically, to natural language processing on mixed data documents.

Machine learning involves teaching a computer algorithm, i.e., the machine, to label or classify a data record in some way, e.g., labelling the subject of a photograph as animate or inanimate. Machine learning involves training the computer algorithm to make the classification by providing labeled examples, e.g., picture subjects (labeled as animate or not). In machine learning training, a classification task is performed repeatedly in order to gradually improve the accuracy of the classification. Machine learning may be used to process documents that are created by humans. The created documents may include unstructured data, such as text, sentences, and paragraphs. The created documents may include other information structured in a tabular form.

SUMMARY

According to embodiments, disclosed are a method, system, and computer program product. A first natural language document is received. The document includes unstructured data and a first table structure that includes a plurality of first table entries. The first table structure is identified based on the document. The first table structure is extracted from the document in response to the identifying. A first machine learning output is generated based on a first machine learning model and from the document. A second machine learning output is generated based on a second machine learning model and from the first table structure. The first output of the document and the second output of the first table structure are combined.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to machine learning; more particular aspects relate to natural language processing on mixed data documents. While the present disclosure is not necessarily limited to such applications, various aspects of the disclosure may be appreciated through a discussion of various examples using this context.

Machine learning may be one or more techniques to process data. One use of machine learning is that of natural language processing. For example, IBM Watson™ may operate to determine meaning based on documents that contain natural language. Machine learning may be embodied in a machine learning model. A machine learning model may be a neural network, a natural langue processing system, a support vector machine, or other relevant computer structure configured for processing data. The machine learning model may be configured to input data in the form of text and documents, and further configured to output syntactical meaning, entity identification, relationship detection, and other features of the input.

Some documents may include natural language in the form of unstructured text or data. Unstructured text may include information that either does not have a pre-defined data model or is not organized in a pre-defined manner. The unstructured text may include words, sentences, paragraphs, and other relevant features. The natural language documents may also be processed by identifying or annotating various features of the document. For example, a document may be annotated to distinguish or otherwise identify document spans, nouns, paragraphs, transitive verbs, and the like. The machine learning model may then process the various features of the annotated document.

Some documents may include mixed data documents that include natural language data in the form of unstructured text and also in the form of table structures. Table structures may include text that is grouped, segmented, or arranged in a specific and consistent manner (e.g., rows, columns, cells). Table structures may include any of the following: tables that include rows and columns; a bulleted list; a numbered list; tabbed text; or any other relevant structured textual data. Structured data may include entities of data that are organized in and standardized in a pre-defined manner. A bulleted list or numbered list may be considered a single column structured text. Each item or entry in table may be organized in a non-natural language manner. For example, in a plurality of entries of a table, each entry may be separated by a specific non-visible character, such as a non-visible dash, comma, new-line, or other relevant character. Table structures may rely on other structures, such as, fields, forms, entries, cells, and the like.

A machine learning model may not be able to parse natural language documents of natural language and also table structures (mixed data documents). In detail, a machine learning model may be configured or trained on the typically text-heavy nature of natural language. While natural language documents do contain data such as dates, numbers, and facts, they may be surrounded by large spans of other qualifying and semi-unrelated textual data. This results in irregularities and ambiguities that make it difficult to understand without the use of the additional configuration and training of the machine learning model. The additional configuration and training of the machine learning model, however, is not designed for processing of table structures. The machine learning model may misinterpret or disassociate the various concepts, features, and data contained within the entries of a table structure that is a part of a mixed data document. For example, in one study, a machine learning model failed to accurately analyze 13% of a mixed data document. Further, the 13% of data that was not accurately analyzed accounted for a majority of the meaning of the document.

Another attempt to process mixed data documents is to augment or supplement the machine learning with one or more structure data processing techniques. The structured data processing techniques may yield some information that is extracted from a table structure. Unfortunately, the techniques of the structured data processing are often not relevant. Further, structured data techniques often output data that is in a format that loses the meaning or features that are identified by the machine learning model. Further, machine learning models may rely on resource intensive annotations of data. For example, a computer annotation process or a human may be employed to annotate natural language. The computer annotation process and the tools that assist humans to perform human annotation may be configured to perform annotation only on natural language spans. Not only can the human/computer annotation be more resource and time intensive, the annotation programs and tools may also be inaccurate and lead to further inaccuracies in performing machine learning on mixed data documents.

Multiple-model mixed-data machine-learning (MMML) may operate to overcome the weaknesses and issues related to processing mixed data documents. The MMML may operate based on two or more machine learning models to process documents. In detail, the MMML may utilize a first machine learning model (ML model) to perform the analysis and natural language processing of natural language in a given document. The document may be annotated by the MMML before being processed by the first ML model. In some embodiments, the document may be annotated by a human before the MMML performs the processing. The MMML may generate features from the natural language portion of the mixed data document. The features may include entities and relationships. The MMML may perform natural language processing on table structures of the mixed data document. In detail, the MMML may utilize a second ML model to perform the analysis and natural langue processing of the table structure of a given document. The MMML may utilize the entities and relationships of the first ML model of the MMML to annotate the table structure before the second ML model of the MMML processes the table structure.

In some embodiments, a natural language processing system of the first ML model and/or the second ML model of the MMML may include various components (not depicted) operating through hardware, software, or in some combination. For example, a natural language processor, one or more data sources, a search application, and a report analyzer. The natural language processor may be a computer module that analyses the received content and other information. The natural language processor may perform various methods and techniques for analyzing textual information (e.g., syntactic analysis, semantic analysis, etc.). The natural language processor may be configured to recognize and analyze any number of natural languages. In some embodiments, the natural language processor may parse passages of documents or content from mixed data documents. Various components (not depicted) of the natural language processor may include, but are not limited to, a tokenizer, a part-of-speech (POS) tagger, a semantic relationship identifier, and a syntactic relationship identifier. The natural language processor may include a support vector machine (SVM) generator to process the content of topics found within a corpus and classify the topics.

In some embodiments, the tokenizer may be a computer module that performs lexical analyses. The tokenizer may convert a sequence of characters into a sequence of tokens. A token may be a string of characters included in an electronic document and categorized as a meaningful symbol. Further, in some embodiments, the tokenizer may identify word boundaries in an electronic document and break any text passages within the document into their component text elements, such as words, multiword tokens, numbers, and punctuation marks. In some embodiments, the tokenizer may receive a string of characters, identify the lexemes in the string, and categorize them into tokens.

Consistent with various embodiments, the POS tagger may be a computer module that marks up a word in passages to correspond to a particular part of speech. The POS tagger may read a passage or other text in natural language and assign a part of speech to each word or other token. The POS tagger may determine the part of speech to which a word (or other text element) corresponds based on the definition of the word and the context of the word. The context of a word may be based on its relationship with adjacent and related words in a phrase, sentence, or paragraph.

In some embodiments, the context of a word may be dependent on one or more previously analyzed electronic documents (e.g., mixed data documents). Examples of parts of speech that may be assigned to words include, but are not limited to, nouns, verbs, adjectives, adverbs, and the like. Examples of other part of speech categories that POS tagger may assign include, but are not limited to, comparative or superlative adverbs, wh-adverbs, conjunctions, determiners, negative particles, possessive markers, prepositions, wh-pronouns, and the like. In some embodiments, the POS tagger may tag or otherwise annotate tokens of a passage with part of speech categories. In some embodiments, the POS tagger may tag tokens or words of a passage to be parsed by the natural language processing system.

In some embodiments, the semantic relationship identifier may be a computer module that may be configured to identify semantic relationships of recognized text elements (e.g., words, phrases) in documents. In some embodiments, the semantic relationship identifier may determine functional dependencies between entities and other semantic relationships.

Consistent with various embodiments, the syntactic relationship identifier may be a computer module that may be configured to identify syntactic relationships in a passage composed of tokens. The syntactic relationship identifier may determine the grammatical structure of sentences such as, for example, which groups of words are associated as phrases and which word is the subject or object of a verb. The syntactic relationship identifier may conform to formal grammar.

In some embodiments, the natural language processor may be a computer module that may parse a document and generate corresponding data structures for one or more portions of the document. For example, in response to receiving a mixed data document at the natural language processing system, the natural language processor may output parsed text elements from the data. In some embodiments, a parsed text element may be represented in the form of a parse tree or other graph structure. To generate the parsed text element, the natural language processor may trigger computer modules including the tokenizer, the part-of-speech (POS) tagger, the SVM generator, the semantic relationship identifier, and the syntactic relationship identifier.

In some embodiments, the natural language processing system may leverage one or more of the example machine learning techniques to perform machine-learning (ML) text operations. Specifically, an MMML may operate to perform machine-learning text classification and/or a machine-learning text comparison. Machine-learning text classification may include ML text operations to convert, characters, text, words, and phrases to numerical values. The numerical values may then be input into a neural network to determine various features, characteristics, and other information of words with respect to a document or in relation to other words (e.g., to classify a numerical value associated with a word may permit the classification of a word). Machine-learning text comparison may include using the numerical values of converted characters, text, words, and phrases for performing a comparison. The comparison may be a comparison of a numerical value of a first word or other text to the numerical value of a second word or other text. The determination of the machine-learning text comparison may be to determine a scoring, a correlation, or a relevant relationship (e.g., a relationship between a first numerical value of a first word and a second numerical value of a second word). The comparison may be used to determine if two words are similar or different based on one or more criteria. The numerical operations of a machine-learning text classification/comparison may be a function of a mathematical operation performed through a neural network, such as performing a linear regression, an addition, or other relevant mathematical operation of the numerical value representative of a word or other text.

The ML text operations may include word encoding, such as one-hot encoding of words from a tokenizer, POS tagger, semantic relationship identifier, syntactic relationship identifier, and the like. The ML text operations may include the use of vectorization of text, such as the vectorization of words from a tokenizer, POS tagger, semantic relationship identifier, syntactic relationship identifier, and the like. For example, a paragraph of text may include the phrase “orange is a fruit that grows on trees.” Vectorization of the word “orange” may include setting input neurons of a neural network to the various words of the phrase including the word “orange.” The output value may be an array of values (e.g, forty-eight numbers, thousands of numbers). The output values may trend towards “1” for related words and may trend towards “0” for unrelated words. The related words may be related based on one or more of the following: similar parts of speech, syntactical meaning, locality within a sentence or paragraph, or other relevant “closeness” between the input and other parts of natural language (e.g., other parts of the phrase “orange is a fruit that grows on trees”, other parts of a paragraph that contains the phrase, other parts of language).

FIG. 1depicts the representative major components of an example computer system100(alternatively, computer) that may be used, in accordance with some embodiments of the present disclosure. It is appreciated that individual components may vary in complexity, number, type, and/or configuration. The particular examples disclosed are for example purposes only and are not necessarily the only such variations. The computer system100may comprise a processor110, memory120, an input/output interface (herein I/O or I/O interface)130, and a main bus140. The main bus140may provide communication pathways for the other components of the computer system100. In some embodiments, the main bus140may connect to other components such as a specialized digital signal processor (not depicted).

The processor110of the computer system100may be comprised of one or more cores112A,112B,112C,112D (collectively112). The processor110may additionally include one or more memory buffers or caches (not depicted) that provide temporary storage of instructions and data for the cores112. The cores112may perform instructions on input provided from the caches or from the memory120and output the result to caches or the memory. The cores112may be comprised of one or more circuits configured to perform one or more methods consistent with embodiments of the present disclosure. In some embodiments, the computer system100may contain multiple processors110. In some embodiments, the computer system100may be a single processor110with a singular core112.

The memory120of the computer system100may include a memory controller122. In some embodiments, the memory120may comprise a random-access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) for storing data and programs. In some embodiments, the memory may be in the form of modules (e.g., dual in-line memory modules). The memory controller122may communicate with the processor110, facilitating storage and retrieval of information in the memory120. The memory controller122may communicate with the I/O interface130, facilitating storage and retrieval of input or output in the memory120.

The I/O interface130may comprise an I/O bus150, a terminal interface152, a storage interface154, an I/O device interface156, and a network interface158. The I/O interface130may connect the main bus140to the I/O bus150. The I/O interface130may direct instructions and data from the processor110and memory120to the various interfaces of the I/O bus150. The I/O interface130may also direct instructions and data from the various interfaces of the I/O bus150to the processor110and memory120. The various interfaces may include the terminal interface152, the storage interface154, the I/O device interface156, and the network interface158. In some embodiments, the various interfaces may include a subset of the aforementioned interfaces (e.g., an embedded computer system in an industrial application may not include the terminal interface152and the storage interface154).

Logic modules throughout the computer system100—including but not limited to the memory120, the processor110, and the I/O interface130—may communicate failures and changes to one or more components to a hypervisor or operating system (not depicted). The hypervisor or the operating system may allocate the various resources available in the computer system100and track the location of data in memory120and of processes assigned to various cores112. In embodiments that combine or rearrange elements, aspects and capabilities of the logic modules may be combined or redistributed. These variations would be apparent to one skilled in the art.

It is to be understood that although this disclosure includes a detailed description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or later developed. Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

Characteristics are as follows:

Rapid elasticity: capabilities can be rapidly and elastically provisioned, in some cases

automatically, to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.

Service Models are as follows:

Deployment Models are as follows:

Hybrid cloud: the cloud infrastructure is a composition of two

or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load-balancing between clouds).

Hardware and software layer60includes hardware and software components. Examples of hardware components include: mainframes61; RISC (Reduced Instruction Set Computer) architecture based servers62; servers63; blade servers64; storage devices65; and networks and networking components66. In some embodiments, software components include network application server software67and database software68. Virtualization layer70provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers71; virtual storage72; virtual networks73, including virtual private networks; virtual applications and operating systems74; and virtual clients75.

FIG. 4depicts an example system400of performing natural language processing of mixed data documents, consistent with some embodiments of the disclosure. System400may be configured to operate on input410and generate an output420. System400may include a MMML430for operating on the input410. For example, MMML430may be configured to generate an output420, based on input410. MMML430may be a computer, such as computer system100. In some embodiments, MMML430may be a part of a cloud-based system, such as cloud computing environment50.

MMML430of system400may include a plurality of hardware and software components. For example, MMML430includes the following: a table analyzer440configured to extract table structures; a natural language model (NL model)450configured to output analysis of natural language; a table structure model (table model)460configured to output analysis of table structures; and an output consolidator470configured to generate a single output from the multiple outputs of the rest of the MMML430. In some embodiments, one or more components of MMML430may operate based on a machine learning processor and/or based on a natural language processor480. For example, ML/NLP processor480may be configured to ingest training datasets, perform annotation of training documents, and generate NL model450and table model460.

Table analyzer440may be software, such as one or more processes and programs configured to operate on input, such as natural language documents. Table analyzer440may be hardware, such as one or more general purpose processors, or special purpose processing units configured to operate on input. In some embodiments, table analyzer440may be a combination of software and hardware, such as low-level functional programing uploaded to a field programmable gate area or re-writeable executable firmware.

Table analyzer440may operate on input410. For example, input410may be a mixed data document that includes unstructured language412and table structure414. Unstructured language412may include natural language, such as words, sentences, paragraphs, and the like. Table structure414may include structures of text, such as cells, fields, entries, bulleted list items, numbered list items, tab separated entries, and the like. Though input410may depict two portions of unstructured language and a singular table structure, other types of natural language documents may be processed as well. For example, MMML430may also be configured to operate on solely natural language documents as input410. In another example, MMML430may be configured to operate on a security whitepaper that includes a summary portion (a single unstructured language portion) and a plurality of tables and bulleted lists that are related to the summary (a plurality of table structures).

Table analyzer440may be configured to separate any unstructured natural language and table structure. In detail, table analyzer440may perform natural language processing, object character recognition, feature detection, annotated document analysis, or another relevant technique to identify a table structure in any received input before further processing of MMML430occurs. For example, table analyzer440may identify in input410(e.g., a mixed data document) table structure414and unstructured language412. Based on the identification of table structure414, table analyzer440may be configured to extract table structure414from unstructured language412. In some embodiments table analyzer may be configured to partition, copy, move, or otherwise separate unstructured language412and table structure414(e.g., save them into two separate documents). Table analyzer440may then be configured to pass the separate unstructured language412and the table structure414to other components of MMML430.

NL model450may be configured to receive unstructured language412. In detail, NL model450may be a machine learning model, a neural network, a support vector machine, or other relevant machine learning processing system executed by a processor, such as processor120. NL model450may be configured to generate output (e.g., entities, relationships of various entities) based on the unstructured language412. For example, NL model450may receive a first document containing the unstructured language412. NL model450may generate an output that is provided to output consolidator470. NL model450may also provide the output to table model460.

Table model460may be configured to receive one or more table structures. In detail, NL model450may be a second machine learning model, neural network, support vector machine, or other relevant machine learning processing system, that is executed by processor120. Table model460may be configured to operate on table structures, such as table structure414. In some embodiments, table model460may be configured to operate on table structures and may be based on output from NL model450. In detail, table model460may perform annotation, matching, identification, or other analysis of table structure414to generate output. Table model460may use the output of NL model450to generate output from analysis of table structure414. In a first example, table model460may analyze table structure414to identify a table based on a relationship identified in output of NL model450. In a second example, table model460may analyze table structure414to identify a column or row based on an entity of output of NL model450.

Output consolidator470may incorporate software, and utilize one or more processes and programs configured to generate output. Output consolidator470may be hardware, such as one or more general purpose processors, or special purpose processing units configured to generate output. In some embodiments, output consolidator470may be a combination of software and hardware, such as an application specific integrated circuit. Output consolidator470may be configured to combine the first machine learning output of the NL model450with the second machine learning output of the table model460. Output consolidator470may employ analysis to aggregate, include, interleave, associate, or otherwise combine the outputs of the NL model450and the table model460. Output consolidator470may utilize natural language processing, bubble sorting, word association, text identification, and/or another other single (or combination of) relevant computer science technique(s).

Output420may be a result of the output consolidator470of the MMML430. For example, NL model450may output from unstructured language412a first entity422-1, a second entity422-2, and a relationship424-1between the first entity and the second entity. Table model460may output from structured language414a third entity422-3and a relationship424-2. In some embodiments, table model460may create a relationship between entity422-2and entity422-3(i.e., relationship424-2). In some embodiments, output consolidator470may append, combine, bring together, merge, concatenate, or otherwise associate entity422-2and422-3(i.e., relationship424-2).

FIG. 5depicts an example method500of processing mixed data documents, consistent with some embodiments of the disclosure. Method500may be performed by a computer, such as computer system100. Method500may be performed by a portion of a cloud service, such as cloud environment50. Method500may be performed by multiple-model mixed-data machine learning, such as MMML430. Method500may be performed responsively, such as in response to detecting, receiving, retrieving, or otherwise obtaining, documents that contain natural language.

From start505, one or more documents may be received at510. The documents may be natural language documents. The documents may be mixed data documents that contain natural language portions and one or more table structures. In some embodiments, the documents may not be annotated. In some embodiments, only a natural language portion of a document may be annotated. For example, a document may be a mixed data document that contains one or more portions of natural language and one or more table structures. The portions of natural language may be annotated, such as parts of speech annotation. The table structures may not be annotated. For example, a document may be a mixed data document and no portion of a table structure may have any annotation, modification, or other analysis performed before being received at510. If a table is not identified at520:N, an output may be processed at570. The output may be processed by a single machine learning model of a natural language processing system, such as NL model450.

If a table is identified, at520:Y, one or more table structures may be extracted at530. The table structures may be extracted by a MMML, such as MMML430. The table structures may be extracted by removing, culling, or otherwise separating the tables structures from the rest of the mixed data document (e.g., inserting the table structures into a second document). After being extracted at530, the first output of a machine learning model may be performed at540on the mixed data document that no longer contains any table structures. For example, a mixed data document that contains only natural language portions, but has any and all table structures removed may be performed by NL model450.

The first machine learning output, generated at540, may include entities and relationships that convey the meaning or other features of the natural language portions of the mixed data document. At550a second machine learning output may be generated. The second output may be generated by a second machine learning model, such as table model460. The second machine learning model may utilize output of the first machine learning model. For example, any entities, relationships, or other features and outputs of a first machine learning model may be fed to a second machine learning model. In some embodiments, the second machine learning model may annotate the table structures based upon the output of the first machine learning model. For example, a second machine learning model may first match, identify, and annotate one or more table structures based upon the output of the first machine learning model.

At560, the output of the machine learning models may be combined. The machine learning model outputs may be combined by an MMML430. For example, output consolidator470may use the output of a first machine learning model, generated at540, as a frame, skeleton, outline, or first output document. The output consolidator470may take the output and append, combine, bring together, or match the output from the second machine learning output, generated at550, to enhance, expand, or otherwise add to the first machine learning output. After the output is combined at560—alternatively, after the natural language document is processed at570—method500ends at595.

FIG. 6a flow diagram600of performing training of a multiple machine learning models for mixed document analysis, consistent with some embodiments of the disclosure. Flow diagram600may depict the operation and movement of data through a system, or consistent with a method or computer program product having one or more program instructions consistent with embodiments of the disclosure. The operations of flow diagram600may be the performance of training a machine learning model based on a training set of data, such as a plurality of mixed data documents. For example, computer system100may perform training of models based upon documents that contain natural language and table structures. In another example, cloud environment50may operate by scanning for and detecting the presence of table structures in a subset of a plurality of natural language documents. The operations of flow diagram600may be performed by MMML430or another relevant configuration of a multiple-model mixed-data machine learning system consistent with embodiments.

The training operations depicted by diagram600may be based on a topic. For example, training may be based on the plurality of documents610that are related to or about the same topic of security anomalies. The plurality of documents610may be directed to computer viruses and other malware. Any models generated as a result of the training may then more accurately be able to analyze mixed data documents that are related to the topic of security and security anomalies. In another example, training may be based on a plurality of documents being about a topic of medical device operation. The plurality of documents610may be directed to various medical conditions and treatment and patient outcomes that are related to a medical device or various medical devices. Any models generated as a result of training may then more accurately be able to analyze mixed data documents that are related to the topic of medicine and medical devices.

A plurality of documents610may be used for training data. The plurality of documents may include one or more natural language documents that contain text, sentences, paragraphs, and other natural language spans. Some of the documents of the plurality (i.e., a subset) may also include table structures. These documents may be mixed data documents that include natural language and table structures. The table structures may include bulleted lists, numbered lists, tab or character separated entries, row and column-organized data, and other structured data.

A table extraction unit620may operate to remove, extract, or otherwise separate the natural language and the table structures. The table extraction unit620may be one or more circuits or computer programs configured to process natural language and mixed data documents. The table extraction unit620may operate by scanning the plurality of documents610to identify the subset that includes table structures. The table extraction unit620may operate by separating the documents into two groups. The first group may be documents from the plurality610that only include natural language portions and do not include any table structures. The second group may be documents from the plurality610that are a part of the subset that included table structures before being extracted by table extraction unit620. The table extraction unit620may transfer the plurality of documents other than the subset (e.g., the documents that do not include table structures) to an annotation unit630.

The table extraction unit620may separate the table structures of the second group from the natural language portions of the second group. The separation may include generation of new table-structure documents. The separation may include an insertion of a document identifier (e.g., a series of numbers, a series of alphanumeric characters) into the mixed data document and the newly created table structure document. For example, if a document from the subset includes natural language portions and one or more table structures, the table extraction unit620may create a new file for the table structures. The table extraction unit620may then place an identifier, such as “43521349087-A”, into the original document that now only contains the natural language portions and no longer contains table structures. The table extraction unit620may then place a similar identifier, such as “43521349087-B”, into the newly created table structure document. The two identifiers may then be used later to keep track of and associate the two separate documents that were originally a singular mixed data document.

An annotation engine630may be used to annotate that natural language portions of the documents. The annotation engine630may be one or more circuits or computer programs configured to process natural language and annotate parts of speech or other features of a document. The annotation engine630may pass the annotated documents to a natural language training (language training) unit640. The language training unit640may perform one or more relevant training operations, such as neural networking, to generate and refine a natural language model (NL model)650. The language training unit640may also output one or more entities and relationships as part of generating and refining the NL model650.

The table structure documents created by the table extraction unit620may be sent to a table structure training (table training) unit660. The table training unit660may be one or more circuits or computer programs configured to process language of a table structure. The table training unit660may also perform auto labeling of the table structures. Specifically, the table training unit660may receive unannotated table structure documents. The table training unit660may also receive one or more entities and relationships from the NL model650or the language training unit640. The table training unit660may auto label relationship types of a table structure. The table training unit660may auto label entity types for columns of a table structure. For example, a bulleted list may be treated like a single column table and the list may be annotated by the table training unit with a relationship type from the NL model650. The single column of the bulleted list may be annotated with the entity type from the NL model650. The table training unit660may then generate table model670on the newly annotated table structures. The training may include performing machine learning, such as neural networking, to train table predictors with the label and table features, such as column/row names being features of a relationship. The training may also include performing machine learning, such as neural networking, to train column predictors with the column features, such as column names being features of an entity. The multiple models (e.g., NL model650and table model670) may then be used by multi-model mixed-data machine learning, such as MMML430using the models on new mixed data documents.