Cross-subject model-generated training data for relation extraction modeling

A first vector representation of a first word within a first narrative text and a machine-generated label corresponding to the first word are constructed. Using the first vector representation, an annotator model is trained. The annotator model is configured to produce a set of probabilities, each probability in the set of probabilities representing a probable output annotation corresponding to a word within a narrative text. The training includes minimizing a difference between a first human-generated label corresponding to the first word and a first probable output annotation corresponding to the first word. Using the trained annotator model and a second narrative text, second training data is generated. The trained annotator model is configured to produce an output annotation corresponding to a word within a narrative text. The second training data is usable to train a relation extraction model.

TECHNICAL FIELD

The present invention relates generally to a method, system, and computer program product for generating training data for a natural language processing model. More particularly, the present invention relates to a method, system, and computer program product for cross-subject model-generated training data for relation extraction modeling.

BACKGROUND

Information extraction (IE), a technique used in natural language processing (NLP), refers to the extraction of structured information from a document in narrative text form. Once the structured information is created, it can be used for end-user applications such as structured search, as well as the basis for more complex tasks such as knowledge-base construction, question answering, semantic role labeling, and other NLP tasks. For example, information extraction can be used to extract all the companies named in a collection of financial news articles, all the named medical conditions from patient medical records, or all the instances of a particular part of speech in a corpus of text documents on a variety of subjects. Although examples herein are taken from the English language and refer to grammar features of English, the techniques disclosed herein are applicable to natural languages other than English.

Relation extraction is a type of information extraction. In particular, relation extraction is the identification of relations, or relationships, between entities, as described in narrative text. An entity can be a person, organization, place, thing, name, or other noun or noun phrase. For example, “Paris is the capital of France” expresses the relation capital (France, Paris), where capital( ) is the relation and France and Paris are entities related by the capital( ) relation. Similarly, “Jill is in France” expresses the relation location (Jill, France). As another example, “ABC Corp. and XYZ Inc. have announced plans to merge, with a closing date in the third quarter of this year” expresses the relation merger (ABC Corp., XYZ Inc.). A user can specify a particular relation to be extracted—for example, all of the week's merger announcements from a corpus of corporate press releases. Alternatively, relation extraction can also be used to learn which relations are expressed within a corpus or document and extract one or more of those relations.

Two types of machine learning methods presently perform relation extraction: supervised-learning and rule-based. In a prior-art supervised-learning method, a human expert generates training data by annotating a corpus of narrative text documents. Annotation, as used herein, is the process of labeling data. Thus, in a supervised method, annotation for relation extraction comprises labeling words or phrases as entitles and relations and, optionally, classifying entities and relations into types. Then the human-annotated training data is used to train a machine learning model to produce labels and classifications similar to those in the training data.

Prior-art rule-based methods do not rely on human-annotated training data. Instead, a model, implemented as a software application, parses sentences using a set of rules. For example, one sample rule could be that if a sentence includes “merge” or “merger”, and other words in the sentence match words in a database of publicly traded companies, this sentence likely expresses a merger( ) relation between the named entities.

SUMMARY

The illustrative embodiments provide a method, system, and computer program product. An embodiment includes a method that constructs a first vector representation of a first word within a first narrative text and a machine-generated label corresponding to the first word. An embodiment trains, using the first vector representation, an annotator model, the annotator model configured to produce a set of probabilities, each probability in the set of probabilities representing a probable output annotation corresponding to a word within a narrative text, the training comprising minimizing a difference between a first human-generated label corresponding to the first word and a first probable output annotation corresponding to the first word. An embodiment generating, using the trained annotator model and a second narrative text, second training data, the trained annotator model configured to produce an output annotation corresponding to a word within a narrative text, the second training data being usable to train a relation extraction model.

DETAILED DESCRIPTION

The illustrative embodiments recognize that both—prior-art supervised-learning and rule-based relation extraction methods—suffer from disadvantages. Human experts must generate training data for supervised learning, which can be time consuming, subjective, inaccurate, and expensive. Human experts may generate data inconsistently over time and inconsistently from each other. Human experts may also generate incorrect data, due to incorrect or incomplete understanding of the annotation task. A model trained using consistently incorrect data will produce incorrect results. A model trained using inconsistent data may take longer to train or may not produce consistently correct results. Furthermore, human-generated training data has to be planned and prepared a significant amount of time ahead of the model's runtime. A model trained on predetermined fixed training data may respond adequately to similar data at runtime but may exhibit undesirable performance with data of significant dissimilarity. For humans to prepare the training data, the amount of lead-time required makes it very difficult, if not impossible, to perform rapid training for highly dissimilar previously unseen data.

Rule-based methods do not require annotated training data. However, rule-based methods require a comprehensive set of rules. If the rule set is insufficiently comprehensive, the results of relation extraction using that rule set can also be insufficiently comprehensive. For example, if a rule set omits a rule for a common synonym of a relation (e.g., “join” for “merge”), text expressing that relation will not be included in the structured information for a document or corpus.

Furthermore, as with supervised learning, rule-based training has to be planned and prepared ahead of the model's runtime. A model trained on predetermined fixed rules may respond adequately to anticipated data at runtime but may exhibit undesirable performance with data of unexpected characteristics. Pre-configuration of rules makes it very difficult, if not impossible, to perform rapid model training for data with unexpected characteristics.

The illustrative embodiments recognize that a model, developed using either type of method, is specific to a particular knowledge domain. Training data for a model using a supervised-learning method must be similar to the data the model will process once trained. For example, a model trained using annotated geographical data, such as the capitals of various countries, will not be able to extract merger announcements from a database of corporate press releases. Similarly, a rule-based model that includes rules for parsing geographical data will not include rules that are applicable for processing the financial information in corporate press releases.

Consequently, the illustrative embodiments recognize that there is a need for automated generation of annotations for use as training data. The automated annotation generation should minimize a necessity for human-annotated training data in a knowledge domain, make use of an existing rule-based system in the same knowledge domain, and be usable, once trained, to generate annotations that apply to a different knowledge domain.

The illustrative embodiments recognize that the presently available tools or solutions do not address these needs or provide adequate solutions for these needs. The illustrative embodiments used to describe the invention generally address and solve the above-described problems and other problems related to cross-subject model-generated training data for relation extraction modeling.

An embodiment can be implemented as a software application. The application implementing an embodiment can be configured as a modification of an existing NLP system, as a separate application that operates in conjunction with an existing NLP system, a standalone application, or some combination thereof.

Particularly, some illustrative embodiments provide a method by which annotation data specific to one knowledge domain can be automatically generated using a model trained using a small amount of human-annotated training data and one or more existing rule-based systems from a different knowledge domain.

An embodiment constructs a set of training data. The training data includes a received unit of narrative text. A unit of narrative text can be a portion of a word (e.g. a root portion of a word, a plural noun with a final ‘s’ removed), a word, a phrase, a sentence, a group of sentences, or another unit of narrative text according to the grammar of the language of the narrative text. Unless expressly disambiguated, the term word as used herein refers to the smallest unit of narrative text in a given language that is assigned an annotation. The term word may include more than one natural language word, or only a portion of a natural language word. Similarly, unless expressly disambiguated the term sentence, as used herein, refers to a portion of narrative text that includes one or more words according to the grammar of the language of the narrative text. The term sentence, as used herein, may actually include only a portion of a natural language sentence, or more than one natural language sentence.

The training data also includes a received, human-generated label corresponding to the unit of narrative text. In an embodiment, a human-generated label is one of E, R, and O. E labels a unit of narrative text as an entity, R labels a unit of narrative text as a relation, and O labels a unit of narrative text as other—i.e., not an entity or a relation. However, other labeling schemes, using other combinations of labels representing other types of units of narrative text, are also possible and usable by the illustrative embodiments.

The training data also includes one or more machine-generated labels corresponding to the unit of narrative text. To produce machine-generated labels, an embodiment applies the unit of narrative text to the inputs of one or more prior-art rule-based relation extraction models. If an embodiment uses more than one model, each model recognizes relations based on a set of common patterns and a set of elements or patterns that are different for different models.

In one embodiment that uses more than one model, the machine-generated labels that are generated by a majority of the models are added to the training data. In another embodiment that uses more than one model, the machine-generated labels that are generated by each model are added to the training data. Other schemes to aggregate the results from more than one model are also possible and contemplated within the scope of the illustrative embodiments. Using training data generated by a variety of models, instead of by only one model, results in a trained model that includes the best features of each rule-based model.

An embodiment generates an overall vector representation corresponding to the unit of narrative text. A vector, also called an embedding, is a representation of a unit of narrative text or a label or annotation corresponding to a unit of narrative text. A vector is an array of real numbers, typically between zero and one, but not limited thereto. The array has a large number of dimensions—for example, 300. However, as long as the vector range and number of dimensions are consistent when using a particular model, the exact range and number of dimensions are unimportant. To generate a vector representation, an embodiment uses any suitable prior-art technique—for example, a previously-trained neural network.

One or more components, each also vector representations, make up the overall vector representation. One embodiment uses, as a component, a vector representation of the unit of narrative text. Another embodiment uses, as a component, a vector representation of a part of speech (e.g. a noun, verb, adjective, adverb, or another part of speech) corresponding to the unit of narrative text. Another embodiment uses, as a component, a vector representation of a syntactic role corresponding to the unit of narrative text. A syntactic role is a role played by a unit of narrative text within a sentence. For example, a noun can have a syntactic role of subject, direct object, or indirect object within a sentence. Another embodiment uses, as a component, a vector representation of a machine-generated label (e.g. E, R,0in one labeling scheme) corresponding to the unit of narrative text. Another embodiment uses, as a component, a vector representation of each machine-generated label, concatenated together, corresponding to the unit of narrative text. Another embodiment uses, as a component, a vector representation of a context of the unit of narrative text. An embodiment using more than one component uses, as the overall vector representation, a vector concatenation of each component.

An embodiment uses the training data to train an annotator model. An annotator model is a neural network that takes as input an overall vector representation of a unit of narrative text and produces, as an output, a label, or annotation, corresponding to the input unit of narrative test. In particular, in an embodiment an annotator model configured to produce training data for relation extraction produces an annotation identifying the input unit of narrative test as an entity (E), relation (R), or other (O) (i.e., not an entity or a relation).

In an embodiment, training is done sentence by sentence. A vector representation corresponding to each word within a sentence is an input to a column of a neural network. The input vector includes one or more components, concatenated together. In one embodiment the input vector includes a vector representation of a word (a word embedding), a vector representation of a part of speech of the word (a part of speech, or POS, embedding), a vector representation of a syntactic role of the word, and a vector representation of one or more machine-generated labels for the word. The input vector feeds into a forward long short-term memory (LSTM), an artificial recurrent neural network. The output of the forward LSTM feeds into an adjacent forward LSTM to take context into account when assessing an adjacent word in an input sentence.

The output of the forward LSTM also feeds into a softmax function. A softmax function takes as input a vector of real numbers, some of which might be negative, might be greater than one, or might not sum to one, and converts the vector into a vector of real numbers, each between zero and one, that sums to one. In an embodiment, the output of the softmax function is a set of probabilities, each probability corresponding to a likelihood that the input word is functioning as an entity, relation, or neither within the input sentence.

The input vector also feeds into a backward LSTM. The output of the backward LSTM feeds into an adjacent backward LSTM to take context into account when assessing an adjacent word in an input sentence. The output of the backward LSTM also feeds into the softmax function.

During the training process, an embodiment compares the set of output probabilities to the human-generated label corresponding to the same unit of narrative text. In each iteration of the training process, the embodiment adjusts weights within the annotator model so as to minimize the difference between the model output and the human-generated label, for each unit of narrative text within the training data. An embodiment continues the training for a fixed number of iterations. Another embodiment continues the training until model performance exceeds a predetermined threshold. When training ends, the model is considered to be trained.

Once an embodiment has trained the annotator model, an embodiment uses the trained model to generate, from a second corpus of narrative text, a second set of training data. The second set of training data is suitable for training a second model to perform relation extraction.

In particular, to use the trained model, an embodiment constructs an input vector to the trained model in the same manner as described herein for training the model. The embodiment applies the input vector to the trained model, which produces a set of probabilities, each probability corresponding to a likelihood that the input word is functioning as an entity, relation, or neither within the input sentence. An embodiment uses the highest probability as the label for the input word, providing the highest probability output is also above a predefined threshold.

For example, consider the sentence, “Alpha Corp has released a new improved version of the BankIt application that runs on the Omega system.” A trained annotator model identifies “has released” as a relation within the sentence and labels each word of the phrase with an R, for relation. The trained model determines that the relation operates on two entities, “Alpha Corp” and “a new improved version of the BankIt application”. Accordingly, the trained model labels each word of each entity phrase with an E, for entity. Because the remainder of the words within the sentence do not participate in the relation, the trained model labels each remaining word of the sentence with an O, for other.

For the same example sentence, a trained annotator model also identifies a different relation—“runs on”. Accordingly, the trained model labels each word of the relation phrase with an R. The trained model determines that this relation operates on two entities, “a new improved version of the BankIt application” and “the Omega system”. Accordingly, the trained model labels each word of each entity phrase with an E. Because the remainder of the words within the sentence do not participate in the relation, the trained model labels each remaining word of the sentence with an O, for other.

Once an embodiment has trained an annotator model using a first set of training data from narrative text in one knowledge domain, the trained model is usable to generate a second set of training data using a corpus from a different knowledge domain. For example, an annotator model trained using training data relating to financial announcements could then be used to generate a new set of annotations for narrative text of medical reports. As a result, the annotator model need not be retrained before generating training data for a new knowledge domain.

The manner of cross-subject model-generated training data for relation extraction modeling described herein is unavailable in the presently available methods in the technological field of endeavor pertaining to models for relation extraction. A method of an embodiment described herein, when implemented to execute on a device or data processing system, comprises substantial advancement of the functionality of that device or data processing system in using machine-generated training data to train an annotator model. Once trained using narrative text from one knowledge domain, the annotator model can produce training data, in a different knowledge domain, for a relation extraction model.

The illustrative embodiments are described with respect to certain types of words, partial words, clauses, sentences, units of narrative text, parts of speech, contexts, vectors, embeddings, models, neural networks, thresholds, data processing systems, environments, components, and applications only as examples. Any specific manifestations of these and other similar artifacts are not intended to be limiting to the invention. Any suitable manifestation of these and other similar artifacts can be selected within the scope of the illustrative embodiments.

Application105implements an embodiment described herein. Application105makes use of NLP engine134to perform NLP-related tasks in a manner described herein.

Servers104and106, storage unit108, and clients110,112, and114, and device132may couple to network102using wired connections, wireless communication protocols, or other suitable data connectivity. Clients110,112, and114may be, for example, personal computers or network computers.

With reference toFIG. 3, this figure depicts a block diagram of an example configuration for cross-subject model-generated training data for relation extraction modeling in accordance with an illustrative embodiment. Application300is an example of application105in FIG.1and executes in any of servers104and106, clients110,112, and114, and device132inFIG. 1.

Annotator model training data generation module310constructs a set of training data. The training data includes a received sentence of narrative text. For a unit of narrative text, such as a word within the sentence, the training data also includes a received, human-generated label corresponding to the unit of narrative text. In one labeling scheme, a human-generated label is one of E, R, and O. E labels a unit of narrative text as an entity, R labels a unit of narrative text as a relation, and O labels a unit of narrative text as an other—i.e., not an entity or a relation. Module310also generates a machine-generated label corresponding to the unit of narrative text.

Annotator model training module320trains annotator model340. Once annotator model340has been trained, annotator model usage module330uses annotator model340to generate training data with which to train a relation extraction model to perform relation extractions.

With reference toFIG. 4, this figure depicts more detail of an example configuration for cross-subject model-generated training data for relation extraction modeling in accordance with an illustrative embodiment. In particular,FIG. 4depicts more detail of module310inFIG. 3.

Module310applies the unit of narrative text to the inputs of a set of rule-based relation extraction models, such as rule-based relation extraction model410, rule-based relation extraction model420, and rule-based relation extraction model430. Each model includes rules that recognize relations based on different elements or patterns. Module310uses, as the machine-generated label added to the training data, the results generated by all of the models.

With reference toFIG. 5, this figure depicts more detail of an example configuration for cross-subject model-generated training data for relation extraction modeling in accordance with an illustrative embodiment. Annotator model training module320and annotator model340are the same as annotator model training module320and annotator model340inFIG. 3.

Annotator model training module320uses training data510, produced by annotator training data generation module310inFIG. 3, to train annotator model340. Training data510for a sentence of narrative text includes unit of narrative text520, unit of narrative text530, and unit of narrative text540. Unit of narrative text520has corresponding human-generated label524and corresponding machine-generated label522. Unit of narrative text530has corresponding human-generated label534and corresponding machine-generated label532. Unit of narrative text540has corresponding human-generated label544and corresponding machine-generated label542. Training data510also includes part of speech data, and syntactic role data for each of narrative text520, unit of narrative text530, and unit of narrative text540(not shown).

With reference toFIG. 6, this figure depicts more detail of an example configuration for cross-subject model-generated training data for relation extraction modeling in accordance with an illustrative embodiment. Annotator model usage module330and annotator model340are the same as annotator model usage module330and annotator model340inFIG. 3.

Annotator model usage module330uses annotator model340, once trained, to produce machine-annotated training data from input narrative text. The machine-annotated training data can then be used to train a relation extraction model. The input narrative text need not be from the same knowledge domain as the training data used in training annotator model340.

With reference toFIG. 7, this figure depicts more detail of an example configuration for cross-subject model-generated training data for relation extraction modeling in accordance with an illustrative embodiment. In particular,FIG. 7depicts more detail of annotator model340inFIG. 3.

Annotator model340takes, as input, narrative text to annotate. The narrative text includes word702, word704, and word706, which are adjacent words within a sentence.

Input vector720feeds into forward LSTM730. The output of forward LSTM730feeds into an adjacent forward LSTM for use in assessing word704. The output of forward LSTM730also feeds into softmax function750. Input vector720also feeds into backward LSTM740. Backward LSTM740also receives input from an adjacent backward LSTM for use in assessing an impact of word704and word706on word702. The output of backward LSTM740also feeds into softmax function750. Softmax function750produces, as output, a set of probabilities, each probability corresponding to a likelihood that word702is functioning as an entity, relation, or neither within the input sentence. The highest probability output of softmax function750is taken as label712corresponding to word702, provided highest probability output is also above a threshold. In a similar fashion, annotator model340produces label714corresponding to word704, and produces label716corresponding to word706.

With reference toFIG. 8, this figure depicts an example of generating cross-subject model-generated training data for relation extraction modeling in accordance with an illustrative embodiment. In particular,FIG. 8depicts results of using annotator model340inFIG. 3once annotator model340has been trained.

Trained annotator model340takes, as input, sentence810: “Alpha Corp has released a new improved version of the BankIt application that runs on the Omega system.” Model340identifies “has released” as a relation within the sentence and labels each word of the phrase with an R, for relation. Model340determines that the relation operates on two entities, “Alpha Corp” and “a new improved version of the BankIt application”. Accordingly, model340labels each word of each entity phrase with an E, for entity. Because the remainder of the words within the sentence do not participate in the relation, model340labels each remaining word of the sentence with an O, for other. Annotated sentence820shows the resulting set of labels.

Model340also identifies, from sentence810, a different relation—“runs on”. Accordingly, model340labels each word of the relation phrase with an R. Model340determines that this relation operates on two entities, “a new improved version of the BankIt application” and “the Omega system”. Accordingly, model340labels each word of each entity phrase with an E. Because the remainder of the words within the sentence do not participate in the relation, model340labels each remaining word of the sentence with an P, for other. Annotated sentence830shows the resulting set of labels.

With reference toFIG. 9, this figure depicts a flowchart of an example process for cross-subject model-generated training data for relation extraction modeling in accordance with an illustrative embodiment. Process900can be implemented in application300inFIG. 3.

In block902, the application receives training data, divided into sentences. The application processes a narrative sentence at a time. For each sentence, the training data includes a first word and a human-generated label corresponding to the first word. In block904, the application uses one or more rule-based relation extraction models to generate a machine-generated label corresponding to the first word. In block906, the application generates a vector representation corresponding to the first word and the machine-generated label. In block908, the application uses the vector representation to train an annotator model. Training includes minimizing a difference between the human-generated label and an output of the annotator model. In block910, the application uses the trained annotator model and a second narrative text to generate an annotation corresponding to a word in the second narrative text. Then the application ends.

Thus, a computer implemented method, system or apparatus, and computer program product are provided in the illustrative embodiments for cross-subject model-generated training data for relation extraction modeling and other related features, functions, or operations. Where an embodiment or a portion thereof is described with respect to a type of device, the computer implemented method, system or apparatus, the computer program product, or a portion thereof, are adapted or configured for use with a suitable and comparable manifestation of that type of device.