SYSTEM AND METHOD FOR NATURAL LANGUAGE PROCESSING WITH PRETRAINED LANGUAGE MODELS

A computer-implemented system and method and for learning an entity-independent representation are disclosed. The method may include: receiving an input text; identifying named entities in the input text; replacing the named entities in the input text with entity markers; parsing the input text into a plurality of tokens; generating a plurality of token embeddings based on the plurality of tokens; generating a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text; generating a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and processing the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model to generate a hidden state vector for each of the plurality of tokens in the input text.

FIELD

Embodiments described herein relate to the field of natural language processing, and in particular, to systems and methods for training and improving one or more language models.

BACKGROUND

Pretrained Language Models (LMs) have been shown to have unmatched performance in a wide range of NLP tasks. However, these LMs could make incorrect predictions when some small perturbations are performed on input entities. Such small perturbations may include, for example, swapping a named entity (which may be referred to as simply “entity” throughout the disclosure herein) with a different named entity of the same class.

Named entities, in language models, refer to names representing real world objects, such as a person, location, organization, brand, product, and so on. For example, a name of a person (e.g., “John” or “John Lee”) can be a named entity. For example, a name of a geographical region, such as New York City, can be another named entity. For yet another example, “Microsoft”, name of a brand, can also be a named entity.

Generally speaking, named entities can be classified into one of several categories or classes: person, location, organization, and so on. The named entities “James” and “Mary” both belong to the same class: i.e., a person or a person's name. The named entity “Toronto” belongs to a different class: i.e., location.

With existing pretrained language models, the performance may be negatively affected when a named entity is swapped with a different named entity in a given input text, even if both named entities belong to the same class.

SUMMARY

In accordance with an aspect, there is provided a computer-implemented method for learning an entity-independent representation, the method comprising: receiving an input text; identifying one or more named entities in the input text; replacing the identified one or more named entities in the input text with one or more entity markers, each of the one or more entity markers corresponding to a respective named entity in the one or more identified named entities; parsing the input text including the one or more entity markers into a plurality of tokens; generating a plurality of token embeddings based on the plurality of tokens; generating a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text; generating a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and processing the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model (“the transformer model”) to generate a hidden state vector for each of the plurality of tokens in the input text.

In some embodiments, each token embedding for a respective token in the plurality of tokens includes a vector representation of fixed dimensions for the respective token.

In some embodiments, when a token in the plurality of tokens is not a named entity, the corresponding token type embedding has a first type value; wherein when a token in the plurality of tokens is a named entity, the corresponding token type embedding has a type value that is different from the first type value; and each unique named entity within the plurality of tokens has a unique type value for the corresponding token type embedding.

In some embodiments, the input text comprises a sentence and each token has a word in the sentence.

In some embodiments, parsing the input text into the plurality of tokens includes: adding a first token representing a beginning of the sentence before a first word of the sentence; adding a second token representing an end of the sentence after a last word of the sentence; and generating the plurality of tokens including the first token and the second token.

In some embodiments, the transformer model has an encoder block, the encoder block having a plurality of layers, and each of the plurality of layers has a multi-head self-attention mechanism and a feed forward network.

In some embodiments, the transformer model is trained based on a masked language modeling to predict masked words in an input sentence.

In some embodiments, the transformer model is trained to optimize a consistency loss Lc.

In some embodiments, the consistency loss Lcis based on:

where P is a probability distribution over a vocabulary during a forward pass on a training sentence, Q is a probability distribution over the vocabulary during a forward pass on a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and KL is a Kullback-Leibler divergence.

In some embodiments, the transformer model is trained to optimize a semantics loss Lsem.

In some embodiments, the semantics loss Lsemis based on:

where S1CLSrepresents a last layer output of the transformer model corresponding to a CLS token for a training sentence, S2CLSrepresents a last layer output of the transformer model corresponding to a CLS token for a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and MSE is the Mean Squared Error Loss.

In some embodiments, the transformer model is trained to optimize an overall loss based on:

where α, β and γ are hyperparameters, S1 is a training sentence, Lcis a consistency loss, Lsemis a semantics loss, and MLM is a masked language modeling loss.

In some embodiments, the transformer model is trained on a commonsense reasoning downstream task.

In some embodiments, the transformer model is trained on a sentiment analysis downstream task.

In accordance with another aspect, there is provided a computer system for learning an entity-independent representation, the system may include a processor and a memory in communication with the processor, the memory storing instructions that when executed, cause the processor to perform: receive an input text; identify one or more named entities in the input text; replace the identified one or more named entities in the input text with one or more entity markers, each of the one or more entity markers corresponding to a respective named entity in the one or more identified named entities; parse the input text including the one or more entity markers into a plurality of tokens; generate a plurality of token embeddings based on the plurality of tokens; generate a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text; generate a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and process the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model (“the transformer model”) to generate a hidden state vector for each of the plurality of tokens in the input text.

In some embodiments, each token embedding for a respective token in the plurality of tokens includes a vector representation of fixed dimensions for the respective token.

In some embodiments, when a token in the plurality of tokens is not a named entity, the corresponding token type embedding has a first type value; wherein when a token in the plurality of tokens is a named entity, the corresponding token type embedding has a type value that is different from the first type value; and each unique named entity within the plurality of tokens has a unique type value for the corresponding token type embedding.

In some embodiments, the input text comprises a sentence and each token has a word in the sentence.

In some embodiments, parsing the input text into the plurality of tokens includes: adding a first token representing a beginning of the sentence before a first word of the sentence; adding a second token representing an end of the sentence after a last word of the sentence; and generating the plurality of tokens including the first token and the second token.

In some embodiments, the transformer model has an encoder block, the encoder block having a plurality of layers, and each of the plurality of layers has a multi-head self-attention mechanism and a feed forward network.

In some embodiments, the transformer model is trained based on a masked language modeling to predict masked words in an input sentence.

In some embodiments, the transformer model is trained to optimize a consistency loss Lc.

In some embodiments, the consistency loss Lcis based on:

where P is a probability distribution over a vocabulary during a forward pass on a training sentence, Q is a probability distribution over the vocabulary during a forward pass on a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and KL is a Kullback-Leibler divergence.

In some embodiments, the transformer model is trained to optimize a semantics loss Lsem.

In some embodiments, the semantics loss Lsemis based on:

where S1CLSrepresents a last layer output of the transformer model corresponding to a CLS token for a training sentence, S2CLSrepresents a last layer output of the transformer model corresponding to a CLS token for a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and MSE is the Mean Squared Error Loss.

In some embodiments, the transformer model is trained to optimize an overall loss based on:

where α, β and γ are hyperparameters, S1 is a training sentence, Lcis a consistency loss, Lsemis a semantics loss, and MLM is a masked language modeling loss.

In some embodiments, the transformer model is trained on a commonsense reasoning downstream task.

In some embodiments, the transformer model is trained on a sentiment analysis downstream task.

In accordance with yet another aspect, there is provided a non-transitory computer-readable medium having computer executable instructions stored thereon for execution by one or more computing devices, the instructions, when executed, cause the one or more computing devices to: receive an input text; identify one or more named entities in the input text; replace the identified one or more named entities in the input text with one or more entity markers, each of the one or more entity markers corresponding to a respective named entity in the one or more identified named entities; parse the input text including the one or more entity markers into a plurality of tokens; generate a plurality of token embeddings based on the plurality of tokens; generate a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text; generate a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and process the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model to generate a hidden state vector for each of the plurality of tokens in the input text.

In this respect, before explaining at least one embodiment in detail, it is to be understood that the embodiments are not limited in application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.

DETAILED DESCRIPTION

Embodiments of methods, systems, and apparatus are described through reference to the drawings.

Traditional pretrained LMs learn different representations for each named entity (hereinafter simply “entity” or “entities”) that they encounter, and not only for each entity, but each context in which they see this entity. Such models can rely too much on specific entities, and fail to generalize across entities. Thus, their predictions can vary widely from just changing an entity.

To address pretrained LMs making incorrect predictions when small perturbations are done to the input entities, embodiments disclosed herein augment existing pretrained LMs to learn entity independent representations. Instead of learning representations to represent one specific entity, representations can be learned to represent the concept of an entity, which may give more consistent results regardless of the entities in the sentence. At the same time, these representations may be robust to different perturbations and can also generalize to unseen entities. Experimental work shows that the embodiments of entity-independent models disclosed herein may be robust to some entity-specific biases that can influence downstream tasks. The improved robustness can provide higher accuracy in downstream tasks, such as predicting a masked word in a given sentence, or predicting a relationship between two given sentences.

The embodiments disclosed herein can accelerate the learning of pretrained language models. Typically, the learning process for language models is data and time intensive. By increasing the speed of learning, the computing resources (e.g., data and/or time) required for training the pretrained language model is reduced.

Deep pretrained transformer (Vaswani et al., 2017) based language models (LMs) are typically trained on large amounts of text. On virtually every downstream natural language processing (NLP) task, these pretrained models have state-of-the-art performance. Models like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) have replaced task-specific NLP models based on static embeddings like GloVe (Pennington et al., 2014). Even though the language models tend to outperform traditional task-specific models based on static embeddings, they still have shortcomings.

Recent work like Trichelair et al. (2018) have shown that pretrained LMs make incorrect predictions in the Winograd Schema Challenge (WSC) test set when the entities in the input sentence are swapped (in an example, a name “Anne” is replaced with the name “Emily”). The traditional way to solve this task is to show enough perturbations like entity swapping during training and train the language model to become as robust as possible to these perturbations (Sakaguchi et al., 2019).

In embodiments disclosed herein, an alternative way to learn input text including named entity representations is disclosed, that may be robust to entity swaps with less performance degradation in the model. To achieve this goal, entity markers are introduced that are used to learn entity-independent representations and auxiliary loss functions are implemented. The auxiliary loss functions have a component that tries to mimic the masked language modeling loss introduced in Devlin et al. (2018) as well as a component specifically designed for entity-swap robustness.

Contextual representations may be learned for entities by using token type embeddings. Embodiments of the entity-independent model as disclosed herein may be able to learn entity-independent representations that generalize across multiple tasks.

Recent work (Shwartz et al., 2020) has also shown that the entity representations learnt by pretrained language models can perpetuate unintentional biases. These biases can then propagate to downstream tasks used to finetune these pretrained models. Experimental work as described herein shows how embodiments of the entity-independent models can be robust to these unintentional biases.

Models for learning entity-independent representations, which can be entity-independent and can also be entity-specific are disclosed herein. Both types of language models are based on pretrained language models (LMs). Pretrained LMs like BERT (Devlin et al., 2018) or RoBERTa (Liu et al., 2019) are usually trained using the Masked Language Modeling (MLM) objective, which involves predicting a masked token given a sequence of tokens.

Embodiments disclosed herein can modify the MLM objective to learn entity-independent representations. In some embodiments, input tokens are embedded with entity markers and entity-specific token types to represent entities. Furthermore, one or more modified auxiliary losses can be used in conjunction with MLM losses to learn the token-type representations and the entity-marker representations.

FIG. 1illustrates a system100for language modeling including an architecture of an entity-independent language model110, that learns entity-independent representations, in an embodiment. In some embodiments, the language model110uses a transformer neural network model180(hereinafter the “transformer model180”) to process a plurality of input170to generate a plurality of hidden state vectors190, which may be used for further language model training based one or more downstream tasks. The plurality of input170may be generated based on an input text102, which may be a single sentence.

Input text102can be tokenized to be represented as tokens, for example, either a full word or part of a word. Each token may be presented by Etoken, each token may include a unique value, which may be for example a unique numeric value, based on the word or string represented by the respective token, as further elaborated below.

The input text102may include one or more named entities. For example, the input text102may be “Ann asked Mary when she visited the library”. Both Ann and Mary are named entities. Entities such as named persons in a sentence can be identified using, in an example, Named Entity Recognizer (NER) provided with the Stanza package (Qi et al., 2020).

Tokens can represent entities. An entity can be a person or thing. In particular, an entity can be a “named entity”, in an example, names of people, countries, places, organizations, and the like, represented by proper nouns. A named entity can include, for example, a named person as discussed herein.

A specific type of token referred to as an entity marker120can be denoted by [E] or a different notation. Every entity, such as a person's name, in the input text120is replaced with this entity marker. In case an entity has more than one token (e.g., New York), all of the tokens are replaced with a single [E].

A reserved word in the RoBERTa vocabulary can be used to represent an entity marker, and therefore it may not be necessary to add any new tokens to the RoBERTa vocabulary, when the language model110is adapted to leverage the RoBERTa vocabulary.

Next, after each entity in the input text102has been replaced by an entity marker [E]120, the original input text102“Ann asked Mary when she visited the library” become “[E] asked [E] when she visited the library”.

In some embodiments, an input text may have different classes of entities, for example, “Ann asked Mary when she visited the New York Public Library.” In this case, in addition to “Ann” and “Mary”, “New York Public Library” is also a named entity. While “Ann” and “Mary” are entities belonging to a first class, e.g., person's names, “New York Public Library” is an entity belonging to a second class, e.g., physical buildings. In this case, a different entity marker [N] may be used to denote an entity for a different class, as compared to the first class. So the input text, after having replaced all entities with a respective entity marker, may read “[E] asked [E] when she visited the [N]”.

The text “[E] asked [E] when she visited the library” can be then processed by a tokenizer process of the system110. The tokenizer process may add a first token representing a beginning of the sentence before a first word of the sentence and a second token representing an end of the sentence after a last word of the sentence. For example, the tokenizer process may add a [CLS] token to the beginning of the sentence, and a [SEP] token to the end of the sentence. [CLS] may signal that the token immediately after [CLS] is the first token of the input text102, while [SEP] may signal that the token immediately prior to [SEP] is the last token of the input text102.

The tokenizer process can then generate a plurality of tokens130based on the sentence “[CLS] [E] asked [E] when she visited the library [SEP]”. Each of the plurality of tokens130in this example embodiment includes, respectively: [CLS], [E], asked, [E], when, she, visited, the, library, [SEP]. In some embodiments, the tokenizer process may be a pretrained machine learning model specifically configured to recognize tokens in an input text. For instance, the tokenizer process may be a WordPiece tokenization process.

In some embodiments, a hidden state vector of the [CLS] token as generated by the transformer model180may be used to represent some meanings of the entire input text.

Each token130in the plurality of tokens130may include a unique numerical value determined based on a vocabulary database.

In some embodiments, each of the tokens130may be looked up in a pre-existing vocabulary database, such as, for example, a RoBERTa vocabulary database or dictionary to determine a unique numerical value for representation of the respective token. Each token130may correspond to a specific and unique numerical value, which may be, for example, an index in the vocabulary database, then the unique numerical may be taken as the value for the respective token130. For example, the token Ewhenfor the word “when” may have a numerical value of 123 in the vocabulary database used; the token Eshefor the word “she” may have a numerical value of 256 in the vocabulary database used; and the token Evisitedfor the word “visited” may have a numerical value of 102 in the vocabulary database used. The tokens “EwhenEsheEvisited” (without the quotation marks) then have values “123 256 102” (without the quotation marks).

The system110may generate a plurality of token embeddings140, each of which may be denoted by, respectively: E[CLS], E[E], Easked, E[E], Ewhen, Eshe, Evisited, Ethe, Elibrary, E[SEP]. In some embodiments, the tokens130are processed by the system100into token embeddings140, each of which may include a vector representation of fixed dimensions, such as a 768-dimensional vector in Bidirectional Encoder Representations from Transformers (BERT).

The system110may generate a plurality of positional embeddings150based on a sequential position (e.g., from left to write in English) of each of the plurality of tokens130. A positional embedding150for a given token130can be a numerical value used to determine a position of the given token130within the plurality of tokens130. In the example tokens130shown inFIG. 1, the token [CLS] has a first position, which may be assigned a positional embedding E0, the token first [E] has a second position, which may be assigned a positional embedding E1, the token “asked” has a third position, which may be assigned a positional embedding E2, the token second [E] has a fourth position, which may be assigned a positional embedding E3, and so on. The positional embeddings150for the plurality of tokens130are therefore: E0, E1, E2, E3, E4, E5, E6, E7, E8, E9.

In some embodiments, each of the positional embeddings150may include a vector representation of fixed dimensions, such as a 768-dimensional vector in Bidirectional Encoder Representations from Transformers (BERT).

The system110may generate a plurality of token type embeddings160based on the plurality of tokens130and the original input text102. The token type embeddings160can be used to distinguish between different named entities and between entities and non-entities in the plurality of tokens130.

As described earlier, the entity marker [E]120provides a way for the model to identify entities. However, it may also be desirable to have a way to distinguish between different entities. Entities can be distinguished by adding entity-specific token type embeddings160to the existing token embeddings140. For example, the RoBERTa model in Liu et al. (2019) utilizes token types to distinguish between the current sentence and the subsequent sentence in the scenario when there are two sentences. As there is only one sentence in the input text102to this model110, the token types can be repurposed or augmented with entity-specific token types disclosed herein. This can be done by assigning a new token type to every unique entity. Thus, at the input layer of model110, each entity [E]120has a unique type embedding160.

For example, when a token in the plurality of tokens130is not a named entity, the corresponding token type embedding160can have a first type value; and when a token in the plurality of tokens130is a named entity, the corresponding token type embedding can have a type value that is different from the first type value. Furthermore, each unique named entity within the plurality of tokens130has a unique type value for the corresponding token type embedding160.

As shown inFIG. 1, a first type value, EA, for token type embedding160is assigned to tokens (e.g., [CLS], asked, etc.) that are not entities in the plurality of tokens130. A second type value, EB, for token type embedding160is assigned to the first entity marker token [E] which corresponds to the name Ann from the input text102. A third type value, EC, for token type embedding160is assigned to the second entity marker token [E] which corresponds to the name Mary from the input text102. As Ann and Mary are different (or unique) entities, the respective value for the respective token type embedding160is also unique.

In some embodiments, when the input text102has a second named entity (e.g., New York) that is of a different class than the first named entity (e.g., Ann), the corresponding token type embedding160may have a type value to indicate that the second named entity belongs to a different class. For example, if the token “Ann” has a token type embedding160EB, the token “New York” may have a respective token type embedding160EDD.

The input170to the transformer architecture or transformer model180includes at least the plurality of token embeddings140, the plurality of positional embeddings150and the plurality of token type embeddings160. In some embodiments, the plurality of token embeddings140, the plurality of positional embeddings150and the plurality of token type embeddings160may be vectors of fixed dimensions, and the input170may include a sum of the plurality of token embeddings140, the plurality of positional embeddings150and the plurality of token type embeddings160. In some embodiments, the plurality of tokens130is also input to the transformer model180.

The transformer architecture or transformer model180of N layers is used to process the input170and generate a plurality of hidden state vectors190: h[CLS], hAnn, hasked, hMary, hwhen, hshe, hvisited, hthe, hlibrary, h[SEP]. Each of these hidden state vector190may correspond to a respective token in the plurality of tokens130.

FIG. 2shows an example system200for language modelling with an entity-independent language model110configured for a downstream task230, according to some embodiments. The downstream task230may include further machine learning models configured to fine-tune or optimize the entity-independent language model110based on the plurality of hidden state vectors190. The output250from the downstream task230may be a prediction value, a probability value, or any other suitable value depending on the type of the downstream task230, which is elaborated further below.

In some embodiments, the output250may be further provided to an output device, which may be for example, a display monitor or a speaker circuit, to show the prediction result generated by the language model110based on at least an input text.

For example, the language model110, once trained and finetuned using the embodiments disclosed herein, may receive part of a sentence and predict the next word, which is the output250. In some embodiments, a smartphone keyboard may use the language model110to suggest the next word based on what a user has already typed into the input field.

In some embodiments, the transformer model180may be referred to as “Entity Independent RoBERTa” or “EI-RoBERTa”, as it may use a similar transformer architecture of N layers as used by the RoBERTa model.

In some embodiments, the transformer model180may include an encoder block185, the encoder block185having a plurality of N layers210a,210b. . .210n. Each layer210a,210b,210nmay have a multi-head self-attention mechanism220and a feed forward network230. The first layer210ais configured to process the input170(e.g., sum of the plurality of token embeddings140, the plurality of positional embeddings150and the plurality of token type embeddings160) and generate an output. Then each of the subsequent layers210b. . .210nis configured to process the output from the previous layer, iteratively one layer after another.

FIG. 3is a schematic diagram of an example neural network300that may be used to implement the feed forward network230, according to some embodiments. The example neural network300can include an input layer, a hidden layer, and an output layer. The neural network300processes input data using its layers based on weights, for example.

In some embodiments, the transformer model180may further include a decoder block (not shown). In some embodiments, a decoder block may include three components: a self-attention mechanism, an attention mechanism over the encodings, and a feed-forward neural network.

Downstream Task and Optimization Objective

In order to optimize the language model110, a masked language modeling to predict masked words in an input sentence may be implemented as a downstream task230. A loss function is implemented herein to learn positive representations for the entity markers120and the token type embeddings160. Considering the following example during training:

S1: Ann asked Mary what time the library [MASK], because she had forgotten.

S2: [E] asked [E] what time the library [MASK], because she had forgotten.

In the example above, S1 is a possible training example and S2 is the same sentence with the entities replaced with the entity markers [E]. A goal is to make sure that the masked token, denoted by [MASK], is predicted correctly by the language model110regardless of the entities provided to the model110.

A new loss function may be applied to achieve similar probability distributions over a given vocabulary at the [MASK] location for both sentences S1 and S2. Let the probability distribution over the given vocabulary during a forward pass on S1 be P, and the probability distribution over the vocabulary during a forward pass on S2 be Q, a consistency loss can be defined as:

where KL is the Kullback-Leibler divergence.

A given vocabulary may be an existing vocabulary database, such as a RoBERTa vocabulary. A forward pass is a pass of input (e.g., S1 or S2) through the transformer model180in one iteration or round.

Furthermore, replacing an entity by the corresponding entity markers [E] may preserve other linguistic properties of the original sentence such as the general sentiment of the sentence, its syntactic structure, and so on. Therefore, a special loss is added to preserve the semantics between S1 and S2.

In addition, to assure that other linguistic properties of the original sentence, including for example, a general sentiment of the sentence, its syntactic structure, and so on are preserved despite replacing an entity by the corresponding entity marker [E], a special loss may be added to preserve the semantics between S1 and S2.

Let S1CLSrepresent an output from the last layer of the encoder block of the transformer model180corresponding to the [CLS] token for S1, and let S2CLSrepresent an output from the last layer of the encoder block of the transformer model180corresponding to the [CLS] token for S2, a loss to preserve semantics between S1 and S2 can be defined by:

where MSE is the Mean Squared Error Loss.

In some embodiments, S1CLSis equivalent to h[CLS]fromFIG. 1when the input text102received by the system110is S1.

The optimized final loss is:

where α, β and γ are hyperparameters, and MLM is the masked language modeling loss.

Datasets and Tasks

Training Dataset

In some embodiments, the language model110is trained on the WikiText-2 dataset. This dataset contains 2 million tokens in the training data.

In some embodiments, a Named Entity Recognizer (NER) provided with the Stanza package (Qi et al., 2020) can be used to extract named entities. Named entities of type PERSON, in an example, can be extracted and assigned token type ids to each unique named entity per sentence.

The maximum number of entities of type PERSON possible per sentence may be set to 10. If a sentence has more than 10 named entities of type PERSON, it is removed from the training set. If there is only one named entity of type PERSON in a sentence, then the token type embedding160may be randomly assigned.

Commonsense Reasoning

One of the downstream tasks230that the language model110can be trained on is a Commonsense Reasoning task. One of the most popular datasets to test commonsense reasoning capabilities is Winogrande (Sakaguchi et al., 2019). The Winogrande task contains a sentence with a blank field, and two options for the blank field with one correct answer. The language model110, after being finetuned by the Commonsense Reasoning task, is responsible for predicting what the correct answer is for the blanked token.

Natural Language Inference

Another downstream task230that the language model110can be trained on is natural language inference. For this task, the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015) can be used.

The natural language inference task includes reading a premise and labeling a hypothesis as either entailed by the premise, in contradiction with the premise, or neutral with respect to the premise. For instance, the hypothesis “Some men are playing a sport” is entailed by the premise “A soccer game with multiple males playing”.

The language model110can be tested on the original test set of SNLI as well as the two test sets proposed by Mitra et al. (2019). The first test set named “Named Change” contains premises with one named entity and hypotheses which are similar to the premises except that the named entity is changed. For instance, a premise is “John went to the kitchen” and the corresponding hypothesis is “Peter went to the kitchen”. A properly trained language model110should label this hypothesis as contradictory. The second test set named “Role Switched” contains premises with two entities and hypotheses that are similar to the premises except that the entities are switched. For example, a premise is “Kendall lent Peyton a bicycle” and the corresponding hypothesis is “Peyton lent Kendall a bicycle”. Again, the correct label is contradiction. These test sets are configured to test whether models trained on the SNLI training dataset understood the role of entities.

Sentiment Analysis

Another downstream task230that the language model110can be trained on is sentiment analysis. For this task, the Stanford sentiment treebank dataset can be used. The model used can be similar to Liu et al. (2019). Sentiment analysis can be used to classify a sentiment of a sentence as “positive” or “negative”.

Results

In experimental work, the Winogrande dataset has been used to evaluate the commonsense reasoning capabilities of model110as a pretrained LM.FIG. 4Ais a table of results for model complexity evaluated on the Winogrande development set, according to an embodiment.

FIG. 4Bis a table of results for models evaluated on two Winogrande development sets, the original one as well as a development set containing only entities that were not included in the training set, according to an embodiment. From the results illustrated in the table ofFIG. 4B, it can be seen that the language model110has a similar performance to the RoBERTa model finetuned on WikiText-2.

To test the generalization capabilities of the LMs to unseen entities, another development set is created, where the entities in the development set are never seen during training. The result was a decrease in performance for both RoBERTa and RoBERTa finetuned on WikiText2. However, performance of the language model110does not change. This may be attributed to the fact that model110learns entity-independent representations as opposed to RoBERTa, which learns separate representations for each entity.

An embodiment of the language model110was also tested on the sentiment classification task with the Stanford Sentiment Treebank to test the language model110. A separate test set was created where the first entity of each sentence was replaced with the token “Trump”. This was done to determine if entity representations extracted from pretrained LMs have some inherent bias that influences the sentiment classification.

FIG. 4Cillustrates models evaluated on a modified sentiment analysis test set, such as Stanford Sentiment Treebank (SST) test set. In testing, the performance of both RoBERTa and RoBERTa finetuned models drops on the test set with entities replaced with “Trump”. This suggests that the entity representations are influencing the final sentiment classification for these models. The language model110(e.g., EI-RoBERTa) performs better than the RoBERTa baseline models on the test set with replaced entities. This is suggestive of the fact that, through the entity markers and token type embeddings, the language model110is able to learn entity-independent representations and therefore the entity representations do not tend to influence the sentiment classification predictions.

FIG. 4Dillustrates models evaluated on SNLI test set. On SNLI, as shown inFIG. 4D, the language model110performs at a similar level as other models on the modified test sets. The performance of the language model110may be due to not having seen examples of this type in the training data, rather than not understanding entities. Further experiments have been performed to test this hypothesis where, during training, examples are progressively added from the modified training sets. The language model110is expected to learn to generalize to examples in the test sets with fewer training samples than BERT or RoBERTa.

Conveniently, existing language models can be augmented using embodiments herein to learn entity-independent representations. As shown in testing described above, embodiments of an entity-independent language model can generalize to unseen entities on the Winogrande task. Further, embodiments of an entity-independent language model may rely less on the identity of the entities while doing sentiment classification.

FIG. 5Aillustrates an embodiment of a method500for learning an entity-independent representation using entity-independent language model110. The steps or blocks are provided for illustrative purposes. Variations of the steps, omission or substitution of various steps, or additional steps may be considered. It should be understood that one or more of the blocks may be performed in a different sequence or in an interleaved or iterative manner.

At block501, an input text is received. The input text may be a sentence having a plurality of words.

At block502, input text is tokenized into a plurality of tokens, for example, either a full word or part of a word. Each token may be presented by Etoken, each token may include a unique value, which may be for example a unique numeric value, based on the word or string represented by the respective token, as further elaborated below.

At block504, entities in the plurality of tokens are identified. Entities such as named persons in a sentence can be identified using, in an example, Named Entity Recognizer (NER) provided with the Stanza package (Qi et al., 2020).

At block506, the tokens of the entities are replaced with an entity marker token. A specific type of token referred to as an entity marker can be denoted by [E] or a different notation. Every entity, such as a person's name, in the input text120is replaced with this entity marker. In case an entity has more than one token (e.g., New York), all of the tokens are replaced with a single [E].

At block508, unique entities in the plurality of tokens are identified. A unique entity means an entity that is different from the other entities.

At block510, a token type embedding is assigned to each of the unique entities. For example, when a token in the plurality of tokens is not a named entity, the corresponding token type embedding can have a first type value; and when a token in the plurality of tokens is a named entity, the corresponding token type embedding can have a type value that is different from the first type value. Furthermore, each unique named entity within the plurality of tokens has a unique type value for the corresponding token type embedding.

In some embodiments, the language model110is trained to a masked language modeling objective to predict masked words in a sentence.

In some embodiments, the language model110is trained to optimize a consistency loss

In some embodiments, the consistency loss Lcis based on:

where P is a probability distribution over a given vocabulary during a forward pass on a training sentence, Q is a probability distribution over the vocabulary during a forward pass on a sentence based on the training sentence with entities replaced with entity markers, and KL is a Kullback-Leibler divergence.

In some embodiments, the language model110is trained to optimize a semantics loss Lsem.

In some embodiments, the semantics loss Lsemis based on:

where S1CLSrepresents a last layer output of the transformer model corresponding to a CLS token for a training sentence, S2CLSrepresents a last layer output of the transformer model corresponding to a CLS token for a sentence based on the training sentence with entities replaced with entity markers, and MSE is the Mean Squared Error Loss.

In some embodiments, the language model110is trained to optimize an overall loss based on:

where α, β and γ are hyperparameters, S1 is a training sentence, Lcis a consistency loss, Lsemis a semantics loss, and MLM is a masked language modeling loss.

In some embodiments, model110is trained on a commonsense reasoning downstream task.

In some embodiments, model110is trained on a sentiment analysis downstream task.

In some embodiments, words in an input sentence can be predicted using model110.

FIG. 5Billustrates an embodiment of a another computer-implemented method520for learning an entity-independent representation using entity-independent language model110. The method520may be performed by system100or200. The steps or blocks are provided for illustrative purposes. Variations of the steps, omission or substitution of various steps, or additional steps may be considered. It should be understood that one or more of the blocks may be performed in a different sequence or in an interleaved or iterative manner.

At block521, the system100may receive an input text102. In some embodiments, the input text102is a sentence and each token is a word in the sentence. For example, the input text102may be “Ann asked Mary when she visited the library”.

At block523, the system100,200may identify one or more named entities in the input text. The input text102may include one or more named entities. Both Ann and Mary are named entities in the input text102“Ann asked Mary when she visited the library”. Entities such as named persons in a sentence can be identified using, in an example, Named Entity Recognizer (NER) provided with the Stanza package (Qi et al., 2020).

At block525, the system100,200may replace the identified one or more named entities in the input text102with one or more entity markers120, each of the one or more entity markers120corresponding to a respective named entity in the one or more identified named entities.

An entity marker120can be denoted by [E] or a different notation. Every entity, such as a person's name, in the input text120is replaced with this entity marker. In case an entity has more than one token (e.g., New York), all of the tokens are replaced with a single [E].

After each entity in the input text102has been replaced by an entity marker [E]120, the original input text102“Ann asked Mary when she visited the library” become “[E] asked [E] when she visited the library”.

At block527, the system100,200may parse the input text102including the one or more entity markers [E] into a plurality of tokens130. Each token may be presented by Etoken, each token may include a unique value, which may be for example a unique numeric value, based on the word or string represented by the respective token.

The text “[E] asked [E] when she visited the library” can be then processed by a tokenizer process of the system100,200. The tokenizer process may add a first token representing a beginning of the sentence before a first word of the sentence and a second token representing an end of the sentence after a last word of the sentence. For example, the tokenizer process may add a [CLS] token to the beginning of the sentence, and a [SEP] token to the end of the sentence. [CLS] may signal that the token immediately after [CLS] is the first token of the input text102, while [SEP] may signal that the token immediately prior to [SEP] is the last token of the input text102.

The tokenizer process can then generate a plurality of tokens130based on the sentence “[CLS] [E] asked [E] when she visited the library [SEP]”. Each of the plurality of tokens130in this example embodiment includes, respectively: [CLS], [E], asked, [E], when, she, visited, the, library, [SEP]. In some embodiments, the tokenizer process may be a pretrained machine learning model specifically configured to recognize tokens in an input text. For instance, the tokenizer process may be a WordPiece tokenization process.

In some embodiments, each of the tokens130may be looked up in a pre-existing vocabulary database, such as, for example, a RoBERTa vocabulary database or dictionary to determine a unique numerical value for representation of the respective token. Each token130may correspond to a specific and unique numerical value, which may be, for example, an index in the vocabulary database, then the unique numerical may be taken as the value for the respective token130. For example, the token Ewhenfor the word “when” may have a numerical value of 123 in the vocabulary database used; the token Eshefor the word “she” may have a numerical value of 256 in the vocabulary database used; and the token Evisitedfor the word “visited” may have a numerical value of 102 in the vocabulary database used. The tokens “EwhenEsheEvisited” (without the quotation marks) then have values “123 256 102” (without the quotation marks).

At block530, the system100,200may generate a plurality of token embeddings140based on the plurality of tokens130. Each of the plurality of token embeddings140may be denoted by, respectively: E[CLS], E[E], Easked, E[E], Ewhen, Eshe, Evisited, Ethe, Elibrary, E[SEP]. In some embodiments, the tokens130are processed by the system100into token embeddings140, each of which may include a vector representation of fixed dimensions, such as a 768-dimensional vector in Bidirectional Encoder Representations from Transformers (BERT).

At block532, the system100,200may generate a plurality of positional embeddings150based on the respective position of each of the plurality of tokens130.

A positional embedding150for a given token130can be a numerical value used to determine a position of the given token130within the plurality of tokens130. In the example tokens130shown inFIG. 1, the token [CLS] has a first position, which may be assigned a positional embedding E0, the token first [E] has a second position, which may be assigned a positional embedding E1, the token “asked” has a third position, which may be assigned a positional embedding E2, the token second [E] has a fourth position, which may be assigned a positional embedding E3, and so on. The positional embeddings150for the plurality of tokens130are therefore: E0, E1, E2, E3, E4, E5, E6, E7, E8, E9.

In some embodiments, each of the positional embeddings150may include a vector representation of fixed dimensions, such as a 768-dimensional vector in Bidirectional Encoder Representations from Transformers (BERT).

At block533, the system100,200may generate a plurality of token type embeddings160based on the plurality of tokens130and the one or more named entities in the input text102.

Entities can be distinguished by adding entity-specific token type embeddings160to the existing token embeddings140. For example, the RoBERTa model in Liu et al. (2019) utilizes token types to distinguish between the current sentence and the subsequent sentence in the scenario when there are two sentences. As there is only one sentence in the input text102to this model110, the token types can be repurposed or augmented with entity-specific token types disclosed herein. This can be done by assigning a new token type to every unique entity. Thus, at the input layer of model110, each entity [E]120has a unique type embedding160.

For example, when a token in the plurality of tokens130is not a named entity, the corresponding token type embedding160can have a first type value; and when a token in the plurality of tokens130is a named entity, the corresponding token type embedding can have a type value that is different from the first type value. Furthermore, each unique named entity within the plurality of tokens130has a unique type value for the corresponding token type embedding160.

As shown inFIG. 1, a first type value, EA, for token type embedding160is assigned to tokens (e.g., [CLS], asked, etc.) that are not entities in the plurality of tokens130. A second type value, EB, for token type embedding160is assigned to the first entity marker token [E] which corresponds to the name Ann from the input text102. A third type value, EC, for token type embedding160is assigned to the second entity marker token [E] which corresponds to the name Mary from the input text102. As Ann and Mary are different (or unique) entities, the respective value for the respective token type embedding160is also unique.

Blocks530,532and533may be performed concurrently, or one after another, or in parallel, or in combination of any order.

At block540, the system100,200may process the plurality of token embeddings140, the plurality of positional embeddings150, and the plurality of token type embeddings160using a transformer neural network model (“the transformer model”)180to generate a plurality of hidden state vectors h550, where each hidden state vector corresponds to a respective token of the plurality of tokens130.

In some embodiments, the plurality of token embeddings140, the plurality of positional embeddings150and the plurality of token type embeddings160may be vectors of fixed dimensions, and the input170may include a sum of the plurality of token embeddings140, the plurality of positional embeddings150and the plurality of token type embeddings160. In some embodiments, the plurality of tokens130is also input to the transformer model180.

The transformer architecture or transformer model180of N layers is used to process the input170and generate a plurality of hidden state vectors: h[CLS], hAnn, hasked, hMary, hwhen, hshe, hvisited, hthe, hlibrary, h[SEP]. Each of these hidden state vector550may correspond to a respective token in the plurality of tokens130.

In some embodiments, the transformer model180has an encoder block185, the encoder block comprising a plurality of layers, and each of the plurality of layers includes a multi-head self-attention mechanism and a feed forward network.

In some embodiments, the transformer model180is trained based on a masked language modeling to predict masked words in an input sentence.

In some embodiments, the transformer model180is trained to optimize a consistency loss Lc.

In some embodiments, the consistency loss Lcis based on:

where P is a probability distribution over a given vocabulary during a forward pass on a training sentence, Q is a probability distribution over the vocabulary during a forward pass on a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and KL is a Kullback-Leibler divergence.

In some embodiments, the transformer model is trained to optimize a semantics loss Lsem.

In some embodiments, the semantics loss Lsemis based on:

where S1CLSrepresents a last layer output of the transformer model corresponding to a CLS token for a training sentence, S2CLSrepresents a last layer output of the transformer model corresponding to a CLS token for a sentence based on the training sentence with entities in the training sentence replaced with entity markers, and MSE is the Mean Squared Error Loss.

In some embodiments, the transformer model180is trained to optimize an overall loss based on:

where α, β and γ are hyperparameters, S1 is a training sentence, Lcis a consistency loss, Lsemis a semantics loss, and MLM is a masked language modeling loss.

In some embodiments, the transformer model180is trained on a commonsense reasoning downstream task.

In some embodiments, the transformer model180is trained on a sentiment analysis downstream task.

System100,200for language modeling may be implemented as software and/or hardware, for example, in a computing device600as illustrated inFIG. 6. Method500, in particular, one or more of blocks502to510, may be performed by software and/or hardware of a computing device such as computing device600.

FIG. 6is a high-level block diagram of computing device600. Computing device600, under software control, may train entity-independent language model110and use a trained entity-independent language model110to model language and generate predictions.

As illustrated, computing device600includes one or more processor(s)610, memory620, a network controller630, and one or more I/O interfaces640in communication over bus650.

Processor(s)610may be one or more Intel x86, Intel x64, AMD x86-64, PowerPC, ARM processors or the like.

Memory620may include random-access memory, read-only memory, or persistent storage such as a hard disk, a solid-state drive or the like. Read-only memory or persistent storage is a computer-readable medium. A computer-readable medium may be organized using a file system, controlled and administered by an operating system governing overall operation of the computing device.

Network controller630serves as a communication device to interconnect the computing device with one or more computer networks such as, for example, a local area network (LAN) or the Internet.

One or more I/O interfaces640may serve to interconnect the computing device with peripheral devices, such as for example, keyboards, mice, video displays, and the like. Such peripheral devices may include a display of device600. Optionally, network controller630may be accessed via the one or more I/O interfaces.

Software instructions are executed by processor(s)610from a computer-readable medium. For example, software may be loaded into random-access memory from persistent storage of memory620or from one or more devices via I/O interfaces640for execution by one or more processors610. As another example, software may be loaded and executed by one or more processors610directly from read-only memory.

Example software components and data stored within memory620of computing device600may include software to perform language modeling, as disclosed herein, and operating system (OS) software allowing for basic communication and application operations related to computing device600.

The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and particularly configured computer hardware arrangements.

Applicant notes that the described embodiments and examples are illustrative and non-limiting. Practical implementation of the features may incorporate a combination of some or all of the aspects, and features described herein should not be taken as indications of future or existing product plans. Applicant partakes in both foundational and applied research, and in some cases, the features described are developed on an exploratory basis.

Although the embodiments have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein.

Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification.

REFERENCES