Token packing for sequence models

Embodiments of the present disclosure include systems and methods for packing tokens to train sequence models. In some embodiments, a plurality of datasets for training a sequence model is received. Each dataset in the plurality of datasets includes a sequence of correlated tokens. A set of training data is generated that includes a subset of a sequence of tokens from a first dataset in the plurality of datasets and a subset of a sequence of tokens from a second, different dataset in the plurality of datasets. The sequence model is trained using the set of training data.

BACKGROUND

The present disclosure relates to a computing system. More particularly, the present disclosure relates to techniques for training a neural network.

Natural-language understanding (NLU) is a subfield of natural-language processing (NLP) in artificial intelligence that addresses comprehension by computers of the structure and meaning of human language. NLU enables voice technology, search engines, and machine translation to deduce what a user means, regardless of the way it is expressed

A neural network is a machine learning model that underpins NLU applications. A neural network is trained for a particular purpose by running datasets through it, comparing results from the neural network to known results, and updating the network based on the differences.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. Such examples and details are not to be construed as unduly limiting the elements of the claims or the claimed subject matter as a whole. It will be evident to one skilled in the art, based on the language of the different claims, that the claimed subject matter may include some or all of the features in these examples, alone or in combination, and may further include modifications and equivalents of the features and techniques described herein.

Described here are techniques for packing tokens into input data used to train sequence models. In some embodiments, a system is used to perform a number of different token packing techniques. For example, the system can receive a set of input data that includes a sequence (e.g., a set of sentences) of tokens (e.g., words). The system may group the sequence of tokens into several groups of tokens (e.g., groups of words that form a sentence). Next, the system can generate input data for training a sequence model by packing a fixed length data structure with each group of tokens. With the remaining unused length left in the defined length of the data structure, the system continues packing the data structure with copies of the groups of words until the length of the data structure has been filled up with tokens.

As another example, the system may generate input data for training a sequence model by packing a first row of a fixed length data structure with a first group of tokens until the length of the first row of the data structure is filled up with tokens from the first group of tokens. Then, the system packs a second row of the fixed length data structure with a second group of tokens until the length of the second row of the data structure is filled up with tokens from the second group of tokens. The system continues to pack subsequent rows of the fixed length data structure with remaining groups of tokens in the same manner.

In another example, the set of input data that the system receives from several datasets (e.g., paragraphs of text, pages of text, documents of text, etc.) that each includes a sequence (e.g., a set of sentences) of correlated tokens. Tokens in the sequences of correlated tokens can be grouped (e.g., based on sentences) together based on their correlation. The input data that the system generates for training a sequence model includes different groups of tokens from different datasets based on the lengths of the different groups of tokens in order to optimally fill up fixed length data structures with groups of tokens.

The techniques described in the present application provide a number of benefits and advantages over conventional methods of training a sequence model. For instance, the various token packing techniques used to generate input data to train the sequence model can increase the speed at which weights of the sequence model reach convergence. In other words, using any of these techniques results in faster training of the sequence model.

FIG.1illustrates a system100for training a sequence model according to some embodiments. As shown, system100includes input data processor105, sequence module110, and output data processor115. Input data processor105is configured to processor input data used for training sequence module110. For example, input data processor105may receive a set of input data that includes a sequence (e.g., a set of sentences) of tokens (e.g., words) and position values for the sequence of tokens. In some embodiments, a position value represents the relative position of a particular token in a sequence of tokens. In some such embodiments, a set of input data can also include a set of sentence values. A sentence value may represent a sentence to which a token in the sequence of tokens belongs. The sentence value can also be used to determine correlations between sentences of words. For example, tokens in a first sentence can each have a sentence value of zero (“0”) and tokens in a second sentence can each have a sentence value of one (“1”). These sentence values indicate that the second sentence follows the first sentence and the two sentences are thus correlated. In some embodiments, the sentence values may be set to a special defined value to indicate that sentences of words are not correlated.

After receiving the set of input data, input data processor105can use several different token packing techniques to generate training data for training sequence module110. Details of token packing techniques are described further below. Once input data processor105finishes generating training data, input data processor105can add label data to the training data. The label data can include the actual next sequence of tokens and data indicating whether tokens in the sequence of tokens not correlated. Then, input data processor105sends the training data and label data to sequence module110for training.

Sequence module110is responsible for a next sequence of tokens for a given input sequence of tokens. In some embodiments, sequence module110includes a sequence model. Different types of neural networks may be used to implement the sequence model. For example, a transformer neural network (also referred to as a transformer or a transformer model) can be used. Examples of transformer models include a bidirectional encoder representations from transformers (BERT) model, a generative pre-training (GPT) model, a GPT-2 model, a robustly optimized BERT pretraining approach (RoBERTa) model, a distilled version of BERT (DistiliBERT) model, an XLNet model, etc.

A transformer neural network has a sequence-to-sequence architecture. That is, the transformer neural network can transforms a given sequence of elements, such as the sequence of words in a sentence, into another sequence. In some embodiments, the transformer neural network includes weights used for predicting the next sequence of tokens. The transformer neural network can adjust these weights based on feedback (e.g., differences between a predicted next sequence of tokens and the actual next sequence of tokens) received from output data processor115using a back propagation technique. Another type of neural networks that may be utilized is a recurrent neural network (RNN).

Sequence module110may determine relationships/correlations between tokens in input data. For instance, sequence module110can process tokens in relation to all the other tokens in a sequence, instead of one-by-one in order. In other words, sequence module110considers the full context of a token by looking at the tokens that come before and after it. Sequence module110may be used for machine translation and search (e.g., conversational queries). Other applications of sequence module110include: document summarization, document generation, named entity recognition (NER), speech recognition, and biological sequence analysis.

When sequence module110receives training data from input data processor105, sequence module110can determine token embeddings for each token in the sequence of tokens using an embedding space generated from a corpus of tokens (e.g., a vocabulary of words). In some embodiments, a token embedding space maps tokens in the corpus, which has many dimension, to numeric representations (e.g., vectors) having a lower number of dimensions. The numeric representation of a token can be a vector of 128, 256, 1024, 2048, 4096, etc. floating-point numbers. Then, sequence module110can determine position embeddings for each unmasked position value in the set of position values using an embedding space generated from a corpus of position values. In some embodiments, a position value embedding space maps position values in the corpus, which has many dimension, to numeric representations (e.g., vectors) having a lower number of dimensions. The numeric representation of a position value can be a vector of 128, 256, 1024, 2048, 4096, etc. floating-point numbers. In cases where the input data includes sentence values, input data processor105may determine sentence embeddings for each sentence value in the set of sentence values using an embedding space generated from a corpus of sentence values. In some embodiments, a sentence value embedding space maps sentence values in the corpus, which has many dimension, to numeric representations (e.g., vectors) having a lower number of dimensions. The numeric representation of a sentence value can be a vector of 128, 256, 1024, 2048, 4096, etc. floating-point numbers. After determining embeddings for tokens, position values, and/or sentence values, sequence module110calculates an aggregate embedding for each token in the sequence of tokens by adding the token embedding, the corresponding position value embedding, and/or the corresponding sentence value embedding together. Sequence module110then uses the aggregate embeddings for training.

Output data processor115is configured to process data output from sequence module110. For example, output data processor115can receive an array of data from sequence module110and label data. The array of data may include a numeric representation (e.g., the aggregate embedding described above) for each token in a sequence of tokens used as input to sequence module110. The label data can include the actual next sequence of tokens and data indicating whether tokens in the sequence of tokens not correlated. Next, output data processor115determines the predicted sequence of tokens. Output data processor115then determines the differences between the predicted sequence of tokens and the actual next sequence tokens specified in the label data. Also, output data processor115determines the differences between the predicted correlation between tokens in the sequence of tokens and the actual correlation between tokens in the sequence of tokens specified in the label data. Finally, output data processor115sends the calculated differences back to sequence module110to adjust the weights of sequence module110.

FIG.2illustrates an architecture of input data processor105according to some embodiments. As shown, input data processor105includes token packing repeater205, token packing stacker210, and token packing optimizer215. As shown, input data processor105receives input data220. Input data processor105may be configured to use either token packing repeater205, token packing stacker210, or token packing optimizer215to process input data220. That is, only one component of input data processor105can be used to process a given input data.

Token packing repeater205is configured to generate training data using a repeating token packing technique. An example of this technique will be described by reference toFIGS.3and4.FIG.3illustrates a process300for a repeating token packing technique according to some embodiments. In some embodiments, system100performs process300. Process300begins by receiving, at310, a set of input data for training a sequence model. The input data includes a sequence of tokens. For example, token packing repeater205can receive input data220. For this example, input data includes a sequence that includes two sentences: “The cat sat.” and “The dog laid.”

Next, process300, groups, at320, the sequence of tokens into a set of groups of tokens. For example, token packing repeater205can group the sequence of tokens into the set of groups of tokens. In some embodiments, process300groups the sequence of tokens into groups of tokens according to sentences. Using the example sequence above, process300groups the sequence of tokens into two groups. The first group of tokens is the first sentence, “The cat sat,” and the second group of tokens is the second sentence, “The dog laid.”

Process300then generates, at330, a set of training data that includes the set of groups of tokens and copies of at least a portion of a group of tokens in the set of groups of tokens. For example, token packing repeater205may generate the set of training data. In some embodiments, the set of training data that process300generates includes a data structure (e.g., an array) having a defined length (e.g., length of tokens). In some such embodiments, process300packs the data structure with the each group of tokens. Then, process300packs the remaining unused length of the data structure with repeated copies of the groups of tokens until the data structure is filled up with tokens.

FIG.4illustrates an example of training data400that includes repeated tokens according to some embodiments. Specifically, training data400is training data that process300generates using the example sequence of tokens above. As shown, training data400is a data structure (e.g., an array) that includes tokens405, position values410, and sentence values415. As mentioned above, a position value can represent the relative position of a particular token in a sequence of tokens and a sentence value can represent a sentence to which a token in the sequence of tokens belongs. In this example, the data structure that process300generates for the training data has a defined length of fourteen. That is, the data structure is configured to store fourteen tokens. As shown, process300packs the first three tokens of tokens405with each word in the first sentence (“The cat sat.”). Then, process300packs the next three tokens in tokens405with each word in the second sentence (“The dog laid”). Next, process300repeatedly packs tokens405with copies of the first sentence followed by the second sentence followed by the first sentence and so on and so forth until the fourteen spaces in tokens405is filled up. Returning toFIG.3, process300finally trains, at340, the sequence model using the set of training data. For example, training data400is used to train sequence module110.

Token packing stacker210serves to generate training data using a stacking token packing technique. Similar to token packing repeater205, when token packing stacker210receives input data to process (e.g., input data220), token packing stacker210groups the sequence of tokens into a set of groups of tokens. In some embodiments, token packing stacker210groups the sequence of tokens into groups of tokens according to sentences. Using the example sequence of tokens above, token packing stacker210groups the sequence of tokens into two groups. The first group of tokens is the first sentence, “The cat sat,” and the second group of tokens is the second sentence, “The dog laid.”

Next, token packing stacker210generates a set of training data that includes a data structure (e.g., an array) having a defined length (e.g., length of tokens). For each group of tokens, the data structure has a row to store tokens. Token packing stacker210then iterates through each group of tokens, stores the group of tokens, and repeatedly stores copies of the group of tokens in the respective row of the data structure until the row is filled up with tokens. Using the example groups of tokens above, token packing stacker210would generate a set of training data that includes a data structure that has two rows for storing tokens. A first row is used to store tokens in the first group of tokens and a second row is used to store tokens in the second group of tokens.

FIG.5illustrates an example of training data500that includes stacked tokens according to another embodiment. In particular, training data500is training data that token packing stacker210generates using the example sequence of tokens above. As shown, training data500is a data structure (e.g., an array) that includes tokens505, tokens,510, position values515, and sentence values520. For this example, the data structure that token packing stacker210generates for the training data has a defined length of fourteen. In other words, the data structure is configured to store fourteen tokens. As shown inFIG.5, the first row, tokens505, is for storing tokens in the first sentence and the second row, tokens510, is for storing tokens in the second sentence. Here, token packing stacker210stores each word in the first sentence, “The cat sat,” in tokens505and then repeatedly stores copies of the first sentence in tokens505until tokens505is filled up with tokens. Similarly, token packing stacker210stores each word in the second sentence, “The dog laid,” in tokens510and then repeatedly stores copies of the second sentence in tokens510until tokens510is filled up with tokens.

Returning toFIG.2, token packing optimizer215is configured to generate training data using an optimizing token packing technique. An example of such a technique will be described by reference toFIGS.6-9.FIG.6illustrates a process600for an optimized token packing technique according to some embodiments. In some embodiments, system100performs process600. Process600begins by receiving, at610, a plurality of datasets for training a sequence model. Each dataset in the plurality of datasets includes a sequence of correlated tokens. For example, token packing optimizer215can receive input data220.FIG.7illustrates an example set of correlated sequences of tokens705-715according to some embodiments. In this example, each of the sequences of tokens705-715is a separate paragraph of text that represents a dataset. That is, sequence of tokens705is a first paragraph of text that represents a first dataset, sequence of tokens is a second paragraph of text that represents a second dataset, and sequence of tokens715is a third paragraph of text that represents a third dataset. Thus, on a paragraph to paragraph basis, the sentences within each of the sequence of tokens705-715are correlated and sentences from different sequences of tokens705-715are not correlated.

Next, process600generates, at620, a set of training data that includes a subset of a sequence of tokens from a first dataset in the plurality of datasets and a subset of a sequence of tokens from a second, different dataset in the plurality of datasets. For example, token packing repeater205can generate the set of training data. In some embodiments, process600generates a set of training data by determining, for the sequence of tokens of each dataset in the plurality of datasets, a set of groups of tokens in the sequence of tokens. In some embodiments, process600groups sentences as groups of tokens. Referring toFIG.7as an example, process600groups each sentence in sequence of tokens705-715as a separate group of tokens. Specifically, process600groups “John ran too fast” as a first group of tokens, “He fell” as a second group of tokens, “Jane went home” as a third group of tokens, “She did not feel well” as a fourth group of tokens, and “The car was on the left” as a fifth group of tokens. Next, process600may determine, for each group of tokens in the set of groups of tokens of each sequence of tokens, a length of the group of tokens. Referring toFIG.7as an example, process600determines the length of the first group of tokens as four, the length of the second group of tokens as two, the length of the third group of tokens as three, the length of the fourth group of tokens as five, and the length of the group of tokens as six.

In some embodiments, the set of training data that process600generates includes data structures (e.g., arrays) that each have a defined length (e.g., length of tokens). Process600may optimally packs such data structures with tokens from the plurality of datasets. For instance, in some embodiments, process600packs a data structure by identifying a group of tokens in the plurality of datasets having the longest length equal to or less than the defined length of the data structure. Process600iteratively packs the data structure with remaining groups of tokens in the plurality of datasets having the longest length that is equal to or less than the remaining length in the data structure. After a data structure is filled up with tokens, process600continues to generate a data structure and pack it in the same manner until there are no more groups of tokens left.

FIG.8illustrates training data800generated from the example set of correlated sequences of tokens illustrated inFIG.6according to some embodiments. In particular, training data800is the first data structure that process600generates based on the methodology described above. For this example, the data structures that process600generates for the training data has a defined length of ten. That is, the data structure is configured to store ten tokens. As shown, process600packs training data800by identifying group of tokens (sentences in this example) having the longest length that is equal to or less than the defined length of the data structure (ten in this example). Here, the sentence “The car was on the left” has the longest length that is less than the data structure defined length of ten. As such, process600fills training data800with the sentence. With a remaining, unused length of four, process600identifies a remaining group of tokens with the longest length that is equal to or less than four. For this example, the sentence “John ran too fast” has a length of four. Thus, process600fills training data800with the sentence.

Since training data800is filled up with groups of tokens, process600proceeds to generate another data structure with a length of ten to pack with remaining groups of tokens.FIG.9illustrates another training data900generated from the example set of correlated sequences of tokens illustrated inFIG.6according to some embodiments. In particular, training data900is the second data structure that process600generates based on the methodology described above. Process600packs training data900by identifying remaining groups of tokens having the longest length that is equal to or less than the defined length of the data structure. As illustrated inFIG.9, process600starts by packing training data900with the sentence “She did not feel well” since it is the remaining sentence that has the longest length that is less than the data structure defined length of ten. With a remaining, unused length of five, process600identifies a remaining group of tokens with the longest length that is equal to or less than five. In this example, the sentence “Jane went home” has a length of three and hence, process600fills training data900with this sentence. Finally, with a remaining, unused length of two, process600identifies a remaining group of tokens with the longest length that is equal to or less than two. In this case, the last remaining sentence “He fell” has a length of two so process600fills training data900with this sentence. Returning toFIG.6, process600then trains, at630, the sequence model using the set of training data. For example, training data800and900are used to train sequence module110.

FIG.10illustrates an architecture of output data processor1151according to some embodiments. As shown, output data processor115includes Data divider1005, sequence manager1010, classification manager1015, sequence loss manager1020, and classification loss manager1025. Data divider1005is responsible for dividing data into sequences of tokens and correlation data. As shown inFIG.10, data divider1005receives sequence output array1030as input. In response, data divider1005sends data in sequence output array1030associated with sequences of tokens to sequence manager1010and sends correlation data (e.g., data pertaining to sentence values) in sequence output array1030to classification manager1015.

Sequence manager1010is configured to predict a next sequence of tokens for a given input sequence of tokens. For example, sequence manager1010can receive from data divider1005vector representations of a next sequence of tokens. Next, sequence manager1010performs a set of projection functions on the vector representations to determine probabilities associated with corpus of tokens (e.g., a vocabulary of words). For each token in the next sequence of tokens, sequence manager1010selects the token having the highest probability as being the token predicted for the token in the next sequence of tokens. After predicting tokens for the next sequence of tokens, sequence manager1010sends the predicted next sequence of tokens to sequence loss manager1020.

Classification manager1015handles predictions of correlations between tokens. For instance, classification manager1015may receive correlation data (e.g., data associated with sentence values) from token divider1005. Classification manager1015may perform a set of classification functions on the correlation data to determine probabilities associated with correlations between tokens and/or groups of tokens in the sequence of tokens. Based on the probabilities, classification manager1015predicts whether tokens and/or groups of tokens in the sequence of tokens are correlated with each other. Once classification manager1015finishes predicting correlations between tokens, classification manager1015sends the predicted correlations to classification loss manager1025.

Sequence loss manager1020is responsible for determining sequence losses. For example, when sequence loss manager1020receives a predicted next sequence of tokens from sequence manager1010, sequence loss manager1020calculates a difference (e.g., an error) between the predicted next sequence of tokens and the actual next sequence of tokens (e.g., stored in label data). The calculated difference is depicted inFIG.10as sequence losses1035. Sequence manager1020may send token losses1035to sequence module110, which sequence module110uses to adjust its weights.

Classification loss manager1025is configured to determine correlation losses. For instance, upon receiving predicted correlations from classification manager1015, classification loss manager1025may calculate differences (e.g., errors) between the predicted correlations between tokens and the actual correlations between tokens (e.g., stored in label data). The calculated differences are depicted inFIG.10as classification losses1040. Classification loss manager1025can send classification losses1040to sequence module110, which sequence module110uses to adjust its weights.

The techniques describe above may be implemented in a wide range of computer systems configured to process neural networks.FIG.11depicts a simplified block diagram of an example computer system1100, which can be used to implement the techniques described in the foregoing disclosure. In some embodiments, computer system1100may be used to implement system100. As shown inFIG.11, computer system1100includes one or more processors1102that communicate with a number of peripheral devices via a bus subsystem1104. These peripheral devices may include a storage subsystem1106(e.g., comprising a memory subsystem1108and a file storage subsystem1110) and a network interface subsystem1116. Some computer systems may further include user interface input devices1112and/or user interface output devices1114.

Bus subsystem1104can provide a mechanism for letting the various components and subsystems of computer system1100communicate with each other as intended. Although bus subsystem1104is shown schematically as a single bus, alternative embodiments of the bus subsystem can utilize multiple busses.

Network interface subsystem1116can serve as an interface for communicating data between computer system1100and other computer systems or networks. Embodiments of network interface subsystem1116can include, e.g., Ethernet, a Wi-Fi and/or cellular adapter, a modem (telephone, satellite, cable, ISDN, etc.), digital subscriber line (DSL) units, and/or the like.

Storage subsystem1106includes a memory subsystem1108and a file/disk storage subsystem1110. Subsystems1108and1110as well as other memories described herein are examples of non-transitory computer-readable storage media that can store executable program code and/or data that provide the functionality of embodiments of the present disclosure.

Memory subsystem1108includes a number of memories including a main random access memory (RAM)1118for storage of instructions and data during program execution and a read-only memory (ROM)1120in which fixed instructions are stored. File storage subsystem1110can provide persistent (e.g., non-volatile) storage for program and data files, and can include a magnetic or solid-state hard disk drive, an optical drive along with associated removable media (e.g., CD-ROM, DVD, Blu-Ray, etc.), a removable flash memory-based drive or card, and/or other types of storage media known in the art.

It should be appreciated that computer system1100is illustrative and many other configurations having more or fewer components than system1100are possible.

FIG.12illustrates a neural network processing system according to some embodiments. In various embodiments, neural networks according to the present disclosure may be implemented and trained in a hardware environment comprising one or more neural network processors. A neural network processor may refer to various graphics processing units (GPU) (e.g., a GPU for processing neural networks produced by Nvidia Corp®), field programmable gate arrays (FPGA) (e.g., FPGAs for processing neural networks produced by Xilinx®), or a variety of application specific integrated circuits (ASICs) or neural network processors comprising hardware architectures optimized for neural network computations, for example. In this example environment, one or more servers1202, which may comprise architectures illustrated inFIG.11above, may be coupled to a plurality of controllers1210(1)-1210(M) over a communication network1201(e.g. switches, routers, etc.). Controllers1210(1)-1210(M) may also comprise architectures illustrated inFIG.11above. Each controller1210(1)-1210(M) may be coupled to one or more NN processors, such as processors1211(1)-1211(N) and1212(1)-1212(N), for example. NN processors1211(1)-1211(N) and1212(1)-1212(N) may include a variety of configurations of functional processing blocks and memory optimized for neural network processing, such as training or inference. The NN processors are optimized for neural network computations. Server1202may configure controllers1210with NN models as well as input data to the models, which may be loaded and executed by NN processors1211(1)-1211(N) and1212(1)-1212(N) in parallel, for example. Models may include layers and associated weights as described above, for example. NN processors may load the models and apply the inputs to produce output results. NN processors may also implement training algorithms described herein, for example.

Further Example Embodiments

In various embodiments, the present disclosure includes systems, methods, and apparatuses for packing tokens into input data used to train sequence models. The techniques described herein may be embodied in non-transitory machine-readable medium storing a program executable by a computer system, the program comprising sets of instructions for performing the techniques described herein. In some embodiments, a system includes a set of processing units and a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to perform the techniques described above. In some embodiments, the non-transitory machine-readable medium may be memory, for example, which may be coupled to one or more controllers or one or more artificial intelligence processors, for example.

The following techniques may be embodied alone or in different combinations and may further be embodied with other techniques described herein.

For example, in one embodiment, the present disclosure includes a system comprising a set of processing units and a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to receive a plurality of datasets for training a sequence model, each dataset in the plurality of datasets comprising a sequence of correlated tokens; generate a set of training data comprising a subset of a sequence of tokens from a first dataset in the plurality of datasets and a subset of a sequence of tokens from a second, different dataset in the plurality of datasets; and train the sequence model using the set of training data.

In one embodiment, generating the set of training data comprises determining, for the sequence of tokens of each dataset in the plurality of datasets, a set of groups of tokens in the sequence of tokens and determining, for each group of tokens in the set of groups of tokens of each sequence of tokens, a length of the group of tokens, wherein generating the set of training data is based on the lengths of the groups of tokens.

In one embodiment, the set of training data comprises a data structure having a defined length and generating the set of training data comprises identifying a group of tokens in the plurality of datasets having the longest length equal to or less than the defined length of the data structure; and packing the data structure with the identified group of tokens.

In one embodiment, generating the set of training data further comprises iteratively packing the data structure with remaining groups of tokens in the plurality of datasets having the longest length that is equal to or less than a remaining length in the data structure.

In one embodiment, the data structure is a first data structure, the set of training data further comprises a second data structure having the defined length, and generating the set of training data comprises identifying a remaining group of tokens in the plurality of datasets having the longest length equal to or less than the defined length of the data structure; and packing the second data structure with the identified group of tokens.

In one embodiment, generating the set of training data comprises adding label data to the set of training data indicating that the subset of the sequence of tokens from the first dataset and the subset of the sequence of tokens from the second, different dataset are not correlated.

In one embodiment, the sequence of correlated tokens in the first dataset is a first set of sentences from a first paragraph of text and the sequence of correlated tokens in the second dataset is a second set of sentences from a second paragraph of text.

In one embodiment, the present disclosure includes a system comprising a set of processing units and a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to receive a set of input data for training a sequence model, the input data comprising a sequence of tokens; group the sequence of tokens into a set of groups of tokens; generate a set of training data comprising the set of groups of tokens and copies of at least a portion of a group of tokens in the set of groups of tokens; and train the sequence model using the set of training data.

In one embodiment, the copies of at least the portion of the group of tokens are copies of at least the portion of a first group of tokens and the set of training data further comprises copies of at least a portion of a second group of tokens in the set of groups of tokens.

In one embodiment, generating the set of training data comprises generating a data structure having a defined length; packing a first row of the data structure with the first group of tokens and the copies of at least the portion of the first group of tokens until the length of the first row of the data structure is filled up with tokens from the first group of tokens; and packing the second row of the data structure with the second group of tokens and the copies of at least the portion of the second group of tokens until the length of the second row of the data structure is filled up with tokens from the second group of tokens.

In one embodiment, the instructions further cause the at least one processing unit to determine a first set of embeddings comprising an embedding for each token in the first group of tokens and the copies of at least the portion of the first group of tokens; determine a second set of embeddings comprising an embedding for each token in the second group of tokens and the copies of at least the portion of the second group of tokens; and add the first set of embeddings to the second set of embeddings.

In one embodiment, generating the set of training data comprises repeating the copies of at least the portion of the first group of tokens and the copies of at least the portion of the second group of tokens.

In one embodiment, generating the set of training data comprises generating a data structure having a defined length; and packing the data structure with the set of groups of tokens and the copies of the at least one portion of the group of tokens in the set of groups of tokens so that a total number of tokens packed into the data structure is equal to the defined length.