Systems for multiple named entity recognition

A machine learning model analyzes text describing an item to determine portions of the text that correspond to multiple characteristics of the item. A first set of embeddings that represent the text describing the item is determined. A second set of embeddings that represent text indicating the characteristics is determined. The second set of embeddings includes a token for each characteristic that is used to indicate tokens that are associated with a particular characteristic. The first set of embeddings and portions of the second set of embeddings for a particular characteristic are used to determine a set of interaction embeddings for each characteristic by determining an element-wise product. These interaction embeddings are analyzed to determine label predictions indicating text that is associated with each characteristic. Text for multiple characteristics may therefore be identified using a single pass rather than multiple passes.

BACKGROUND

Named Entity Recognition (NER) processes analyze text to determine portions of the text related to specific characteristics. This process is typically performed using sequence labeling operations, or in some cases using machine learning models, to match a portion of the text with a single characteristic. The process is repeated for each desired characteristic, such as by passing the same text through a machine learning system multiple times. Performing multiple passes in this manner may consume significant time and computational resources during training and during analysis of the text.

DETAILED DESCRIPTION

Computing devices may be used to analyze text and extract information from the text that corresponds to selected characteristics. This process is sometimes referred to as Named Entity Recognition (NER). For example, unstructured text may describe an item available for purchase using an online store. Certain portions of the text may indicate the brand of the item, an item category, a color of the item, a size of the item, and other characteristics of the item. An NER process may be performed to determine the specific portions of the text that indicate each desired characteristic. An NER process may be performed as a sequence labeling task, which may determine portions of the text that are associated with multiple characteristics. In other cases, an NER process may be performed using one or more machine learning models to determine a portion of the text that is likely to be associated with a selected single characteristic. To determine portions of the text associated with multiple characteristics, multiple passes associated with a machine learning model may be used. Based on the output determined using NER processes, data regarding an item may be stored or categorized based on various characteristics, associated with data or metadata indicating the determined characteristics of the item, and so forth.

Performance of existing NER processes to determine portions of text that correspond to multiple characteristics may consume significant time and computational resources. For example, each process to analyze the text and determine a portion of the text associated with a single characteristic may involve passing the text through a machine learning model. When determining text that corresponds to multiple characteristics, the same text may be passed through the machine learning system multiple times (e.g., one pass for each characteristic). The time and resources utilized by such a process may result in a system that is not readily scalable to accommodate a large number of characteristics. For example, in cases where significant quantities of information are analyzed, such as a catalog that includes text describing a large number of items available using an online store, determining a large number of characteristics for each item using NER processes may represent a significant investment of time and resources.

Described in this disclosure are techniques for analyzing text using a machine learning model, and extracting information regarding multiple characteristics in a single pass, resulting in faster performance of operations, use of smaller amounts of computational resources, and more efficient training of machine learning models. When text describing items available using an online store is analyzed, the characteristics to be extracted from the text for each item may be similar across multiple items. For example, when analyzing text describing clothing items, it may be desirable to determine portions of the text for multiple items that indicate an item brand, category, color, size, and so forth. Rather than analyzing the text describing an item multiple times, such as by passing the text through a machine learning model once for each characteristic, the characteristics to be determined may be combined into a single input that is provided to the machine learning model. The text describing the item may also be provided as an input to the machine learning model. In some implementations, the machine leaning model may include a Bidirectional Encoder Representations from Transformers (BERT) model, a Multilingual Bert (M-BERT) model, or other types of language representation models.

The machine learning model may be used to determine a first set of output embeddings based on the text describing the item. The embeddings may be indicative of words, characters, positions, segments, and so forth associated with various tokens in the text. The machine learning model may also be used to determine a second set of output embeddings based on the input text that indicates each of the characteristics. In some implementations, the input text may include a characteristic token associated with each characteristic. The characteristic token may include a random value, an arbitrary value, another type of known value, or any type of data that may be identified by the machine learning model, and may therefore be used to indicate different characteristics included within the input text. For example, a first characteristic token may precede one or more tokens associated with a first characteristic, which may be followed by a second characteristic token that precedes one or more tokens associated with a second characteristic. The machine learning model may be trained to identify the tokens associated with different characteristics based on the characteristic tokens included in the input text. Therefore, the machine learning model may determine particular portions of the second set of output embeddings that correspond to each characteristic.

Each portion of the second set of output embeddings that is associated with a characteristic may be used in combination with the first set of output embeddings (that represent the text describing the item) to determine a set of interaction embeddings for the particular characteristic. For example, an element-wise product may be determined based on the portion of the second set of output embeddings associated with a characteristic and based on the first set of output embeddings. In other implementations, another function, such as a layer product, may be used to determine interaction embeddings. A set of interaction embeddings may be determined for each characteristic identified in the second set of output embeddings. The interaction embeddings may therefore function as entity-specific representations for each token included in the text describing the item. The interaction embeddings may be used to generate label predictions for each characteristic, each label prediction being associated with one or more embeddings that represent a portion of the text describing the item. Therefore, labels indicating multiple characteristics may be determined by performing a single pass of the text describing the item using a machine learning model, rather than performing multiple passes with each pass extracting a single characteristic. As a result, text may be analyzed more efficiently and using a smaller quantity of computational resources. Additionally, machine learning models may be trained more quickly using this process. For example, based on the determined interaction embeddings and label predictions, a cross-entropy loss function may be determined. The machine learning model(s) may be trained to minimize the value of the loss function.

FIG.1is a diagram100depicting an implementation of a process for using a machine learning system102to determine multiple characteristics of an item based on text describing the item. WhileFIG.1describes an implementation in which text describing an item is analyzed to determine characteristics of the item presented in the text, in other implementations, other types of text may be processed using similar techniques.

In some cases, text describing an item may include unstructured text which may include alphanumeric characters such as uppercase letters, lowercase letters, numerals, punctuation, spacing and formatting characteristics, and so forth. The machine learning system102may be trained to process the item text104to determine portions of the item text104that are associated with particular characteristics. For example,FIG.1depicts item text104that describes a clothing item as “ExampleBrand Men's Sneaker, Midnight Blue, Size 12 Wide”. Continuing the example, certain portions of the item text104may indicate characteristics of the associated item, such as a brand name of “ExampleBrand”, a color of “Midnight Blue”, a category of “Men's”, and a size of “12 Wide”. In some cases, a single word of the item text104may correspond to a characteristic of the item, such as the brand name “ExampleBrand”. In other cases, multiple words of the item text104may correspond to a characteristic of the item, such as the color “Midnight Blue”. In still other cases, different portions of the item text104may correspond to the same characteristic of the item. For example, text such as “ExampleBrand Men's Blue Sneaker, Midnight Blue, Size 12 Wide” may include more than one portion having text corresponding to the color of the item (e.g., “Blue” and “Midnight Blue”). Additionally, in some cases, one or more portions of the item text104may not be associated with a characteristic that is to be determined using the machine learning system102. For example, the word “Sneaker” may not correspond to a category to be determined using the machine learning system102. Furthermore, in some cases, one or more characteristics that are to be determined using the machine learning system102may not be present in the item text104. For example, the machine learning system102may be configured to determine a characteristic relating to the material from which the item is constructed, and the item text104may lack words that indicate this information.

The machine learning system102may also access characteristics text106indicating one or more characteristics to be determined using the machine learning system102. The machine learning system102may be trained to determine relationships between the characteristics text106and the item text104that represent portions of the item text104associated with the characteristics indicated in the characteristics text106. For example, the machine learning system102may include a BERT machine learning model, an M-BERT machine learning model, or other type of system that is configured to parse language, such as by determining language tokens, embeddings, and so forth, and determine probabilities of relationships based on values indicated in the determined embeddings.FIG.1depicts example characteristics text106indicative of four categories: “Brand Name”, “Category”, “Color”, and “Size”, however any number and any type of characteristics may be determined.

Input text data108may be determined based on the item text104and the characteristics text106. In some cases, the input text data108may be determined by the machine learning system102or generated by one or more computing devices based on the item text104and the characteristics text106. In other implementations, the text input data108may be input directly to the machine learning system102, such as through user input, or by storing the input text data108in data storage accessible to the machine learning system102. The input text data108may include the item text104and the characteristics text106, as well as one or more additional tokens that may be processed by the machine learning system102. The tokens may include text, numerals, or any other type of value or data that may be identified by the machine learning system102. For example, the tokens included in the input text data108may include arbitrary values, random values, known values, values that have been selected by a user or automated system, values selected by the machine learning system102, and so forth.FIG.1depicts example input text data108that includes an initial token “[CLS]” that indicates the input text data108and a separator token “[September]” that indicates a separation between portions of the input text data108that include the item text104and portions of the input text data108that include the characteristics text106. The format of the initial token and separator token may be determined based on the type of machine learning system102that is used. In other implementations, one or more of the initial token or the separator token may be omitted.FIG.1also depicts the input text data108including multiple characteristic tokens “[CH]” that indicate text associated with specific characteristics. For example, a characteristic token may precede one or more words of the characteristics text106associated with a single characteristic, enabling the machine learning system102to determine that the words that follow the characteristic token are associated with a particular characteristic. In other cases, a characteristic token may follow one or more words associated with a particular characteristic, or may otherwise indicate portions of the input text data108.

The machine learning system102may process the input text data108to determine text embeddings110that represent portions of the input text data108associated with the item text104, and characteristics embeddings112that represent portions of the input text data108associated with the characteristics text106. For example, the machine learning system102may be trained to identify a separator token that separates the portions of the input text data108associated with the item text104from those associated with the characteristics text106and may generate separate sets of embeddings for each portion of the input text data108.

As shown inFIG.1, the text embeddings110may include an embedding representing the initial token (labeled “[CLS]”), as well as embeddings representing the portions of the input text data108that correspond to the item text104. For example, an embedding of the text embedding(s)110may represent a particular word of the input text data108, or in other cases a partial word. Each text embedding110may include values associated with the word or partial word represented by the text embedding110, the characters included in the text embedding110, the position of a word or character relative to other words or characters, and a segment of the input text data108associated with the text embedding110. For example, portions of the input data108that correspond to the item text104may be associated with a first segment value represented in the text embeddings110, while portions of the input text data108that correspond to the characteristics text106may be associated with a second segment value represented in the characteristics embeddings112.

The characteristics embeddings112may include embeddings representing the characteristic tokens (labeled “[CH]”), as well as embeddings representing the portions of the input text data108that correspond to the characteristics text106. For example, an embedding of the characteristics embedding(s)112may represent a particular word of the characteristics text106. Continuing the example, an embedding representing a first characteristic token “[CH1]” may be followed by characteristics embeddings112that represent words associated with a particular characteristic, such as the words “Brand” and Name”. Similarly, a second characteristic token “[CH2]” may be followed by a characteristics embedding112representing a word associated with a second characteristic, such as the word “Category”, a third characteristic token “[CH3]” may be followed by a characteristics embedding112representing a word associated with a third characteristic, such as the word “Color”, and a fourth characteristic token “[CH4]” may be followed by a characteristics embedding112representing a word associated with a fourth characteristic, such as the word “Size”. Each characteristics embedding112may include values associated with the word or partial word represented by the characteristics embedding112, the characters included in the characteristics embedding112, the position of a word or character relative to other words or characters, and a segment of the input text data108associated with the characteristics embedding112.

Based on the text embeddings110and the characteristics embeddings112, the machine learning system102may determine sets of interaction embeddings114that represent portions of text embeddings110specific to a particular subset of characteristics embeddings112. In some implementations, a set of interaction embeddings114may be determined by performing a relationship function120based on the text embeddings110and characteristics embeddings112. For example, the interaction embeddings114may be determined based on an element-wise product of the text embeddings110and a subset of characteristics embeddings112associated with a particular characteristic token. As another example, the interaction embeddings114may be determined based on a layer product. In some implementations, an element-wise product may be determined based on Equation 1 below:
Pi=T*ENTi  (Equation 1)

In Equation 1, Pi represents the Ith entity-specific context vector of a shape (n, dim). T is the context vector of the shape (n, dim), where n is the context length. ENTi is the entity vector of the shape (dim) for the Ith entity.

In some implementations, a layer product may be determined based on Equation 2 below:
Pi=W1(T)*W2(ENTi)  (Equation 2)

In Equation 2, Pi represents the Ith entity-specific context vector of a shape (n, dim). T is the context vector of the shape (n, dim), where n is the context length. ENTi is the entity vector of the shape (dim) for the Ith entity. W1 and W2 represent linear weight matrices.

The machine learning system102may determine a portion of the characteristics embeddings112based on a characteristic token included in the input text data108. Continuing the example, a characteristics embedding112associated with the characteristic token “[CH1]” may be used to determine other characteristics embeddings112“E1” and “E2” that are associated with a particular characteristic indicated in the input text data108. The machine learning system102may determine an element-wise product, layer product, or other function based on this subset of the characteristics embeddings112and the text embeddings110to determine a first set of interaction embeddings114(1) specific to a first characteristic. Similarly, a second portion of the characteristics embeddings112and the text embeddings110may be used to determine a second set of interaction embeddings114(2), a third portion the characteristics embeddings112and the text embeddings110may be used to determine a third set of interaction embeddings114(3), a fourth portion the characteristics embeddings112and the text embeddings110may be used to determine a fourth set of interaction embeddings114(4), and any number of additional subsets of characteristics embeddings112may similarly be used to determine corresponding additional sets of interaction embeddings114.

The machine learning system102may be configured to determine one or more sets of label predictions116based on the interaction embeddings114. For example, a set of label predictions116for a particular set of interaction embeddings114may indicate a probability that a particular embedding within the set of interaction embeddings114is associated with the characteristics embeddings112from which the interaction embeddings114were generated. Continuing the example, the machine learning system102may be trained to determine the probability that one or more values associated with a particular interaction embedding114are related to a particular characteristic represented by a characteristics embedding112. In some cases, a label prediction116may indicate a single embedding that is likely to correspond to a characteristic. In other cases, a label prediction116may indicate multiple embeddings, such as when multiple words within the item text104are related to a particular characteristic. In some cases, multiple portions of item text104may be related to a characteristic. For example, item text104may include multiple brand names or may include a single brand name at more than one location within the text. As such, whileFIG.1depicts label predictions116illustrative of a case in which each characteristic is referenced once within the item text104, in other implementations, a set of label predictions116may indicate the presence of text that corresponds to a characteristic at multiple locations within the item text104. Specifically,FIG.1depicts a first set of label predictions116(1) determined based on the first interaction embeddings114(1), which indicate a particular interaction embedding114(1) associated with a probability of being related to a first characteristic. For example, an embedding representing the item text104“ExampleBrand” may correspond to the characteristics text106of “Brand Name”. Similarly, a second set of label predictions116(2) may be determined based on the second interaction embeddings114(2), which indicate a particular interaction embedding114(2) with a probability of being related to a different characteristic. For example, an embedding representing the item text104“Men's” may correspond to the characteristics text106of “Category”. In a similar manner, label predictions116(3),116(4) may be determined for other sets of interaction embeddings114.

Output data118may be determined based on at least a subset of the label predictions116. For example, output data118may include text that is presented or stored for presentation. As another example, output data118may include data or metadata that is stored in association with item data, such as to cause item data to be stored based on particular characteristics, retrieved based on queries that indicate particular characteristics, and so forth. In some implementations, a loss function, such as a cross-entropy loss function or another type of loss function, may be determined based on one or more label predictions116and expected values for the label prediction(s)116, or in some cases based on one or more interaction embeddings114and expected values for the interaction embedding(s)114. For example, by minimizing a loss function, accuracy of the machine learning system102may be improved for future embeddings and predictions.

FIGS.2A-2Care diagrams200depicting performance of a process using a machine learning system102to determine characteristics of an item based on input text data108. As described with regard toFIG.1, input text data108that is used as an input for the machine learning system102may be determined based on item text104that includes text describing an item and characteristics text106that includes text indicating one or more characteristics of items to be determined using the machine learning system102. In addition to including tokens that represent the text included in the input text data108, the input text data108may include one or more functional tokens, such as tokens indicating the beginning of the input text data108, tokens separating the portion(s) of the input text data108associated with the item text104from those associated with the characteristics text106, and tokens that indicate portions of the input text data108that represent particular characteristics. In some implementations, the machine learning system102or another system or computing device may determine input embeddings based on the input text data108. For example, embeddings representing the input text data108may include data indicative of words, characters, positions, segments, and so forth associated with various tokens in the input text data108.

FIG.2Adepicts the input text data108including an initial token202that indicates the beginning of the input text data108. The initial token202may be identified by the machine learning system102and may cause the machine learning system102to process the input text data108to determine relationships between text tokens204that are separated by a separator token206included in the input text data108. For example, a first set of text tokens204(1) that represent text included in the item text104may follow the initial token202. In other implementations, use of an initial token202may be omitted. A separator token206may follow the first set of text tokens204(1) and may indicate a separation between the first set of text tokens204(1) determined based on the item text104and other text tokens204determined based on the characteristics text106.

In some implementations, the initial token202may also serve as a null token that may be indicated if the item text104does not include text that corresponds to one or more characteristics indicated in the characteristics text106. For example, if the machine learning system102does not determine that any embeddings that represent the first set of text tokens204(1) include text that corresponds to the characteristics represented by the other sets of text tokens204, label predictions116based on those embeddings may indicate the location of the initial token202as having a probability of being associated with a characteristic, while other embeddings may be determined to have a lower probability of such an association. In other implementations, a classifier or other type of module may be configured to determine the presence or absence of text that corresponds to a characteristic within the text embeddings110or input text data108, and output from the module indicating the absence of text that corresponds to a characteristic may cause label predictions116associated with that characteristic to be disregarded. As described with regard toFIG.1, the machine learning system102may determine text embeddings110based on the portion(s) of the input text data108that include the text token(s)204(1) that represent the item text104. For example, an embedding module216associated with the machine learning system102may determine text embeddings110for text tokens204(1) in the input text data108based on the words, partial words, characters, positions, and segments associated with each text token204(1). The text embeddings110may include an embedding associated with the initial token202, which may enable the initial token202to be indicated in a label prediction116in cases where the text embeddings110do not represent text associated with a particular category.

As shown inFIG.2A, one or more sets of text tokens204that represent the characteristics text106, and one or more characteristic tokens208, may follow the separator token206in the input text data108. In other implementations, portions of the input text data108that represent the characteristics text106may precede portions that represent the item text104. In still other implementations, the machine learning system102may be trained to identify separator tokens206or other tokens that denote whether a text token204is associated with item text104or characteristics text106, or the machine learning system102may be trained to determine segment data for embeddings that represent the text tokens204. In such a case, the text tokens204associated with item text104or characteristics text106may be included in the input text data108with any order or arrangement.

A first characteristic token208(1) is shown preceding a set of text tokens204(2) that represent a first characteristic (e.g., “Brand Name”). A second characteristic token208(2) is shown preceding a set of text tokens204(3) that represent a second characteristic (e.g., “Category”). A third characteristic token208(3) is shown preceding a set of text tokens204(4) that represent a third category (e.g., “Color”). A fourth characteristic token208(4) is shown preceding a set of text tokens204(5) that represent a fourth category (e.g., “Size”). The characteristic tokens208may include one or more random values, arbitrary values, other types of known values, or any other type of data that may be identified by the machine learning system102. Therefore, the characteristic tokens208may enable the machine learning system102to determine particular text tokens204that correspond to particular characteristics indicated in the characteristics text106. For example, the embedding module216may determine characteristics embeddings112based on portions of the input text data108associated with the characteristics text106, such as portions that follow the separator token206. The characteristics embedding(s)112may include embeddings associated with the characteristic tokens208, which may indicate that other embeddings that precede or follow the embeddings based on the characteristic tokens208may be associated with a particular characteristic indicated in the text that followed or preceded the characteristic token208.

As shown inFIG.2Band described with regard toFIG.1, the text embedding(s)110and one or more portions of the characteristics embedding(s)112may be used to determine one or more sets of interaction embeddings114that are based on particular portions of the characteristics embedding(s)112that represent particular characteristics. In some implementations, a set of interaction embeddings114may be determined based on an element-wise product or layer product of the text embeddings110and a subset of characteristics embeddings112associated with a particular characteristic token208. For example, based on the text embeddings110and a first portion of the characteristics embeddings112associated with a first characteristic token208(1) (“CH1”, followed by the embeddings “E1” and “E2”), a context module210associated with the machine learning system102may determine a first set of interaction embeddings114(1). Based on the text embeddings110and a second portion of the characteristics embeddings112associated with a second characteristic token208(2) (“CH2”, followed by the embedding “E3”), the context module210may determine a second set of interaction embeddings114(2). Based on the text embeddings110and a third portion of the characteristics embeddings112associated with a third characteristic token208(3) (“CH3”, followed by the embedding “E4”), the context module210may determine a third set of interaction embeddings114(3). Based on the text embeddings110and a fourth portion of the characteristics embeddings112associated with a fourth characteristic token208(4) (“CH4”, followed by the embedding “E5”), the context module210may determine a fourth set of interaction embeddings114(4).

As shown inFIG.2Cand described with regard toFIG.1, the interaction embeddings114for each characteristic may be used to determine a corresponding set of label predictions116. A set of label predictions116may indicate a probability that a particular embedding, or multiple embeddings, within a set of interaction embeddings114, is associated with the characteristics embeddings112from which the interaction embeddings114were generated. For example, the machine learning system102may be trained to determine the probability that one or more values associated with a particular interaction embedding114are related to a particular characteristic represented by a characteristics embedding112.

As shown inFIG.2C, a label module212associated with the machine learning system102may determine a first set of label predictions116(1) based on the first set of interaction embeddings114(1). The first set of label predictions116(1) may indicate that a particular embedding has a probability of being associated with the characteristic indicated in the portion of the characteristics embeddings112associated with the first characteristic token208(1). For example, the interaction embedding114(1) representing the item text104of “ExampleBrand” may correspond to the portion of the characteristics embeddings112that represent the characteristics text106“Brand Name”.

The label module212may determine a second set of label predictions116(2) based on the second interaction embeddings114(2). The second set of label predictions116(2) may indicate that a particular embedding has a probability of being associated with the characteristic indicated in the portion of the characteristics embeddings112associated with the second characteristic token208(2). For example, the interaction embedding114(2) representing the item text104of “Men's” may correspond to the portion of the characteristics embeddings112that represent the characteristics text106“Category”.

The label module212may determine a third set of label predictions116(3) based on the third interaction embeddings114(3). The third set of label predictions116(3) may indicate that multiple embeddings have a probability of being associated with the characteristic indicated in the portion of the characteristics embeddings112associated with the third characteristic token208(3). For example, the interaction embeddings114(3) representing the item text104of “Midnight Blue” may correspond to the portion of the characteristics embeddings112that represent the characteristics text106“Color”.

The label module212may determine a fourth set of label predictions116(4) based on the fourth interaction embeddings114(4). The fourth set of label predictions116(4) may indicate that multiple embeddings have a probability of being associated with the characteristic indicated in the portion of the characteristics embeddings112associated with the fourth characteristic token208(4). For example, the interaction embeddings114(4) representing the item text104of “12 Wide” may correspond to the portion of the characteristics embeddings112that represent the characteristics text106“Size”.

While each set of label predictions116shown inFIG.1depicts predictions in which a single embedding or two adjacent embeddings have a probability of being associated with a characteristic, any number of adjacent or non-adjacent embeddings may have a probability of being associated with a characteristic. For example, the item text104“ExampleBrand Men's Blue Sneaker, Midnight Blue, Size 12 Wide” may result in a set of label predictions116in which text representing a possible color of the item is present at two locations within the text.

An output module214associated with the machine learning system102may determine output data118based on at least a portion of the label predictions116. In some implementations, output data118may include text, audio data, or another type of data that may be presented or stored for presentation. In other cases, output data118may be stored in association with item data or used to modify item data associated with an item, such as to cause the item data to be associated with particular characteristics, able to be retrieved based on queries that indicate particular characteristics, and so forth. By performing the process described with regard toFIG.1andFIGS.2A-2C, label predictions116and output data118may be determined based on input text data108using a single pass associated with a machine learning system102. For example, the machine learning system102may determine item text104associated with multiple characteristics in a single pass, rather than using one pass for each characteristic to be determined.

FIG.3is a flow diagram300depicting an implementation of a method for using a single pass associated with a machine learning system102to determine portions of text describing an item that are associated with particular characteristics of the item. At302, text data that includes first text describing an item and second text indicative of item characteristics may be accessed. For example, input text data108may be determined based on item text104that describes an item, and characteristics text106that indicates one or more characteristics of items. The input text data108may therefore include first text based on the item text104, and second text based on the characteristics text106. As descried previously with regard toFIGS.1and2A-2C, input text data108may include text tokens204based on at least a portion of the item text104and characteristics text106, as well as other tokens, such as an initial token202, separator token206, and one or more characteristic tokens208. For example, the separator token206may be used to determine portions of the input text data108associated with the item text104and portions associated with the characteristics text106. The characteristic tokens208may be used to determine particular portions of the second text that are associated with particular characteristics.

At304, a machine learning model may be used to determine first output embeddings based on the first text. For example, as described with regard toFIG.1, a machine learning system102, such as a BERT or M-BERT system, may determine one or more text embeddings110based on a portion of input text data108that is associated with item text104. The portion of the input text data108associated with the item text104may be determined based at least in part on a separator token206. For example, portions of the input text data108that precede the separator token206may be associated with the item text104, while portions that follow the separator token206may be associated with the characteristics text106. In other implementations, portions of the input text data108associated with the item text104may follow the separator token206rather than precede the separator token206. A text embedding110may include one or more values associated with the word represented by the text embedding110, the character(s) included in the text embedding110, the position of a word or character relative to other words or characters, and a segment of the input text data108associated with the text embedding110. In some cases, the text embeddings110may also include an embedding associated with an initial token202included in the input text data108. For example, the initial token202may be indicated in a label prediction116if the machine learning system102determines that other text embeddings110are not associated with a particular characteristic.

At306, the machine learning model may be used to determine second output embeddings based on the second text. As described with regard toFIG.1, a machine learning system102may determine one or more characteristics embeddings112based on a portion of the input text data108that is associated with characteristics text106. The portion of the input text data108associated with the item text104may be determined based at least in part on a separator token206included in the input text data208. A characteristics embedding112may include one or more values associated with the word represented by the characteristics embedding112, the character(s) included in the characteristics embedding112, the position of a word or character relative to other words or characters, and a segment of the input text data108associated with the characteristics embedding112. The characteristics embeddings112may include embeddings that represent the characteristic tokens208included in the input text data108. For example, an embedding determined for a characteristic token208may be associated with embeddings associated with text that describes a particular characteristic. Continuing the example, text that follows or precedes a first characteristic token208may be associated with a first characteristic, while text that follows or precedes a second characteristic token208may be associated with a second characteristic. The machine learning system102may be trained to determine text tokens204associated with different characteristics based on the characteristic tokens208included in the input text data108. WhileFIG.3depicts block304prior to block306, these steps may be performed in any order. For example, the first output embeddings representing the first text may be determined before determining the second output embeddings, after determining the second output embeddings, or at least partially concurrently with determination of the second output embeddings.

At308, a portion of the second output embeddings that is associated with a particular characteristic may be determined. For example, a characteristics embedding112may be determined based on a characteristic token208, and a portion of the characteristics embeddings112may be determined based on the text tokens204that are associated with the characteristic token208. Therefore, based on the characteristic tokens208included in the input text data108, the machine learning system102may determine particular characteristics embeddings112that are associated with a characteristic.

At310, based on the first output embeddings and the portion of the second output embeddings determined at308, a set of interaction embeddings114associated with the particular characteristic may be determined. In some implementations, a set of interaction embeddings114may be determined based on a product of the text embeddings110and a subset of characteristics embeddings112associated with a particular characteristic token208, such as an element-wise product or layer product.

At312, the machine learning model and the interaction embeddings114determined at310may be used to determine a set of label predictions116that associate one or more of the interaction embeddings114with the particular characteristic. As described with regard toFIGS.1and2A-2C, the machine learning system102may be trained to determine label predictions116based on values indicated in the interaction embeddings114. For example, a set of label predictions116for a particular set of interaction embeddings114may indicate probabilities that particular embeddings within the set of interaction embeddings114are associated with the characteristics embeddings112from which the interaction embeddings114were determined. In some cases, a label prediction116may indicate a single embedding that has a probability to correspond to a characteristic. In other cases, a label prediction116may indicate multiple embeddings, such as when multiple words within item text104are related to a particular characteristic.

The process described in steps308,310, and312may be repeated multiple times, sequentially or concurrently, to determine portions of the characteristics embeddings112associated with a particular characteristic, determine interaction embeddings114based on the text embeddings110and that portion of the characteristics embeddings112, and determine label predictions116based on the interaction embeddings114. This process may therefore determine label predictions116for multiple characteristics indicated in the characteristics text106using a single pass of the machine learning system102.

At314, output data118may be determined based on each of the sets of label predictions116. In some cases, output data118may include text, audio data, or another type of data that may be presented or stored for presentation. In other cases, output data118may be stored in association with item data or used to modify item data, such as to cause the item data to be associated with particular characteristics, able to be retrieved based on queries that indicate particular characteristics, and so forth.

FIG.4is a block diagram400illustrating an implementation of a computing device402within the present disclosure. In some implementations, the computing device402may store or execute one or more machine learning models that are part of a machine learning system102. In other implementations, the computing device402may communicate with one or more other computing devices402thar are associated with a machine learning system102. In still other implementations, the computing device402may access a machine learning system102and may receive instructions from one or more other computing devices402for use with the machine learning system102. Additionally, whileFIG.4depicts a single block diagram400of a computing device402, any number and any type of computing devices402may be used to perform the functions described herein.

One or more power supplies404may be configured to provide electrical power suitable for operating the components of the computing device402. In some implementations, the power supply404may include a rechargeable battery, fuel cell, photovoltaic cell, power conditioning circuitry, and so forth.

The computing device402may include one or more hardware processor(s)406(processors) configured to execute one or more stored instructions. The processor(s)406may include one or more cores. One or more clock(s)408may provide information indicative of date, time, ticks, and so forth. For example, the processor(s)406may use data from the clock408to generate a timestamp, trigger a preprogrammed action, and so forth.

The computing device402may include one or more communication interfaces410, such as input/output (I/O) interfaces412, network interfaces414, and so forth. The communication interfaces410may enable the computing device402, or components of the computing device402, to communicate with other computing devices402or components of the other computing devices402. The I/O interfaces412may include interfaces such as Inter-Integrated Circuit (I2C), Serial Peripheral Interface bus (SPI), Universal Serial Bus (USB) as promulgated by the USB Implementers Forum, RS-232, and so forth.

The I/O interface(s)412may couple to one or more I/O devices416. The I/O devices416may include any manner of input devices or output devices associated with the computing device402. For example, I/O devices416may include touch sensors, displays, touch sensors integrated with displays (e.g., touchscreen displays), keyboards, mouse devices, microphones, image sensors, cameras, scanners, speakers or other types of audio output devices, haptic devices, printers, and so forth. In some implementations, the I/O devices416may be physically incorporated with the computing device402. In other implementations, I/O devices416may be externally placed.

The network interfaces414may be configured to provide communications between the computing device402and other devices, such as the I/O devices416, routers, access points, and so forth. The network interfaces414may include devices configured to couple to one or more networks including local area networks (LANs), wireless LANs (WLANs), wide area networks (WANs), wireless WANs, and so forth. For example, the network interfaces414may include devices compatible with Ethernet, Wi-Fi, Bluetooth, ZigBee, Z-Wave, 4G, 5G, LTE, and so forth.

The computing device402may include one or more buses or other internal communications hardware or software that allows for the transfer of data between the various modules and components of the computing device402.

As shown inFIG.4, the computing device402may include one or more memories418. The memory418may include one or more computer-readable storage media (CRSM). The CRSM may be any one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, and so forth. The memory418may provide storage of computer-readable instructions, data structures, program modules, and other data for the operation of the computing device402. A few example modules are shown stored in the memory418, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SoC).

The memory418may include one or more operating system (OS) modules420. The OS module420may be configured to manage hardware resource devices such as the I/O interfaces412, the network interfaces414, the I/O devices416, and to provide various services to applications or modules executing on the processors406. The OS module420may implement a variant of the FreeBSD operating system as promulgated by the FreeBSD Project; UNIX or a UNIX-like operating system; a variation of the Linux operating system as promulgated by Linus Torvalds; the Windows operating system from Microsoft Corporation of Redmond, Washington, USA; or other operating systems.

One or more data stores422and one or more of the following modules may also be associated with the memory418. The modules may be executed as foreground applications, background tasks, daemons, and so forth. The data store(s)422may use a flat file, database, linked list, tree, executable code, script, or other data structure to store information. In some implementations, the data store(s)422or a portion of the data store(s)422may be distributed across one or more other devices including other computing devices402, network attached storage devices, and so forth.

A communication module424may be configured to establish communications with one or more other computing devices402. Communications may be authenticated, encrypted, and so forth.

The memory418may also store an interface module426. An interface module426may receive input from one or more input devices or other computing devices402, such as item text104, characteristics text106, or input text data108provided via user input, or input from a service, application, client, automated system, or other computing device402. For example, the machine learning system102may determine input text data108based on other text, such as item text104or characteristics text106received from other sources. In other cases, the machine learning system102may receive input text data108as an input and may determine embeddings and label predictions116based on the received input text data108. In other implementations, one or more embeddings, such as text embeddings110or characteristics embeddings112may be determined by one or more other computing devices402, and the machine learning system102may receive these embeddings as inputs and may determine interaction embeddings114and label predictions116based on the received embeddings. The interface module426may also present output, such as output data118that may include text or audio data, using one or more interfaces. In other cases, output data118may be provided to other computing devices402, or other portions of the computing device402, such as for storage in association with item data, retrieval in response to queries, and so forth.

The memory418may additionally store the machine learning module102. The machine learning module102may be trained to determine embeddings based on accessed text data, embeddings (such as interaction embeddings114) based on other embeddings, label predictions116based on embeddings, and output data118based on label predictions116. In other cases, output data118may be determined using other computing devices402or other portions of the computing device402.

The memory418may store a training module428, which may be used to receive, process, and generate training data for use with the machine learning system102. In some implementations, the machine learning module102may be trained based in part on a loss function. For example, a cross-entropy loss function may be determined based on one or more interaction embeddings114or label predictions116, and expected values for the interaction embedding(s)114or label prediction(s)116. For example, by minimizing a loss function, accuracy of the machine learning system102may be improved for future embeddings and predictions. Training data may also enable the machine learning system102to identify and process functional tokens, such as initial tokens202indicative of input text data108, separator tokens206that indicate portions of input text data108associated with item text104or characteristics text106, and characteristic tokens208that may indicate particular text tokens204associated with particular characteristics.

Other modules430may also be present in the memory418. For example, other modules430may include permission or authorization modules to enable users to access and modify data associated with the computing device402. Other modules430may also include encryption modules to encrypt and decrypt communications between computing devices402, authentication modules to authenticate communications sent or received by computing devices402, and so forth.

Other data432within the data store(s)422may include configurations, settings, preferences, and default values associated with computing devices402. Other data432may also include encryption keys and schema, access credentials, and so forth. Other data432may further include training data associated with the machine learning system102, algorithms, functions, threshold data, and models used in association with the machine learning system102, and so forth.

In different implementations, different computing devices402may have different capabilities or capacities. For example, servers that store or access machine learning systems102may have greater processing capabilities or data storage capacity than personal computing devices used to provide data to or receive data from such servers or other computing devices402.