Neural models for keyphrase extraction

A keyphrase extraction service implements techniques for determining a set of keyphrases associated with set of words. A word is selected from the set of words and a neural model is used to determine a label for the word based on features of the word and labels corresponding to other words of the set of words. The set of keyphrases is determined from the labels associated with the set of words.

BACKGROUND

The use of hosted computing services and storage has proliferated in recent years where large-scale networks of computer systems execute a variety of applications and services. This variety of applications and services that are used for various purposes can generate large amounts of data that, in some instances, can be in the form of documents of various lengths and in various languages. In some cases, the volume of data available creates challenges for those who wish to characterize such documents to aid in processing the data contained in the document. Determining the salient features of the documents, for example, can be a challenging problem often addressed through significant compute resource usage and/or human intervention.

DETAILED DESCRIPTION

In one embodiment, keyphrase extraction is used to identify phrases in a document or other sequence of words that are representative of the content of the document or sequence of words. In one embodiment, keyphrase extraction is used to generate metadata associated with a document that can be used to summarize the document, can be used to retrieve information from the document, or can be used to answer questions about the content of the document. Typical approaches to keyphrase extraction focus on specific datasets, specific document types, and/or specific languages. In the preceding and following description, techniques for keyphrase extraction that are independent of the dataset, independent of the document type, and independent of language are described in relation to respective embodiments.

A keyphrase extraction service, in one embodiment, analyzes a sequence of words (e.g., from a document) using a stacked neural network model to learn and select relevant keyphrases from the document. The keyphrase extraction service first selects words from the document and, in one embodiment, the keyphrase extraction service selects the individual words from the document in sequence (e.g., in the order that the words are in the document). In one embodiment, the keyphrase extraction service selects the words from the document using an index to determine the order of the words in the document and selects the words from the document in reverse order. In one embodiment, the keyphrase extraction service selects the words from the document in random order and determines a sequence for the words using data and/or metadata associated with the document.

In one embodiment, the keyphrase extraction service analyzes the words to determine features of words in a document, which include, but are not limited to, character-level features and word-level features of the words as described herein. In one embodiment, the keyphrase extraction service analyzes the characters of the selected word. This analysis of the characters (i.e., character analysis) encodes the character-level features of the word (e.g., features of the word corresponding to each character in the word) using a neural model. In one embodiment, the keyphrase extraction service encodes character-level features using a bidirectional long short-term memory (“LSTM”) model. A bidirectional LSTM model is a neural model that includes a forward LSTM model and a backward LSTM model, both described herein. A bidirectional LSTM model (also referred to herein as a “bidirectional LSTM”) combines the results of a forward LSTM (e.g., an LSTM that iterates on the characters of the word from the beginning to the end of the word) with the results of a backward LSTM (e.g., an LSTM that iterates on the characters of the word from the end to the beginning) to encode the character-level features of the word.

In one embodiment, the keyphrase extraction service encodes character-level features using a convolutional neural network (“CNN”) model. A CNN model (also referred to as a “CNN”) is a neural model that includes multiple layers and links between elements in a layer as well as links between elements in different layers. A CNN locally combines the analysis of individual characters of a word with the analysis of nearby characters in the word (e.g., the previous and next characters) to encode the character-level features of the word.

The keyphrase extraction service, one embodiment, then combines the character-level features of the word with other features of the word to generate the word-level input features of the word. In one embodiment, the keyphrase extraction service combines the character-level features of the word with previously generated categorizations of the word taken from a word corpus or dictionary. In one embodiment, a word corpus is a structured set of text data that is used to inform statistical analysis on text data. In one embodiment, a word corpus is a single language word corpus. In one embodiment, a word corpus includes text data in multiple languages. Examples of word corpora include, but are not limited to, the Googlem Books NGram Corpus, the American National Corpus, and the Corpus of Contemporary American English (COCA). Such categorizations of the word denote parts of speech of the word (e.g., noun, verb, pronoun, adverb, etc.), common misspellings of the word, common associated words (e.g., other words that frequently appear with the word), common grammar errors associated with the word, synonyms of the word, antonyms of the word, and other such categorizations. In one embodiment, these categorizations are language and/or dialect dependent. In one embodiment, these categorizations are language and/or dialect independent.

In one embodiment, the keyphrase extraction service next analyzes the words of the document. This analysis of the words (i.e., word analysis) encodes the word-level features of the word (e.g., features of the word) from the word-level input features of the word using a neural model. In one embodiment, the keyphrase extraction service encodes word-level features of the word from the word-level input features of the word using a bidirectional LSTM. The bidirectional LSTM combines the results of a forward LSTM (e.g., an LSTM that iterates on the words from the beginning to the end of the document) with the results of a backward LSTM (e.g., an LSTM that iterates on the words from the end to the beginning of the document) to encode the word-level features of the word. In one embodiment, the keyphrase extraction service encodes word-level features of the document using a CNN. The CNN combines the analysis of individual words of the document with the analysis of nearby words of the document (e.g., the previous and next words) to encode the word-level features of the word.

In one embodiment, the keyphrase extraction service generates the tags (also referred to herein as “tag labels” and “labels”) for the document. The tags for the document are generated by decoding the encoded word-level features of the document using a forward LSTM (e.g., an LSTM that iterates on the words from the beginning to the end of the document) where each tag and each word is used to generate succeeding tags. In one embodiment, the first tag is a specialized tag that indicates the start of the LSTM processing. In one embodiment, the first tag is based at least in part on a particular sequencing task (e.g., keyphrase extraction) associated with the LSTM.

In one embodiment, the keyphrase extraction service performs the keyphrase extraction by first using a bidirectional LSTM to encode the character-level features of the word, by next using a bidirectional LSTM to encode the word-level features of the word based at least in part on the encoded character-level features, and by next using an LSTM to decode the word-level features and to generate the tags for the word. In one embodiment, the keyphrase extraction service performs the keyphrase extraction by first using a CNN to encode the character-level features of the word, by next using a CNN to encode the word-level features of the word based at least in part on the encoded character-level features, and by next using an LSTM to decode the word-level features and to generate the tags for the word. In one embodiment, the keyphrase extraction service performs the keyphrase extraction by first using a bidirectional LSTM to encode the character-level features of the word, by next using a CNN to encode the word-level features of the word based at least in part on the encoded character-level features, and by next using an LSTM to decode the word-level features and to generate the tags for the word. In one embodiment, the keyphrase extraction service performs the keyphrase extraction by first using a CNN to encode the character-level features of the word, by next using a bidirectional LSTM to encode the word-level features of the word based at least in part on the encoded character-level features, and by next using an LSTM to decode the word-level features and to generate the tags for the word. In one embodiment, the keyphrase extraction service uses other neural models to encode the character-level features of the word, to encode the word-level features of the word, and/or to decode the word-level features and to generate the tags for the word.

In one embodiment, the keyphrase extraction service performs multiple sequencing tasks on the word of the document using the techniques described herein including, but not limited to, keyphrase extraction, part of speech tagging, position tagging, etc. In one embodiment, the keyphrase extraction service performs these multiple sequencing tasks in parallel (e.g., simultaneously). In one embodiment, the keyphrase extraction service performs these multiple sequencing tasks individually (e.g., not simultaneously). In one embodiment, the results of one sequencing task are used as inputs to another sequencing task (e.g., the results of a position tagging task are used as an input to a keyphrase extraction task).

FIG. 1illustrates a system100where keyphrases are identified using a keyphrase extraction service in accordance with one embodiment. In the system100illustrated inFIG. 1, a keyphrase extraction service104operating within an environment of a computing resource service provider102receives a document106for keyphrase analysis. In the system100illustrated inFIG. 1, the document106is provided by a service108operating within the computing resource service provider102environment as described herein. In one embodiment, the document106is provided by a user, entity, or service operating outside of the computing resource service provider102environment. In one embodiment, the keyphrase extraction service104receives the document106and uses a keyphrase extraction model110to identify keyphrases112in the document106. In the system100illustrated inFIG. 1, the keyphrase extraction service104provides the keyphrases112to the service108operating within the computing resource service provider102environment.

In one embodiment, the computing resource service provider102provides access to one or more host machines hosted by the computing resource service provider102. In one embodiment, the computing resource service provider102provides access to one or more services in an environment therein. In one embodiment, the one or more services provided by the computing resource service provider102are implemented as and/or utilize one or more virtual machine instances that are operating on host machines operating within the computing resource service provider102environment. In one embodiment, the computing resource service provider102provides a variety of services to users and/or customers of the computing resource service provider102such as the service108. In one embodiment, the users and/or customers of the computing resource service provider102communicate with the computing resource service provider102via an interface such as a web services interface. In one embodiment, each of the services operating in an environment of the computing resource service provider102(e.g., the keyphrase extraction service104and/or the service108) has its own interface and, generally, subsets of the services can have corresponding interfaces in addition to or as an alternative to the single interface.

In one embodiment, a user interacts with services of the computing resource service provider (via appropriately configured and authenticated API requests) using a client computing device to provision and operate services that are instantiated on physical computing devices hosted and operated by the computing resource service provider102as described herein. These services are configured to manage resources (e.g., storage, processors, memory, network, etc.) associated with the services. These resources are used for various purposes, such as to operate as servers supporting a website, to operate business applications or, generally, to serve as computing power for the customer. Other applications of the services can support database applications, electronic commerce applications, business applications, and/or other such applications.

In one embodiment, the keyphrase extraction service104is a service provided by the computing resource service provider102to analyze text (e.g., the document106) and identify keyphrases (e.g., the keyphrases112) of the text using the techniques described herein. In one embodiment, the keyphrase extraction service104is a collection of computing resources that operate collectively to analyze text and identify keyphrases within the computing resource service provider102environment. These computing resources are configured to process requests to analyze text and identify keyphrases within the computing resource service provider102environment and include at least one of: computer systems (the computer systems including processors and memory), networks, storage devices, executable code, services, processes, modules, or applications. In one embodiment, the computing resources configured to analyze text and identify keyphrases within the computing resource service provider102environment include virtual systems that are implemented on shared hardware hosted by a computing resource service provider such as the computing resource service provider102. In one embodiment, the keyphrase extraction service104is implemented as a single system. In one embodiment, the keyphrase extraction service104is implemented as a distributed system, with a plurality of instances operating collectively to analyze text and identify keyphrases within the computing resource service provider102environment. In one embodiment, the keyphrase extraction service104operates using computing resources (e.g., databases and/or virtual machine instances) that enable the keyphrase extraction service104to process requests to analyze text and identify keyphrases within the computing resource service provider102environment.

In one embodiment, the keyphrase extraction service104maintains data and/or metadata such that, when a request to analyze text and identify keyphrases of a document within the computing resource service provider102environment is received, the data and/or metadata is located, processed, and provided (or a streaming data object is initiated) for use in processing the request. In one embodiment, the data and/or metadata maintained by the keyphrase extraction service104is organized as data objects. In one embodiment, these data objects have arbitrary sizes. In one embodiment, these data objects have constraints on size or durability. Thus, the keyphrase extraction service104is configured to manage numerous data objects of varying sizes. In one embodiment, the keyphrase extraction service104stores the data objects in storage resources controlled by the keyphrase extraction service104. In one embodiment, the keyphrase extraction service104stores the data objects using resources controlled by some other service such as a data storage service. In one embodiment, the keyphrase extraction service104stores the data objects using a combination of storage locations. In one embodiment, the keyphrase extraction service104operates as a key value store that associates the data objects with identifiers of the data objects that are used to retrieve or perform other operations in connection with the data objects. In one embodiment, the keyphrase extraction service104generates metadata associated with the data objects and causes such metadata to process requests to migrate data processing systems to the computing resource service provider102environment.

In one embodiment, the service108is a service that performs a variety of functions within the computing resource service provider102environment. The variety of functions performed by the service108includes, but is not limited to, data warehousing functions or applications, data processing functions or applications, data analysis functions or applications, data storage functions or applications, data security functions or applications, and data management functions or applications. In one embodiment, the service108is a service that performs this variety of functions and is hosted outside of the computing resource service provider102environment. In one embodiment, the service108is hosted in an environment provided by a customer of the computing resource service provider102. In one embodiment, the service108is hosted in an environment provided by a third party (e.g., neither the customer of the computing resource service provider102nor the computing resource service provider102).

In one embodiment, the document106is a text document such as a book or a research paper with a large number of words (e.g., in excess of a thousand words). In one embodiment, the document106is a shorter document such as an advertisement, a blog post, a product review, or a user comment with a smaller number of words (e.g., between fifty and a thousand words). In one embodiment, the document106is a very short document such as a log entry or a “Tweet” with less than fifty words. In one embodiment, the document106contains a mixture of text and non-text data. In one embodiment, the document106contains text that is in a single language and/or in a single dialect. In one embodiment, the document106contains text that is in multiple languages and/or dialects. In one embodiment, the document106contains text that is structured according to a defined structure (e.g., computer source code). In one embodiment, the document106contains text that is unstructured. In one embodiment, the document106is provided to the keyphrase extraction service104using a link to a data storage service such as the data storage services described herein.

FIG. 2illustrates a system200where keyphrases are identified and stored using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In the system200illustrated inFIG. 2, a keyphrase extraction service210operating within an environment of a computing resource service provider212receives a document222for keyphrase analysis that is provided by a service220operating within the computing resource service provider212environment as described above in connection withFIG. 1. In one embodiment, the keyphrase extraction service210receives the document222and uses a keyphrase extraction model218to identify keyphrases224in the document222. In the system200illustrated inFIG. 2, the keyphrase extraction service210provides the keyphrases224to the service220operating within the computing resource service provider212environment also as described above in connection withFIG. 1.

In the example illustrated inFIG. 2, the keyphrase extraction service210operating within the environment of a computing resource service provider212receives a document214for keyphrase analysis that is provided by a user202associated with the computing resource service provider212using a client computing device204that connects206to the keyphrase extraction service210of the computing resource service provider212via a network208. In one embodiment, the keyphrase extraction service210receives the document214and uses a keyphrase extraction model218to identify keyphrases216in the document214. In the system200illustrated inFIG. 2, the keyphrase extraction service210provides the keyphrases216to the client computing device204of the user202using the network208.

In the system200illustrated inFIG. 2, the user202(also referred to herein as a “customer”) is a user associated with the computing resource service provider212that begins the process of analyzing text and identifying keyphrases of a document by issuing a command to begin the analysis of the text from the client computing device204. In one embodiment, the command to begin the analysis of the text is generated by the user202of the computing resource service provider212who uses the client computing device204to connect to a variety of services provided by the computing resource service provider212as described herein. In one embodiment, the user202uses the client computing device204to connect to the computing resource service provider212over a network208such as those networks described herein. As described herein, a computing resource service provider212provides a distributed, virtualized, and/or datacenter environment within which one or more applications, processes, services, virtual machines, and/or other such computer system entities are executed. In one embodiment, the user202is a process running on one or more remote computer systems controlled by a customer of the computing resource service provider212.

In one embodiment, commands to the computing resource service provider212to analyze text and identify keywords originate from an outside computer system. In one embodiment, commands to the computing resource service provider212originate from within the computing resource service provider212environment. In one embodiment, the commands to connect to the computing resource service provider212are sent to the computing resource service provider212and/or to other services operating in the environment therein, without the direct intervention of the user202(i.e., commands to connect to the computing resource service provider212are generated automatically in response to one or more events). In one embodiment, the user202connects to the computing resource service provider212environment using a privileged user account associated with the customer of the computing resource service provider212. In one embodiment, the user202uses a privileged user account associated with and/or provided by the computing resource service provider212to connect to the computing resource service provider212environment.

In the system200illustrated inFIG. 2, the keyphrases216and/or the keyphrases224are provided226to a data storage service228and stored in a keyphrase repository230. In one embodiment, the keyphrases216and/or the keyphrases224are provided226to the data storage service228and stored in the keyphrase repository230, in addition to being provided to the service220and/or the client computing device204of the user202. In one embodiment, the keyphrases216and/or the keyphrases224are provided226to the data storage service228and stored in the keyphrase repository230as an alternative to being provided to the service220and/or the client computing device204of the user202(e.g., the keyphrases216and/or the keyphrases224are stored and/or provided according to a system configuration of the keyphrase extraction service210). In one embodiment, the keyphrases provided226to the data storage service228and stored in the keyphrase repository230include additional data and/or metadata linking the keyphrases to the document (e.g., linking the keyphrases216to the document214) that is stored in the keyphrase repository230.

In one embodiment, the data storage service228is a service provided by the computing resource service provider212to store data associated with analyzing text and identifying keyphrases of the text using the techniques described herein. In one embodiment, the data storage service228is a collection of computing resources that operate collectively to analyze text and identify keyphrases within the computing resource service provider212environment. These computing resources are configured to store data related to requests to analyze text and identify keyphrases within the computing resource service provider212environment and include at least one of: computer systems (the computer systems including processors and memory), networks, storage devices, executable code, services, processes, modules, or applications. In one embodiment, the computing resources configured to store data related to requests to analyze text and identify keyphrases within the computing resource service provider212environment include virtual systems that are implemented on shared hardware hosted by a computing resource service provider such as the computing resource service provider212. In one embodiment, the data storage service228is implemented as a single system. In one embodiment, the data storage service228is implemented as a distributed system, with a plurality of instances operating collectively to store data related to requests to analyze text and identify keyphrases within the computing resource service provider212environment. In one embodiment, the data storage service228operates using computing resources (e.g., databases and/or virtual machine instances) that enable the data storage service228to process requests to store data related to requests to analyze text and identify keyphrases within the computing resource service provider212environment. In one embodiment, the data related to requests to analyze text and identify keyphrases within the computing resource service provider212environment includes the requests, the text, the keyphrases, parameters used by the keyphrase extraction model218to analyze the text, metadata associated with the request to analyze the text, or other such data.

In one embodiment, the data storage service228maintains data and/or metadata such that, when a request to analyze text and identify keyphrases of a document within the computing resource service provider212environment is received, the data and/or metadata is located, processed, and provided (or a streaming data object is initiated) for use in processing the request. In one embodiment, the data and/or metadata maintained by the data storage service228is organized as data objects. In one embodiment, these data objects have arbitrary sizes. In one embodiment, these data objects have constraints on size or durability. Thus, the data storage service228is configured to manage numerous data objects of varying sizes. In one embodiment, the data storage service228stores the data objects in storage resources controlled by the data storage service228(e.g., the keyphrase repository230). In one embodiment, the data storage service228stores the data objects using resources controlled by some other service such as a database service. In one embodiment, the data storage service228stores the data objects using a combination of storage locations. In one embodiment, the data storage service228operates as a key value store that associates the data objects with identifiers of the data objects that are used to retrieve or perform other operations in connection with the data objects. In one embodiment, the data storage service228generates metadata associated with the data objects and causes such metadata to process requests to migrate data processing systems to the computing resource service provider212environment.

FIG. 3illustrates a process300for identifying keyphrases using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In one embodiment, a keyphrase extraction service such as the keyphrase extraction service104described in connection withFIG. 1performs the process300described in connection withFIG. 3. In one embodiment, the keyphrase extraction service receives302a sequence of words and selects304the first/next word. In one embodiment, the keyphrase extraction service detects306the features of the selected word using a first neural model. In one embodiment, the first neural model is a bidirectional LSTM. In one embodiment, the first neural model is a CNN. In one embodiment, the keyphrase extraction service associates308the detected features with the word. The keyphrase extraction service then determines310if there is a next word in the sequence of words to select.

In one embodiment, if the keyphrase extraction service determines310that there is a next word in the sequence of words to select, the keyphrase extraction service selects304the next word and detects306the features of that word. In one embodiment, if the keyphrase extraction service determines310that there is not a next word in the sequence of words to select, the keyphrase extraction service starts again at the beginning of the sequence and selects312the first/next word. In one embodiment, the keyphrase extraction service selects314the first/next feature of the selected word and labels316the word to produce one or more labels of features of words using a second neural network model based at least in part on the selected feature. In one embodiment, the second neural model is an LSTM. In one embodiment, the labels of features of words of the document include labels associated with keyphrase extends such as beginning of keyphrase (e.g., the first word of a keyphrase), inside keyphrase (e.g., a word inside a keyphrase that is not the beginning of keyphrase or end of keyphrase), end of keyphrase (e.g., the last word of a keyphrase), singleton keyphrase (e.g., a keyphrase that is a single word), or outside of keyphrase (e.g., a word that is not part of a keyphrase). The keyphrase extraction service next determines318if there is a next feature to select.

In one embodiment, if the keyphrase extraction service determines318that there is a next feature to select, the keyphrase extraction service selects314the next feature. In one embodiment, if the keyphrase extraction service determines318that there is not a next feature to select, the keyphrase extraction service determines320if there is a next word to select. In one embodiment, if the keyphrase extraction service determines320that there is a next word to select, the keyphrase extraction service selects312the next word. In one embodiment, if the keyphrase extraction service determines320that there is not a next word to select, the keyphrase extraction service extracts322the keyphrases for the sequence of words using the labeled words.

FIG. 4illustrates a diagram400of a long short-term memory model usable to identify keyphrases using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. The diagram400shows the equations402of a forward LSTM, which is a LSTM that processes a set of data from the start of the data to the end of the data. Given a sequence of data x1, x2, . . . , xn, an LSTM iteratively computes (or encodes) the hidden state htat step t using the equations illustrated.

In the diagram400, equation404is “it=σ(W(i)xt+U(i)ht−1+b(i))” which computes the input gate for the LSTM that represents a parameter for the LSTM related to acquiring new information.

In the equations402of the forward LSTM, cis a sigmoid activation function, tanh is a hyperbolic tangent activation function, ⊙ is an element-wise product operator, and W, U, and b are learnable parameters of the LSTM. Also in the equations402of the forward LSTM, xtis the element being analyzed (e.g., the character or word) and ht−1is the hidden state of the previous element (e.g., the hidden state of element xt−1). In one embodiment, the previous element is the previous letter in the word, as described herein. In one embodiment, the previous element is the previous word in the sequence of words, as described herein.

In the diagram400, equation406is “ot=σ(W(0)xt+U(0)ht−1+b(0))” which computes the output gate for the LSTM that represents a parameter for the LSTM related to outputting new information.

In the diagram400, equation408is “ft=σ(W(f)xt+U(f)ht−1+b(f))” which computes the forget gate for the LSTM that represents a parameter for the LSTM related to forgetting (e.g., discarding) previously acquired information.

In the diagram400, equation410is “gt=tanh (W(g)xt+U(g)ht−1+b(g))” which computes the pre-activation state for the LSTM that represents a parameter usable for computing the hidden state, described below.

In the diagram400, equation412is “ct=ft⊙ct−1+it⊙gt” which computes the cell state for the LSTM. Equation412is based on the results of equation404, equation408, and equation410, and is also based on the previous result of equation412.

In the diagram400, equation414is “ht=ot⊙ tanh (ct)” which computes the hidden state for the LSTM. Equation414is based on the result of equation406and the result of equation412. The hidden state of the LSTM is the encoded result for the LSTM for the particular data type.

As described above, the diagram400shows the equations402of a forward LSTM, which is a LSTM that processes a set of data from the start of the data to the end of the data. In various techniques described herein, both forward LSTM models and backward LSTM models (e.g., a LSTM that processes a set of data from the end of the data to the beginning of the data) are used. The equations for a backward LSTM are “it=σ(W(i)xt+U(i)ht+1+b(i))” (corresponding to equation404), “ot=σ(W(0)xt+U(0)ht+1+b(0))” (corresponding to equation406), “ft=σ(W(f)xt+U(f)ht+1+b(f)” (corresponding to equation408), and “gt=tanh(W(g)xt+U(g)ht+1+b(g))” (corresponding to equation410). Equations “ct=ft⊙ct−1+it⊙gt” and “ht=ot⊙ tanh(ct)” (e.g., equation412and equation414) are the same in both the forward LSTM and the backward LSTM.

FIG. 5illustrates a process500for selecting a model usable to identify keyphrases using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In one embodiment, a keyphrase extraction service such as the keyphrase extraction service104described in connection withFIG. 1performs the process500described in connection withFIG. 5. In one embodiment, the keyphrase extraction service receives502a sequence of words.

In one embodiment, the keyphrase extraction service determines504whether to use a bidirectional LSTM (“BiLSTM”) model to encode the word-level features of the sequence of words. In one embodiment, a BiLSTM includes a forward LSTM that analyzes data from the beginning of the data to the end of the data and a backward LSTM that analyzes data from the end of the data to the beginning of the data. In one embodiment, if the keyphrase extraction service determines504to use a BiLSTM model to encode the word-level features, the keyphrase extraction service encodes506position specific features for each word of the sequence of words using the BiLSTM model, decodes514the position specific features to produce a decoder state for each word of the sequence of words, and extracts516keyphrases from the sequence of words using the decoder states.

In one embodiment, the keyphrase extraction service decodes514the position specific features to produce a decoder state for each word of the sequence using an LSTM. In one embodiment, the keyphrase extraction service decodes514the position specific features to produce a decoder state for each word of the sequence using an BiLSTM. In one embodiment, the keyphrase extraction service decodes514the position specific features to produce a decoder state for each word of the sequence using a CNN. In one embodiment, the keyphrase extraction service decodes514the position specific features to produce a decoder state for each word of the sequence using another neural model (e.g., a gated recurrent unit).

In one embodiment, if the keyphrase extraction service determines504not to use a BiLSTM model to encode the word-level features of the sequence of words, the keyphrase extraction service determines508whether to use a CNN model to encode the word-level features of the sequence of words. In one embodiment, if the keyphrase extraction service determines508to use a CNN model to encode the word-level features of the sequence of words, the keyphrase extraction service encodes510position specific features for each word of the sequence of words using the CNN model, decodes514the position specific features to produce a decoder state for each word of the sequence of words as described herein, and extracts516keyphrases from the sequence of words using the decoder states.

In one embodiment, if the keyphrase extraction service determines508not to use a CNN model to encode the word-level features of the sequence of words, the keyphrase extraction service encodes512position specific features for each word of the sequence of words using another model, decodes514the position specific features to produce a decoder state for each word of the sequence of words as described herein, and extracts516keyphrases from the sequence of words using the decoder states. In one embodiment, the other model is a gated recurrent unit (“GRU”).

FIG. 6illustrates a process600for performing multiple sequencing tasks to identify keyphrases using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In one embodiment, a keyphrase extraction service such as the keyphrase extraction service104described in connection withFIG. 1performs the process600described in connection withFIG. 6. In one embodiment, the keyphrase extraction service receives602a sequence of words. In one embodiment, the keyphrase extraction service selects604the first/next sequence labeling task from one or more sequence labeling tasks. In one embodiment, the sequence labeling task is keyphrase extraction. In one embodiment, the sequence labeling task is position tagging. In one embodiment, the sequence labeling task is part of speech tagging (e.g., to label the word as a noun, verb, adverb, adjective, pronoun, etc.). In one embodiment, part of speech tagging includes tagging of multiple words (e.g., tagging “President of the United States” as a noun phrase).

In one embodiment, the keyphrase extraction service labels606the start of the sequence with a start symbol associated with the sequence labeling task as described herein. In one embodiment, the keyphrase extraction service labels608each of the words in the sequence of words according to the sequence labeling task. In one embodiment, the keyphrase extraction service determines610if there are more sequence labeling tasks to perform. In one embodiment, if the keyphrase extraction service determines610that there are more sequence labeling tasks to perform, the keyphrase extraction service selects604the next sequence labeling task to perform.

In one embodiment, if the keyphrase extraction service determines610that there are no more sequence labeling tasks to perform, the keyphrase extraction service combines612the output of the sequence labeling tasks to encode position-specific features for each word of the sequence of words, decodes614the position specific features to produce a decoder state for each word of the sequence of words, and extracts616features for the sequence of words using the decoder states. In one embodiment, the keyphrase extraction service decodes614the position specific features to produce a decoder state for each word of the sequence using an LSTM. In one embodiment, the keyphrase extraction service decodes614the position specific features to produce a decoder state for each word of the sequence using a BiLSTM. In one embodiment, the keyphrase extraction service decodes614the position specific features to produce a decoder state for each word of the sequence using a CNN. In one embodiment, the keyphrase extraction service decodes614the position specific features to produce a decoder state for each word of the sequence using another neural model (e.g., a GRU). In one embodiment, the features are keyphrases. In one embodiment, the keyphrase extraction service extracts multiple features for each word of the sequence of words.

FIG. 7illustrates a system700where a word representation is generated using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In the system700illustrated inFIG. 7, a word702(“quick”) is provided for character analysis704using techniques described herein. The character analysis704extracts character-level features706, which are then provided for word analysis708using techniques described herein to extract word-level features712from the word. In one embodiment, the word analysis708uses language features710such as a word corpus (or dictionary) or a word gazetteer to perform the word analysis708. In one embodiment, the language features710include language specific features (e.g., are features specific to the English language).

In one embodiment, a word gazetteer is a set of additional information about words that represent classifications of the words according to various classes. In one embodiment, a word gazetteer is created from one or more existing data sets of words. In one embodiment, a word gazetteer includes, as a class, a set of proper names that can be generated from a list of proper names from, for example, a Wikipedia page. In one embodiment, a word gazetteer can include, as a class, a set of business names that can be generated from, for example, Yelp® pages. In one embodiment, a word gazetteer is generated based at least in part on a word corpus such as those described herein. In one embodiment, a word gazetteer and/or a word corpus includes phrases of multiple words (e.g., “President of the United States”). In one embodiment, the neural models for keyphrase extraction described herein treat such phrases of multiple words as single words for the purposes of analysis.

In one embodiment, word702is combined with the character-level features706and/or with the word-level features712to produce a word representation714that includes an aggregate of the extracted features. Use of a word representation such as the word representation714is described in detail below.

FIG. 8illustrates a system800where a word is encoded with features and the features are decoded to produce tag labels usable to identify a keyphrase using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In the system800illustrated inFIG. 8, a set of words802(e.g., “The quick brown fox jumps over . . . ”) is provided for keyphrase extraction. In one embodiment, the set of words802is a sequence of words with a prescribed sequential order. In one embodiment, a word804(e.g., “quick”) of the set of words802is selected for analysis as part of the keyphrase extraction. In one embodiment, the characters806(e.g., “q” “u” “i” “c” “k”) of the word804are selected for character-level encoding as described herein.

In one embodiment, a character-level encoder808uses techniques described herein to extract character-level features810from the characters806of the word804. In one embodiment, the character-level features810from the characters806of the word804are provided to a word-level encoder812that extracts the word-level features814of the word. In one embodiment, the word804is provided to the word-level encoder812in addition to the character-level features810. In one embodiment, the characters806are provided to the word-level encoder812in addition to the character-level features810.

In one embodiment, the character-level features810and the word-level features814are provided to a tag decoder816that generates a set of tags (also referred to herein as tag labels) for the words802, which are used to identify the keyphrase818(e.g., “quick brown fox”) in the set of words802using the techniques described herein.

FIG. 9illustrates a process900for doing hierarchical encoding to encode a word with features that are decoded to produce tag labels usable to identify a keyphrase using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In one embodiment, a keyphrase extraction service such as the keyphrase extraction service104described in connection withFIG. 1performs the process900described in connection withFIG. 9. In one embodiment, the keyphrase extraction service receives902a sequence of words. In one embodiment, the keyphrase extraction service selects904a first/next word from the sequence of words. In one embodiment, the keyphrase extraction service selects906the first/next character of the selected word. In one embodiment, the keyphrase extraction service performs908character-level encoding using the selected character to encode character-level features for the selected character using techniques described herein.

In one embodiment, the keyphrase extraction service determines910whether there are any remaining characters in the word to encode. In one embodiment, if the keyphrase extraction service determines910that there are remaining characters in the word to encode, the keyphrase extraction service selects906the next character of the word. In one embodiment, if the keyphrase extraction service determines910that there are no remaining characters in the word to encode, the keyphrase extraction service performs912word-level encoding on the character-level features of the selected word to encode word-level features for the selected word as described herein. In one embodiment, the keyphrase extraction service determines914whether there are any remaining words to encode. If the keyphrase extraction service determines914that there are remaining words to encode, the keyphrase extraction service selects904the next word. In one embodiment, if the keyphrase extraction service determines914that there are not any remaining words to encode, the keyphrase extraction service performs916tag decoding using the character-level features and the word-level features to generate keyphrases for the sequence of words as described herein.

FIG. 10illustrates a system1000where character encoding is performed with a bidirectional long short-term memory model using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In an embodiment, a BiLSTM includes a forward LSTM1018that analyzes data from the beginning to the end and a backward LSTM1020that analyzes data from the end to the beginning. In the system1000, the character identifiers (e.g., c24for letter “i”1008) are used to identify characters and should not be confused with the cell state for the LSTM described in connection withFIG. 4.

In the system1000, a sequence of characters representing the word “quick” are analyzed. The sequence of characters includes a beginning of word marker1002(e.g., “[BOW]”), the letter “q”1004, the letter “u”1006, the letter “i”1008, the letter “c”1010, the letter “k”1012, an end of word marker1014(e.g., “[EOW]”), and a padding marker1016(e.g., “PAD”) representing the space between “quick” and the next word (e.g., “brown”).

In the system1000, the forward LSTM1018starts with an initial hidden state {right arrow over (h)}20and, using the equations described above at least in connection withFIG. 4, computes forward hidden states for the sequence of letters in the word “quick” (e.g., “BOW” “q” “u” “i” “c” “k” “EOW” “PAD”). The result of this sequence of forward hidden states is the final forward hidden state {right arrow over (h)}29of the sequence. Forward hidden states inFIG. 4(e.g., states of a forward LSTM) are denoted with an arrow from left to right and backward hidden states (e.g., states of a backward LSTM) are denoted with an arrow from right to left.

Also in the system1000, the backward LSTM1020starts with an initial hidden state20and, using the equations described above at least in connection withFIG. 4, computes backward hidden states for the sequence of letters in the word “quick” (e.g., “[PAD]” “[EOW]” “k” “c” “i” “u” “q” “[BOW]”). The result of this sequence of backward hidden states is the final backward hidden state29of the sequence.

In an embodiment, the character-level features for the word “quick” (in this case, the third word) are concatenated together as illustrated in the equation1022(e.g., w3char:=({right arrow over (h)}29;29)) where w3charrepresents the character-level features for the third word and w3charis a combination of the final forward hidden state {right arrow over (h)}29and the final backward hidden state29of the sequence.

FIG. 11illustrates a system1100where character encoding is performed with a convolutional neural network model using a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In the system1100illustrated inFIG. 11, the CNN has two layers (e.g., CNN layer1118and CNN layer1120). In one embodiment, the CNN has more than two layers. In the system1100, the character identifiers (e.g., c24for letter “i”1108) are used to identify characters and should not be confused with the cell state for the LSTM described in connection withFIG. 4.

As with the system1000, in the system1100, a sequence of characters representing the word “quick” are analyzed. The sequence of characters includes a beginning of word marker1102(e.g., “[BOW]”), the letter “q”1104, the letter “u”1106, the letter “i”1108, the letter “c”1110, the letter “k”1112, an end of word marker1114(e.g., “[EOW]”), and a padding marker1116(e.g., “[PAD]”) representing the space between “quick” and the next word (e.g., “brown”).

In the system1100, CNN layer1118analyzes a character and neighboring characters to encode a first set of hidden states. For example, the hidden state h24(1)of the CNN layer1118(which represents the hidden state for the CNN layer1118corresponding to character c24(e.g., the letter “i”1108)) is determined from the character c24as well as from the character c23(e.g., the letter “u”1106) and the character c2(e.g., the letter “c”1110).

In the system1100, CNN layer1120uses the first set of hidden states to encode a second set of hidden states. For example, the hidden state h24(2)of the CNN layer1120(which represents the hidden state for the CNN layer1120corresponding to character c24(e.g., the letter “i”1108)) is determined from the hidden state h24(1)of the CNN layer1118as well as from the hidden state h23(1)of the CNN layer1118and the hidden state h25(1)of the CNN layer1118. As illustrated in the system1100, the two CNN layers (e.g., CNN layer1118and CNN layer1120) encode the hidden state from a number of characters. For example, the hidden state h24(2)of CNN layer1120(which represents the hidden state for the CNN layer1120corresponding to character c24(e.g., the letter “i”1108)) is based on the characters “q” “u” “i” “c” and “k.”

In one embodiment, the character-level features for the word “quick” (in this case, the third word) are concatenated1122together to produce w3char, which represents the character-level features for the third word and is a combination of the hidden states from the CNN layer1120.

FIG. 12illustrates an example1200of an equation for extracting word-level features based in part on a character encoding result usable by a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In the example1200, the equation1202(wifull:=(wichar, wiword, wigaz) encodes the word-level input features (described below and used to encode the word-level features of a word) wfullby combining the character-level features wchardescribed above with word features from a word corpus (wword) and/or a word gazetteer (wgaz). In one embodiment, when a model is being constructed or trained, various dropout parameters are added to the word-level input features to change the weighting of the word-level input features, which, for some models, improves the efficacy of the training phase.

FIG. 13illustrates a system1300where a word representation is generated from extracted word-level features using a bidirectional long short-term memory model in a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In the system1300illustrated inFIG. 13, a sequence of words representing the sequence “The quick brown fox” is analyzed. The sequence of words includes a beginning of sequence marker1302(e.g., “[BOS]”), the word “The”1304, the word “quick”1306, the word “brown”1308, and the word “fox”1310. In the system1300, a forward hidden state {right arrow over (h)}3Encis encoded from the word-level input features w3fullof the word “quick”1306using the forward LSTM1312and a backward hidden state3Encis also encoded from the word-level input features w3fullof the word “quick”1306using the backward LSTM1314. In one embodiment, the word representation h3Encis generated1316from the forward hidden state {right arrow over (h)}3Enc, the backward hidden state3Enc, and the word-level input features w3fullof the word as illustrated by the equation1318(hiEnc=({right arrow over (h)}iEnc,iEnc,wifull)). In one embodiment, the word representation h3Encis generated1316only from the forward hidden state {right arrow over (h)}3Encand the backward hidden state3Enc. It should be noted that, while not illustrated inFIG. 13, the other word representations for the other words (e.g., the word “brown”1308) are computed using the same techniques.

FIG. 14illustrates a system1400where a word representation is generated from extracted word-level features using a convolutional neural network model in a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In the system1400illustrated inFIG. 14, a sequence of words representing the sequence “The quick brown fox” is analyzed. The sequence of words includes a beginning of sequence marker1402(e.g., “[BOS]”), the word “The”1404, the word “quick”1406, the word “brown”1408, and the word “fox”1410. In the example illustrated inFIG. 14, the convolutional neural network model has two layers (e.g., CNN layer1412and CNN layer1414).

As with the CNN used to encode character-level features described in connection withFIG. 11, the CNN used to encode a word representation described in connection withFIG. 14encodes a first hidden state from the words in the CNN layer1412and then encodes a second hidden state from the first hidden states in the CNN layer1414. In the system1400, the CNN layer1412encodes the first hidden state h3(1)corresponding to the word “quick”1406using the full word representation w2fullof the word “The”1404, the full word representation w3fullof the word “quick”1406, and the full word representation w4fullof the word “brown”1408. Similarly, the CNN layer1414encodes the second hidden state h3(2)(also corresponding to the word “quick”1406) using the hidden state h2(1)from CNN layer1412, the hidden state h3(1)from CNN layer1414, and the hidden state h4(1)from CNN layer1412.

In one embodiment, the word representation1416(h3Enc) is encoded from the hidden state h3(2)from CNN layer1414and the word-level input features w3fullof the word “quick”1406as illustrated by the equation1418(hiEnc=(hi(1), wifull)). In one embodiment, the word representation1416(h3Enc) is encoded from the hidden state h3(2)from CNN layer1414without the word-level input features w3fullof the word “quick”1406.

FIG. 15illustrates a system1500where tag labels are generated from word representations using a long short-term memory model in a keyphrase extraction service as described in connection withFIG. 1and in accordance with one embodiment. In the system1500illustrated inFIG. 15, a sequence of words representing the sequence “The quick brown fox” is analyzed. The sequence of words includes the word “The”1502, the word “quick”1504, the word “brown”1506, and the word “fox”1508. In the system1500illustrated inFIG. 15, a forward LSTM1510decodes word representations (described above) from a previous hidden state, the encoded word representation, and a tag for the previous encoded word representation. Although not illustrated inFIG. 15, in one embodiment, the tag labels are generated from word representations using another neural model (e.g., a BiLSTM, a CNN, a GRU, or another neural model).

In one embodiment, a tag that indicates the start of a sequencing task (e.g., keyphrase extraction) is provided as the tag for the previous encoded word representation for the first word in the sequence (e.g., the word “The”1502). In the system1500, the tag1512(e.g., the tag “[GO]”) is provided as the tag for the previous encoded word representation for the first word in the sequence (e.g., the word “The”1502). In one embodiment, a tag that indicates the start of a particular sequencing task (e.g., the tag “[GO_KEYPHRASE]” that indicates the start of a keyphrase extraction task) is provided as the tag for the previous encoded word representation for the first word in the sequence.

In the system1500, the forward LSTM1510receives the word representation h2Enccorresponding to the word “The”1502, a previous hidden state h1Decand the tag1512(“[GO]”) and produces a next hidden state h2Decand a next tag1514(e.g., a null tag “[0]”. The forward LSTM1510continues to analyze the word sequence and next receives the word representation h3Enc, the hidden state h2Dec(e.g., the hidden state from the previous step), the tag1514(“[0]”) and produces a next hidden state h3Decand a tag1516(“[BEG]”) that indicates a probability that the word “quick” is the start of a keyphrase. In one embodiment, the forward LSTM1510continues to analyze the sequence of words until all key phrases are found. In the system1500, the forward LSTM1510generates the tag1518(“[IN]”) that indicates a probability that the word “brown” is inside the keyphrase and the tag1520(“[END]”) that indicates a probability that the word “fox” is the end of the keyphrase. In the system1500, the forward LSTM1510has identified the keyphrase “quick brown fox.”

The illustrative environment includes at least one application server1608and a data store1610. It should be understood that there can be several application servers, layers, or other elements, processes, or components, which may be chained or otherwise configured, which can interact to perform tasks such as obtaining data from an appropriate data store. Servers, as used herein, may be implemented in various ways, such as hardware devices or virtual computer systems. In some contexts, servers may refer to a programming module being executed on a computer system. As used herein, unless otherwise stated or clear from context, the term “data store” refers to any device or combination of devices capable of storing, accessing, and retrieving data, which may include any combination and number of data servers, databases, data storage devices, and data storage media, in any standard, distributed, virtual, or clustered environment. The application server can include any appropriate hardware, software, and firmware for integrating with the data store as needed to execute aspects of one or more applications for the client device, handling some or all of the data access and business logic for an application. The application server may provide access control services in cooperation with the data store and is able to generate content including, but not limited to, text, graphics, audio, video, and/or other content usable to be provided to the user, which may be served to the user by the web server in the form of HyperText Markup Language (“HTML”), Extensible Markup Language (“XML”), JavaScript, Cascading Style Sheets (“CSS”), JavaScript Object Notation (JSON), and/or another appropriate client-side structured language. Content transferred to a client device may be processed by the client device to provide the content in one or more forms including, but not limited to, forms that are perceptible to the user audibly, visually, and/or through other senses. The handling of all requests and responses, as well as the delivery of content between the client device1602and the application server1608, can be handled by the web server using PHP: Hypertext Preprocessor (“PHP”), Python, Ruby, Perl, Java, HTML, XML, JSON, and/or another appropriate server-side structured language in this example. Further, operations described herein as being performed by a single device may, unless otherwise clear from context, be performed collectively by multiple devices, which may form a distributed and/or virtual system.

Each server typically will include an operating system that provides executable program instructions for the general administration and operation of that server and typically will include a computer-readable storage medium (e.g., a hard disk, random access memory, read only memory, etc.) storing instructions that, when executed (i.e., as a result of being executed) by a processor of the server, allow the server to perform its intended functions.

Operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. Processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. The code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-transitory. In some embodiments, the code is stored on set of one or more non-transitory computer-readable storage media having stored thereon executable instructions that, when executed (i.e., as a result of being executed) by one or more processors of a computer system, cause the computer system to perform operations described herein. The set of non-transitory computer-readable storage media may comprise multiple non-transitory computer-readable storage media and one or more of individual non-transitory storage media of the multiple non-transitory computer-readable storage media may lack all of the code while the multiple non-transitory computer-readable storage media collectively store all of the code. Further, in some examples, the executable instructions are executed such that different instructions are executed by different processors. As an illustrative example, a non-transitory computer-readable storage medium may store instructions. A main CPU may execute some of the instructions and a graphics processor unit may execute other of the instructions. Generally, different components of a computer system may have separate processors and different processors may execute different subsets of the instructions.

Accordingly, in some examples, computer systems are configured to implement one or more services that singly or collectively perform operations of processes described herein. Such computer systems may, for instance, be configured with applicable hardware and/or software that enable the performance of the operations. Further, computer systems that implement various embodiments of the present disclosure may, in some examples, be single devices and, in other examples, be distributed computer systems comprising multiple devices that operate differently such that the distributed computer system performs the operations described herein and such that a single device may not perform all operations.