Patent ID: 12230247

DETAILED DESCRIPTION

FIG.1shows exemplary queries and corresponding results generated for output, in accordance with some embodiments of the disclosure. A first query100“Play ‘Star Wars’” is received from a user. Query100may be incorrectly transcribed or otherwise understood as “Play ‘Star Horse’.” The word “play” is recognized as a command to retrieve and/or output media content. An entity recognition module therefore searches for entities matching the term “Star Horse” and generates a search query for content related to the matching entity. This causes the system to search for and display results matching audio or video content related to the band Star Horse. Search results102are then generated for output to the user visually on a screen such as a smartphone or tablet and/or audibly using a speaker such as on a smart speaker device (e.g., Amazon® Echo®, Google® Home®, etc.). Being dissatisfied with search results102, the user may enter query104“‘Star Wars’ please!” in an attempt to correct the search. While the term “Star Wars” may continue to be incorrectly transcribed as “Star Horse,” the politeness term “please” is recognized as a trigger term indicating that the user wishes to correct or was otherwise dissatisfied with the results of the previous query. For example, each term of query100may be compared with a set of known trigger words, such as a database or other listing comprising politeness terms, negative terms, and corrective terms. In response to the trigger term, the entity recognition module searches for other entities that may have a similar name or phonetic sound to “Star Horse” and correctly determines that the entity for which the user intended to search is “Star Wars.” Using the context of the previous query (i.e., the “play” command to retrieve/output media content), a search query is generated for content matching the term “Star Wars.” Search results106are then generated for output to the user, visually and/or audibly as described above.

FIG.2is a block diagram showing processing of exemplary first and second queries, in accordance with some embodiments of the disclosure. Voice query200“Play ‘Star Wars’” is received and processed using transcription model200, which transcribes voice query200into corresponding text. Query200is transcribed as text204which includes three words “PLAY,” “STAR,” and “HORSE.” Entity recognition module206processes terms204aand204b. “PLAY”204ais recognized as a command to access media content. The system infers from this that the remaining words “STAR” and “HORSE” together identify the desired content, and thus recognizes them as a single term204b. Entity recognition module206determines whether term204bmatches any known entities for which content is available. Entity recognition module206may, for example, access a database or other listing of content creators, content titles, or other metadata describing available content items and search for content items having data matching term204b“STAR HORSE.” The result is a set of data208matching each term of query200(i.e., terms204aand204b) with corresponding portions of a content query. For example, “PLAY” is recognized as being a media request, and is tagged as such, while “STAR HORSE” is recognized as a content filter to be used as a search parameter. Data208is then fed into query construction module210, which generates a database query, such as an SQL “SELECT” statement, to retrieve content items from content database212, listings for which are then generated for output to the user.

If the user is dissatisfied with the results, the user may enter voice query214“No, ‘Star Wars’” in an attempt to correct the results of the previous query. Query214is again processed using transcription model202to generate text216. Query214is again transcribed into three words “NO,” “STAR,” and “HORSE.” The term “NO”216ais identified by entity recognition module206as a trigger term for a corrective query, and again infers that “STAR” and “HORSE” together form a single term216b. In response to the trigger term, entity recognition module206increases a relaxation rate218for the term “STAR HORSE.” For example, the entity recognition module may consider entities that are phonetically similar or may search for additional types of content. By increasing the relaxation rate, entity recognition module206correctly identifies “Star Wars” as the content to which query214refers. Since neither term216anor term216bprovides a context for the query, entity recognition module206retrieves the context of the previous query200. The result is a set of data220that combines the context of query200(i.e., a media request based on the “PLAY” command) with the content filter “STAR WARS” from query214. Data220is then fed into query construction module210, which generates a database query, such as an SQL “SELECT” statement, to retrieve content items from content database212, listings for which are then generated for output to the user.

FIG.3is a block diagram showing components and data flow therebetween of a system for improving content discovery in response to a voice query, in accordance with some embodiments of the disclosure.

A first voice query is received300at input circuitry302. Input circuitry302may be part of a media device on which the system of the present disclosure is implemented, or may be a separate device, such as an Amazon Echo® or Google Home® device, or any other device capable of receiving and relaying user input to a media device. Input circuitry302may be a data interface such as a Bluetooth module, WiFi module, or other suitable data interface through which data entered on another device or audio data captured by another device can be received. Alternatively, input circuitry302may include a microphone through which audio information is captured directly. Input circuitry302may convert the audio to a digital format such as WAV. Input circuitry302communicates304the voice query to control circuitry306. Control circuitry306may be based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputer. In some embodiments, processing circuitry may be distributed across multiple separate processors or processing units, for example, multiple of the same type of processing units (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor).

Control circuitry306receives and processes the voice query using natural language processing circuitry308. In some embodiments, control circuitry306or natural language processing circuitry308may include the transcription circuitry/programming to transcribe audio data of the voice query into a corresponding string of text. Natural language processing circuitry308identifies a plurality of terms in the voice query. For example, natural language processing circuitry308may identify individual words in the voice query using spaces in a transcription or pauses or periods of silence in the voice query. Natural language processing circuitry308analyzes a first word and determines whether the first word can be part of a larger phrase. For example, natural language processing circuitry308may access a dictionary or other word list or phrase list from memory312. Memory312may be an electronic storage device such as random-access memory, read-only memory, hard drives, optical drives, solid state devices, quantum storage devices, or any other suitable fixed or removable storage devices, and/or any suitable combination of the same.

Using the dictionary or word list or phrase list, natural language processing circuitry308determines if the first word can be followed by at least a second word. If so, natural language processing circuitry308analyzes the first word together with the word immediately following the first word to determine if the two words together form a phrase. If so, the phrase is identified as a single term in the voice query. Otherwise, the first word alone is identified as a single term in the voice query.

Natural language processing circuitry308determines whether any of the identified terms corresponds to a command. For example, the word “play” or the phrase “show me” may be recognized as a command to access media content for playback. Natural language processing circuitry308then determines that the term or terms that follow the command correspond to a type of content or a specific content item to which the voice query refers. Natural language processing circuitry308, using an entity recognition module such as entity recognition module206, identifies the specific entity to which the voice query refers. The entity recognition module of natural language processing circuitry308requests310a list of entities matching the appropriate term of the voice query from memory312. The entity recognition module receives314at least one matching entity and selects a candidate entity based on the context of the voice query. For example, if the command given in the voice query relates specifically to audio content (e.g., music, or audiobooks), the entity recognition module may select a candidate entity corresponding to a musical group or an author.

Once a candidate entity has been selected, natural language processing circuitry308transmits316the context of the voice query and the selected candidate entity to query construction circuitry318. Natural language processing circuitry308also stores data describing the voice query, including the context and the selected candidate entity in, for example, memory312. Query construction circuitry318then constructs a search query corresponding to context and candidate entity. For example, if the context is media content and the candidate entity is Star Horse, query construction circuitry318constructs a query for content created by or including the band Star Horse. For example, query construction circuitry318may generate an SQL “SELECT” statement such as “SELECT*FROM video_content WHERE creator CONTAINS ‘STAR HORSE’.” Query construction circuitry318transmits320the constructed search query to transceiver circuitry322, which transmits324the search query to, for example, content database212. Transceiver circuitry322may be a network connection such as an Ethernet port, WiFi module, or any other data connection suitable for communicating with a remote server. Transceiver circuitry322then receives326search results from content database212, which may include content identifiers for a plurality of content items matching the search criteria. Transceiver circuitry322transmits328the search results to output circuitry330. Output circuitry330may be any video or graphics processing circuitry suitable for generating an image for display on a display device associated with control circuitry306, and/or any audio processing circuitry suitable for generating an audio signal for output using a speaker or other audio device associated with control circuitry306. Output circuitry330then outputs332the content identifiers.

A second voice query is the received334using input circuitry302. The second voice query is processed by input circuitry302just as the first voice query was processed and transmitted336to control circuitry306. The second voice query is then transcribed using natural language processing circuitry308, and terms of the second voice query are identified. If the second voice query contains a trigger term, such as a politeness term (e.g., “please”), negative term (e.g., “no”) or corrective term (e.g., “I meant”), then natural language processing circuitry308determines that the second voice query is a corrective query, and that the results of the previous query were not the correct results. Natural language processing circuitry308requests338the stored data describing the previous query from memory312. Natural language processing circuitry308receives340the stored data and identifies the context of the previous voice query as the context of the second voice query. Natural language processing circuitry308also determines a similarity between the remaining terms of the second voice query and those of the previous voice query as described by the stored data. If the terms are similar (e.g., at least a threshold percentage of the words of a given term are the same, are spelled similarly, or are phonetically similar), then natural language processing circuitry308increases a relaxation rate of the entity recognition module. The relaxation rate controls the number of candidate entities considered by the entity recognition module. The relaxation rate is normally low to conserve system resources by limiting the number of variants of a term searched by the entity recognition module. However, for a corrective search to be able to retrieve the correct search results, the relaxation rate is increased to allow the entity recognition module to consider additional variants of the terms of the query. The entity recognition module requests342an expanded list of entities matching the appropriate term of the second voice query in accordance with the relaxation rate. The entity recognition module receives344the expanded list and selects a candidate entity different from that which was selected for the previous voice query. Natural language processing circuitry308then transmits346the context of the voice query and the selected candidate entity to query construction circuitry318, which constructs a new query as described above. Query construction circuitry318transmits348the new query to transceiver circuitry322, which in turn transmits350the new query to content database212. Transceiver circuitry322then receives352new search results from content database212, including content identifiers of a new plurality of content items that match the new search criteria, and transmits354the search results to output circuitry330. Output circuitry330then outputs356the content identifiers.

FIG.4is a flowchart representing an illustrative process400for processing first and second voice queries and generating corresponding search results for output, in accordance with some embodiments of the disclosure. Process400may be implemented on control circuitry306. In addition, one or more actions of process400may be incorporated into or combined with one or more actions of any other process or embodiment described herein.

At402, control circuitry306initializes a variable N representing the number of queries that have been received, including a query received at the current time. For example, control circuitry306may count the number of queries received within a period of time, such as five minutes. It may be assumed that a query received at the current time is not related to a query received earlier than the period of time prior to the current time. Alternatively, control circuitry306may continue to count the number queries received until a threshold amount of time has passed in which no queries have been received. It may be assumed that if a threshold amount of time has passed since the last query was received, then a query received at the current time will not be related to any past queries.

At404, control circuitry306, using natural language processing circuitry308, generates a transcription of the Nthquery, where the Nthquery is the query received at the current time. At406, control circuitry306, using natural language processing circuitry308, identifies a plurality of terms of the Nthquery. For example, natural language processing circuitry308may identify a plurality of words in the Nthquery and determine a part of speech for each word. Natural language processing circuitry308may then determine, based on the part of speech of each word, whether each word is part of a phrase including a neighboring word. If so, the neighboring words are identified together as a single term. Otherwise, a single word is identified as a term of the Nthquery.

At408, control circuitry306, using natural language processing circuitry308, determines whether the transcription of the Nthquery includes a trigger term, such as a politeness term (e.g., “please”), a negative term (e.g., “no”), a corrective term (e.g., “I meant”) or any combination thereof. If the Nthquery does not include a trigger term (“No” at408), then, at410, control circuitry306, using natural language processing circuitry308, identifies a context of the Nthquery based on the plurality of terms. For example, the query may include a “play” command, indicating that the context of the query is a request for media content. Other examples include phrases such as “I want to hear,” indicating a request for audio content, and “tell me,” indicating a request for information.

If the Nthquery does contain a trigger term (“Yes” at408), then, at412, control circuitry306retrieves stored data describing a previous number of queries K. For example, control circuitry306may retrieve stored data for the previous five queries, or may retrieve stored data for each query of the K queries prior to the Nthquery. Then, at414, control circuitry306determines whether the Nthquery contains a term that is similar to a term of one of the previous K queries. For example, natural language processing circuitry308may determine if a term on the Nthquery is phonetically similar to a term of one of the previous K queries. If the Nthquery contains a term similar to a term of one of the previous K queries (“Yes” at414), then, at416, control circuitry306retrieves the context of the query containing the similar term. For example, control circuitry306may retrieve a stored context for the particular previous query. Alternatively or additionally, control circuitry306may determine if the particular previous query refers to similar subject matter or entities as a group of temporally adjacent queries. If so, a composite context may be generated from the specific contexts of each of these related queries. At418, control circuitry306also modifies an entity recognition model for the similar term. For example, natural language processing circuitry308may contain an entity recognition module, which executes an entity recognition model. Control circuitry306may temporarily increase a relaxation rate of the entity recognition module, which controls the number of variants of the particular term considered by the entity recognition module in identifying entities that match the particular term. Increasing the relaxation rate allows the entity recognition module to consider more variants of the particular term in order to arrive at the correct entity. If the Ni query does not contain a term that is similar to a term of any of the previous K queries (“No” at414), then processing continues at step410, and control circuitry306determines the context of the Nthquery as though no trigger term were present.

At420, control circuitry306identifies a plurality of candidate entities to which the Nthquery refers. In cases where the Nthquery does not include a trigger term, this occurs after identifying the context of the Nthquery at step410. In cases where the Nthquery does include a trigger term, this may occur upon determining that the Nthquery does not contain any terms that are similar to any term of any of the previous K queries (“No” at414), or after modifying the entity recognition at step418. For example, natural language processing circuitry308, or an entity recognition module thereof, requests a list of candidates from a database or data structure in which the names of various known entities are stored. A number of entities that match the terms of the Nthquery, up to a maximum number as governed by the relaxation rate, are then retrieved. At422, control circuitry306, using the entity recognition module of natural language processing circuitry308, selects an entity of the plurality of candidate entities based on the context. Control circuitry306then stores data describing the Nthquery for use in processing later queries.

At426, control circuitry306performs a search based on the context of the Nthquery and the candidate entity or entities selected by the entity recognition module. In response to the search, control circuitry306receives at least one search result which is, at428, generated for output. For example, the search results can be generated for output visually on a screen or other display device. Alternatively or additionally, the results can be generated for output audibly using a text-to-speech engine.

The actions and descriptions ofFIG.4may be used with any other embodiment of this disclosure. In addition, the actions and descriptions described in relation toFIG.4may be done in suitable alternative orders or in parallel to further the purposes of this disclosure.

FIG.5is a flowchart representing an illustrative process500for refining a transcription model, in accordance with some embodiments of the disclosure. Process500may be implemented on control circuitry306. In addition, one or more actions of process500may be incorporated into or combined with one or more actions of any other process or embodiment described herein.

At502, control circuitry306determines whether the voice query contains a trigger word. If so (“Yes” at502), then, at504, control circuitry306stores an indication that the first transcription is incorrect. For example, as part of the data describing each query, control circuitry306may also include a flag or other indication that the transcription of the voice query is incorrect. At506, based on the indication that the transcription of the voice query was incorrect, control circuitry306refines the transcription model. For example, natural language processing circuitry308may employ a transcription model for performing transcription of voice queries into corresponding text. The transcription model may be a default transcription model that is optimized for an average accent in the language for which the transcription model is set. For example, a US English transcription model may be optimized for a Midwestern accent, which has very little stress on each syllable and open vowel qualities. However, a voice query may be received by a user having a heavy Southern accent, or by a non-native English speaker. Control circuitry306may identify the accent used by the speaker and refine the transcription model to more accurately process speech in the identified accent.

The actions and descriptions ofFIG.5may be used with any other embodiment of this disclosure. In addition, the actions and descriptions described in relation toFIG.5may be done in suitable alternative orders or in parallel to further the purposes of this disclosure.

FIG.6is a flowchart representing an illustrative process600for identifying a plurality of candidate entities to which a term of a corrective voice query refers, in accordance with some embodiments of the disclosure. Process600may be implemented on control circuitry306. In addition, one or more actions of process600may be incorporated into or combined with one or more actions of any other process or embodiment described herein.

At602, control circuitry306determines whether a term of the voice query is phonetically similar to a term of a previous voice query. For example, control circuitry306may retrieve audio data of previous voice queries and compare audio corresponding to a particular term with audio corresponding to each term of the voice query. If the audio data is within a threshold difference from the audio data of a term of a previous query, control circuitry306determines that the terms are phonetically similar. Alternatively or additionally, control circuitry306may compare the transcription of each term to determine, based on spelling, whether the terms should be phonetically similar in pronunciation.

If a term of the voice query is phonetically similar to a term of a previous query (“Yes” at602), then, at604, control circuitry306modifies an entity recognition model by temporarily increasing a relaxation rate, wherein the number of interpretations considered for a particular term is based on the relaxation rate. At606, control circuitry306identifies the second plurality of candidate entities to which the term of the voice query refers using the modified entity recognition model.

The actions and descriptions ofFIG.6may be used with any other embodiment of this disclosure. In addition, the actions and descriptions described in relation toFIG.6may be done in suitable alternative orders or in parallel to further the purposes of this disclosure.

FIG.7is a flowchart representing an illustrative process700for identifying entities to which a term refers, in accordance with some embodiments of the disclosure. Process700may be implemented on control circuitry306. In addition, one or more actions of process700may be incorporated into or combined with one or more actions of any other process or embodiment described herein.

At702, control circuitry306identifies at least one phrase in the transcription. For example, control circuitry306, using natural language processing circuitry308, determines whether each word can be combined with an adjacent word to form a larger phrase. At704, control circuitry306determines a plurality of variants of the at least one phrase. For example, control circuitry306, using natural language processing circuitry308, compares each phrase with a list of known phrases that are phonetically similar to the identified phrase, or have different orders of the same words of the identified phrase. At706, control circuitry306then maps the at least one phrase to at least one entity based on the plurality of variants. For example, a non-native English speaker may say “Show me the Star Wars” when requesting playback of “Star Wars.” The phrase “the Star Wars” is a variant of “Star Wars” that includes the definite article which is not needed in the grammatical context of this query. Control circuitry306therefore maps the phrase “the Star Wars” to the movie “Star Wars.”

The actions and descriptions ofFIG.7may be used with any other embodiment of this disclosure. In addition, the actions and descriptions described in relation toFIG.7may be done in suitable alternative orders or in parallel to further the purposes of this disclosure.

FIG.8is a flowchart representing an illustrative process800for determining the context of a query, in accordance with some embodiments of the disclosure. Process800may be implemented on control circuitry306. In addition, one or more actions of process800may be incorporated into or combined with one or more actions of any other process or embodiment described herein.

At802, control circuitry306determines whether the voice query was received within a threshold amount of time from a time at which the previous voice query was received. For example, if the previous voice query was received over five minutes ago, it may not be relevant to a query received at the current time. If the voice query was received within the threshold amount of time (“Yes” at802), then, at804, control circuitry306retrieves the context of the previous voice query as described above at step416ofFIG.4. If the voice query was received after the threshold amount of time has passed (“No” at802), then, at806, control circuitry306identifies the context of the voice query based on the plurality of terms it contains, as described above at step410ofFIG.4.

The actions and descriptions ofFIG.8may be used with any other embodiment of this disclosure. In addition, the actions and descriptions described in relation toFIG.8may be done in suitable alternative orders or in parallel to further the purposes of this disclosure.

The processes described above are intended to be illustrative and not limiting. One skilled in the art would appreciate that the steps of the processes discussed herein may be omitted, modified, combined, and/or rearranged, and any additional steps may be performed without departing from the scope of the invention. More generally, the above disclosure is meant to be exemplary and not limiting. Only the claims that follow are meant to set bounds as to what the present invention includes. Furthermore, it should be noted that the features and limitations described in any one embodiment may be applied to any other embodiment herein, and flowcharts or examples relating to one embodiment may be combined with any other embodiment in a suitable manner, done in different orders, or done in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and/or methods described above may be applied to, or used in accordance with, other systems and/or methods.