Patent Description:
In some instances, an automated assistant may be accessible to a variety of users having variations in their respective accents, thereby creating accuracy issues with respect to speech recognition. For instance, certain accents can cause a speech recognition process to consistently misrecognize certain words, phrases, and/or phonemes. As a result, users with such accents may have to repeatedly provide spoken utterances until the speech recognition process generates an accurate response to the users. Although certain speech recognition processes may attempt to adapt to certain accents, such processes may only learn from errors that have been explicitly identified by a user. This can result in delays in resolving certain speech recognition issues, and may waste computational resources. For instance, processing spoken utterances directed at an automated assistant in furtherance of correcting recognition mistakes of the automated assistant can be burdensome on portable electronic devices with limited resources.

<CIT> proposes discriminative training of speech recognition language models. A confusion matrix can be generated from acoustic model training data for use in discriminative training. The confusion matrix can include probabilities for the substitution, insertion, and/or deletion of some or all subword units of a language. Probabilities can be calculated based on the presence or absence of subword units in a processed acoustic model training data audio recording when compared to a correct transcription of the recording. The probabilities can be used to generate erroneous transcriptions in language model training corpora, and the language model can be trained to distinguish the erroneous transcriptions from the correct transcriptions.

It is an object of the invention to overcome the shortcomings in the prior art. This object of the invention is solved by the independent claim <NUM>. Hence, a first aspect of the present disclosure provides a method implemented by one or more processors for performing speech recognition on a spoken utterance from a user, as defined in claim <NUM>. Preferred features of the method are set out in claims <NUM> to <NUM>.

Implementations set forth herein relate to speech recognition processes for supplementing a variety of speech recognition hypotheses, in view of differences in accents and/or other speech differences between users, in order to reduce occurrences of misrecognition of speech content. Speech differences (such as but not limited to accents and speech impediments) can exist across a user base, even between users that are located in a similar geographic area. By reducing occurrences of misrecognition, the disclosed technology may provide a more effective speech recognition process. Being able to accurately process speech of a variety of different accents can reduce instances of spoken utterances being misrecognized, users needing to repeat misrecognized spoken utterances, and/or computing devices initializing actions that were not intended by spoken utterances. Each of these accidental occurrences can waste computational resources. However, implementations discussed herein can mitigate such occurrences, thereby preserving essential computational resources such as battery life, processing bandwidth, and/or network bandwidth.

Initially, in order to adapt speech recognition capabilities of a device to variations in accents and/or other speech differences, audio data corresponding to a spoken utterance provided by user is processed in order to generate multiple different speech recognition hypotheses. Each speech recognition hypothesis of the multiple different speech recognition hypotheses includes natural language content that is estimated to correspond to the original natural language content of the spoken utterance provided by the user. For example, a user with an Irish accent can provide a spoken utterance to a computing device in order to identify an object that is within a field of view of a camera of the computing device. The object can be, for instance, a statue located in a downtown area of a city, and in order to identify the object, the user can provide a spoken utterance, such as "Assistant, what is the object to the right of the camera?" Audio data characterizing the spoken utterance can be processed in order to generate a plurality of speech recognition hypotheses. The speech recognition hypotheses generated in response can include phrases, such as "Assistant, what is the object to Detroit of the camera," and, "Assistant, what is the object to the right of the camera.

In order to adapt the speech recognition process according to the accent and/or other speech differences of the user, one or more matching natural language terms can be identified in a set of speech recognition hypotheses. Furthermore, two or more inconsistent natural language terms within the set of speech recognition hypotheses can also be identified. In some implementations, the two or more inconsistent natural language terms can be identified according to whether the two or more inconsistent natural language terms share a common position within their respective speech recognition hypotheses. For example, the term "object to" and the term "of the camera" are matching terms relative to the two speech recognition hypotheses provided above. Furthermore, the term "Detroit" and "the right" are inconsistent between the two speech recognition hypotheses. Moreover, these two inconsistent terms share a common position between the natural language terms, "object to" and "of the camera," provided in each speech recognition hypothesis, where the term "object to" is considered a matching term because the term is present in two speech recognition hypotheses, and the term "of the camera" is also considered a matching term because the term is present in two speech recognition hypotheses.

The multiple speech recognition hypotheses are prioritized and/or ranked in order to identify a particular speech recognition hypothesis, of the multiple speech recognition hypotheses, that is estimated to most correspond to, or to most accurately reflect, the original natural language content of the spoken utterance from the user. For instance, the speech recognition hypothesis, "Assistant, what is the object to the right of the camera" can be selected, for instance, based on a frequency of occurrence of all of, and/or portions of, the prioritized speech recognition hypothesis being greater than the other speech recognition hypotheses, and/or based on an output of a natural language model. In order to further adapt this speech recognition process to the accent of the user (e.g., the accent that resulted in the phrase "the right" being interpreted as "Detroit"), features of (e.g. correspondences and/or inconsistencies between) the plurality of spoken utterance hypotheses can be used in order to improve accuracy of subsequent speech recognition processes.

Term correspondence data is generated to characterize a relationship between two or more inconsistent natural language terms that were identified during processing of a spoken utterance from a user. For instance, the term correspondence data can identify "Detroit" and "the right" as corresponding terms with a candidate term "the right" being the misrecognized term of the original spoken utterance, and a related term "Detroit" being a corresponding predicted term. In some implementations, the term correspondence data can also identify relative position of the related term(s) and/or candidate term(s) within one or more speech recognition hypotheses. For example, the term "to" and/or the term "of" can be identified by the term correspondence data, and the term correspondence data can provide a position indicator that characterizes previous recognized terms as being located between the two terms, "to" and "of. " In some implementations, the term correspondence data can have a format such as, "term[] = term_1 ("Detroit"; "the right"; position[<NUM>, <NUM>, <NUM>] = "to", "term_1", "of")," thereby indicating that the two corresponding terms are "Detroit" and "the right" and, optionally, that the related term "Detroit" is found between terms "to" and "of.

Subsequent to the term correspondence data being generated, the user can provide, at some later time, another spoken utterance that is processed according to the speech recognition techniques discussed herein. Audio data corresponding to the subsequent spoken utterance can be processed in order to generate multiple different speech recognition hypotheses. The speech recognition hypotheses can be analyzed to determine whether any candidate term, identified in the term correspondence data, is included in one or more speech recognition hypotheses. Furthermore, the speech recognition hypotheses can be analyzed to determine whether a corresponding related term, which can be a previously predicted term that corresponds to a candidate term of a previous hypothesis, is also identified in the multiple speech recognition hypotheses.

As an example, the subsequent spoken utterance from the user can be, "Assistant, navigate to Detroit from my current location. " Audio data corresponding to the subsequent spoken utterance can be processed in order to generate a plurality of speech recognition hypotheses. These speech recognition hypotheses can include, "Assistant, navigate to the right of my car's location," "Assistant, navigate to the right from my current location," and "Assistant, navigate to the right for my current vacation.

When a determination is made that a candidate term (e.g., "the right") that is included in term correspondence data is identified in one or more speech recognition hypotheses, a determination can be made as to whether a corresponding related term (e.g., "Detroit") is identified in any speech recognition hypothesis of the multiple speech recognition hypotheses. When the corresponding related term (e.g., "Detroit"), provided in the term correspondence data, is not identified in the multiple speech recognition hypotheses but the candidate term (e.g., "the right") is, one or more supplemental speech recognition hypotheses can be generated (e.g., "Assistant, navigate to Detroit from my current location. A supplemental speech recognition hypothesis can include other terms from the speech recognition hypothesis that have already been generated, as well as the related term (e.g., "Detroit") that has a relationship in the term correspondence data with the candidate term (e.g., "the right"). In this way, the supplemental speech recognition hypotheses can be further processed with the initially-generated speech recognition hypotheses in order to identify a particular speech recognition hypothesis that should be prioritized over the other speech recognition hypotheses (i.e., selected as the correct/most likely to be correct recognition result). A highest prioritized speech recognition hypothesis can then be used for rendering further content and/or initializing an action in response to the subsequent spoken utterance from the user. In some implementations, a supplemental speech recognition hypothesis can be generated by using an existing speech recognition hypothesis and replacing a term from the existing speech recognition hypothesis with a corresponding term from the term correspondence data.

In some implementations, one or more supplemental speech recognition hypotheses can be generated based on one or more initial speech recognition hypotheses having terms that are positioned relative to a candidate term identified in the term correspondence data. For example, a related term can be incorporated into a supplemental speech recognition hypothesis based on a candidate term being located between two particular terms within a speech recognition hypothesis. Alternatively, or additionally, a related term can be incorporated into a supplemental speech recognition hypothesis based on a candidate term being located adjacent to a first particular term within a first initial speech recognition hypothesis and the candidate term also being located adjacent to a second particular term within a second initial speech recognition hypothesis. For example, referring to the aforementioned example, a supplemental speech recognition hypothesis can be generated to include the related term "Detroit" based on an initial speech recognition hypothesis including the term "to" adjacent to the candidate term "the right," and another speech recognition hypothesis including the term "of" adjacent to the candidate term "the right. " These conditions can satisfy position data set forth in the term correspondence data for generating supplemental speech recognition hypotheses when interpreting a given spoken utterance.

Other implementations may include a non-transitory computer readable storage medium or a computer program product storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations may include a system of one or more computers that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

<FIG> illustrates a view <NUM> of a speech recognition process for generating term correspondence data <NUM> from various speech recognition hypotheses <NUM> in order to supplement sets of speech recognition hypotheses in response to subsequent spoken utterances. Specifically, a user <NUM> can be operating a computing device <NUM> and provide a spoken utterance <NUM> in order to control an automated assistant <NUM> that is accessible via the computing device <NUM>. The spoken utterance <NUM> can be received at an audio interface of the computing device <NUM> in order to generate audio data from the spoken utterance <NUM>. The audio data can be processed by a speech recognition engine <NUM> of the computing device <NUM>, which can generate one or more speech recognition hypotheses <NUM> in furtherance of characterizing the spoken utterance <NUM>. Each speech recognition hypothesis <NUM> includes one or more terms <NUM>, which can be words, phrases, letters, and/or phonemes that are estimated, by the speech recognition engine <NUM>, to characterize one or more portions of the spoken utterance <NUM> from the user <NUM>.

One or more speech recognition hypotheses <NUM> and/or terms <NUM> can be processed by a related term engine <NUM> of the computing device <NUM>. The related term engine <NUM> can identify terms that are common to two or more speech recognition hypotheses and, in some implementations, generate a count for each term that is incorporated into two or more speech recognition hypotheses <NUM>. A count for a term <NUM> can indicate a strength of correlation between that term <NUM> and another term <NUM> that is determined to have a lower count. A particular term <NUM> can be subject to counting or other tracking when the particular term <NUM> is a less common term relative to other parts of speech, such as prepositions, pronouns, and conjunctions. For example, for each speech recognition hypothesis <NUM>, the more common parts of speech (e.g., prepositions, pronouns, and conjunctions) can be bypassed in order to identify terms <NUM> for use when generating term correspondence data <NUM>. For example, when considering terms <NUM> that are remaining after common parts of speech are eliminated (e.g., eliminated after a distribution and/or frequency one or more common parts of speech reaches or exceeds a threshold corresponding to the user and/or a population of users), a TERM_1 may appear X number of times and a TERM_2 may appear Y number of times in the speech recognition hypotheses.

The related term engine <NUM> can determine these counts (X and Y, where X and Y are any number(s)) and generate term correspondence data <NUM> that characterizes a relationship between TERM_1 and TERM_2. For example, the term correspondence data <NUM> that is generated by the related term engine <NUM> can include relationship data that characterizes TERM <NUM> as a "RELATED_TERM_1" and TERM <NUM> as a "RELATED_TERM <NUM>. " Furthermore, the term correspondence data <NUM> can characterize these two terms as having previously been included in one or more sets of speech recognition hypotheses <NUM> in response to one or more corresponding spoken utterances from the user <NUM> and/or one or more other users. In this way, when a subsequent spoken utterance is provided by the user <NUM>, and a set of speech recognition hypotheses are generated as including RELATED_TERM_2 but not RELATED _TERM_1, the computing device <NUM> can use the term correspondence data <NUM> in order to generate one or more supplemental speech recognition hypotheses <NUM> that include the RELATED_TERM_1. This can expand a scope of speech recognition hypotheses that will be considered when identifying an appropriate response for the automated assistant <NUM> to provide to the user <NUM>.

During the process of generating the term correspondence data <NUM>, the automated assistant <NUM> can nonetheless respond to the user <NUM> with an output <NUM>, which can include initializing one or more actions via the computing device <NUM>. The response from the automated assistant <NUM> may or may not depend on the term correspondence data <NUM>, considering TERM_1 and TERM_2 were both present in the speech recognition hypotheses <NUM>. However, if a particular related term is identified in at least one speech recognition hypothesis, and another term identified in the term correspondence data <NUM> is not identified in at least one speech recognition hypothesis, the computing device <NUM> generates one or more supplemental speech recognition hypotheses based on the other term identified in the term correspondence data <NUM>.

In some implementations, the term correspondence data <NUM> can provide other data that characterizes the context in which TERM_1 and TERM_2 were both identified as related terms. For example, in some implementations, the term correspondence data <NUM> can be generated to include position data that characterizes a position of TERM_1 relative to one or more other terms in a speech recognition hypothesis <NUM> in which TERM_1 was identified. For instance, TERM_1 and TERM_2 can be identified by the term correspondence data <NUM> as being located directly adjacent to, or otherwise included in a speech recognition hypothesis <NUM>, with a separate term, which can be explicitly identified by the term correspondence data <NUM>. Alternatively, or additionally, TERM_1 and TERM_2 can be identified by the term correspondence data <NUM> as being associated with an application <NUM> that the user <NUM> was accessing when the user <NUM> provided a spoken utterance <NUM> that resulted in TERM_1 being initially included in a speech recognition hypothesis <NUM> generated by the speech recognition engine <NUM>. Alternatively, or additionally, TERM_1 and TERM_2 can be identified by the term correspondence data <NUM> as being associated with a website, a location, a time, one or more other users, and/or any other contextual data, and/or combination thereof, when the user <NUM> provided a spoken utterance <NUM> that resulted in TERM_1 being initially included in a speech recognition hypothesis <NUM> generated by the speech recognition engine <NUM>.

<FIG> illustrates a view <NUM> of a user <NUM> providing a spoken utterance <NUM> and causing one or more supplemental speech recognition hypotheses <NUM> to be generated based on term correspondence data <NUM>. The user <NUM> can provide the spoken utterance <NUM> subsequent to term correspondence data <NUM> being generated for characterizing a relationship between two or more terms (e.g., RELATE_TERM_1 and RELATE_TERM_2), such as in the scenario described with respect to <FIG>. In response to receiving the spoken utterance <NUM>, a speech recognition engine <NUM> of a computing device <NUM> can generate one or more speech recognition hypotheses <NUM>. Each speech recognition hypothesis <NUM> can include one or more terms, which can be a word, phrase, letter, and/or phoneme that can be used to compile a hypothesis with the intention of characterizing natural language content of the spoken utterance.

A related term engine <NUM> can process the speech recognition hypotheses <NUM> in order to determine whether terms <NUM> in the speech recognition hypotheses <NUM> correspond to terms identified in the term correspondence data <NUM>. When a term <NUM> (e.g., "TERM <NUM>) is identified in the term correspondence data <NUM> as having a related term (e.g., "RELATE_TERM4"), the related term engine <NUM> can communicate the term and the related term to the speech recognition engine <NUM>. For example, TERM_4 can be determined to have a corresponding related term, RELATE_TERM_4, which can be identified by the related term engine <NUM> and provided to the speech recognition engine <NUM>. The speech recognition engine <NUM> can use the related term to generated a supplemental speech recognition hypothesis <NUM>, which can include one or more of the terms of an existing speech recognition hypothesis <NUM>, but can replace one or more other terms of the existing speech recognition hypothesis <NUM> with one or more "related" terms identified from the term correspondence data <NUM>. For instance, TERM_1 and TERM_N can be included in a speech recognition hypothesis <NUM> and a supplemental speech recognition hypothesis <NUM>-however, TERM_4 from the speech recognition hypothesis <NUM> can be replaced with RELATED_TERM_4 <NUM> from the term correspondence data <NUM>.

In some implementations, the supplemental speech recognition hypothesis <NUM> can be generated based on whether the user <NUM> provided the spoken utterance <NUM> within a context that corresponds to contextual data provided in the term correspondence data <NUM>. For instance, if the contextual data indicates that the RELATED_TERM _4 <NUM> is related to TERM_4 when the user is accessing application <NUM>, the RELATED_TERM_4 <NUM> can be incorporated into a supplemental speech recognition hypothesis <NUM> when: (i) TERM_4 is provided in a speech recognition hypothesis <NUM> and (ii) the speech recognition hypothesis <NUM> is generated in response to a spoken utterance <NUM> that was provided by the user <NUM> while the user <NUM> was accessing the application <NUM>. Alternatively, or additionally, the supplemental speech recognition hypothesis <NUM> can be generated based on the related term engine <NUM> determining that the TERM_4 <NUM> is positioned within a speech recognition hypothesis <NUM> in accordance with position data included in the term correspondence data <NUM>. For example, the position data can indicate that, when the TERM_4 is positioned at a particular position relative to one or more other terms (e.g., positioned between two particular terms, positioned directly between two particular terms, positioned to the right or left of a particular term, positioned directly to the right or directly to the left of a particular term(s), and/or any other position for a term) within a speech recognition hypothesis <NUM>, the TERM_4 can be related to a RELATED_TERM_4 identified in the term correspondence data <NUM>.

When the speech recognition hypotheses and/or one or more supplemental speech recognition hypotheses have been generated, the computing device <NUM> and/or a server device rank the hypotheses in order to identify a particular speech recognition hypothesis that is prioritized over other hypotheses. The most prioritized speech recognition hypothesis can be one that is determined to most correlate to the spoken utterance <NUM> and/or the context in which the user <NUM> provided the spoken utterance <NUM>. The automated assistant <NUM> can then provide an output <NUM> based on the most prioritized speech recognition hypothesis by initializing one or more actions in accordance with the spoken utterance <NUM>. The degree of priority of each speech recognition hypothesis can be determined based on predicted accuracy of the speech recognition hypothesis relative to the spoken utterance and/or audio data, relevance of each speech recognition hypothesis to a context of the user, historical data characterizing previous interactions between the user and one or more applications, application data, device data, and/or any other information that can be used to prioritize a particular speech recognition hypothesis.

<FIG> illustrates a system <NUM> for supplementing a variety of speech recognition hypotheses, in view of differences in accents and/or other speech differences between users, in order to reduce occurrences of misrecognition of speech content. The system <NUM> can optionally include an automated assistant <NUM>, which can operate as part of an assistant application that is provided at one or more computing devices, such as a computing device <NUM>. A user can interact with the automated assistant <NUM> via an assistant interface <NUM>, which can be a microphone, a camera, a touch screen display, a user interface, and/or any other apparatus capable of providing an interface between a user and an application. For instance, a user can initialize the automated assistant <NUM> by providing a verbal, textual, or a graphical input to the assistant interface to cause the automated assistant <NUM> to perform a function (e.g., provide data, control a peripheral device, access an agent, generate an input and/or an output, etc.). The computing device <NUM> can include a display device, which can be a display panel that includes a touch interface for receiving touch inputs and/or gestures for allowing a user to control applications of the computing device <NUM> via the touch interface. In some implementations, computing device <NUM> can lack a display device, thereby providing an audible user interface output, without providing a graphical user interface output. Furthermore, the computing device <NUM> can provide a user interface, such as a microphone, for receiving spoken natural language inputs from a user. In some implementations, the computing device <NUM> can include a touch interface and can be void of a camera, but can optionally include one or more other sensors.

The computing device <NUM> and/or other computing devices can be in communication with a server device over a network, such as the internet. Additionally, the computing device <NUM> and the other computing devices can be in communication with each other over a local area network (LAN), such as a Wi-Fi network. The computing device <NUM> can offload computational tasks to the server device in order to conserve computational resources at the computing device <NUM>. For instance, the server device can host the automated assistant <NUM>, and computing device <NUM> can transmit inputs received at one or more assistant interfaces <NUM> to a server device. However, in some implementations, the automated assistant <NUM> can be hosted at the computing device <NUM> as a client automated assistant.

In various implementations, all or less than all aspects of the automated assistant <NUM> can be implemented on the computing device <NUM>. In some of those implementations, aspects of the automated assistant <NUM> are implemented via a client automated assistant of the computing device <NUM> and interface with the server device that implements other aspects of the automated assistant <NUM>. The server device can optionally serve a plurality of users and their associated assistant applications via multiple threads. In implementations where all or less than all aspects of the automated assistant <NUM> are implemented via a client automated assistant at the computing device <NUM>, the client automated assistant can be an application that is separate from an operating system of the computing device <NUM> (e.g., installed "on top" of the operating system) - or can alternatively be implemented directly by the operating system of the computing device <NUM> (e.g., considered an application of, but integral with, the operating system).

In some implementations, the automated assistant <NUM> and/or the client automated assistant can include an input processing engine <NUM>, which can employ multiple different engines for processing inputs and/or outputs for the computing device <NUM> and/or the server device. For instance, the input processing engine <NUM> can include a speech processing engine <NUM> that can process audio data received at an assistant interface <NUM> to identify the text embodied in the audio data and/or generate speech recognition hypotheses according to implementations discussed herein. In some implementations, the audio data can be transmitted from, for example, the computing device <NUM> to the server device for further processing in order to preserve computational resources at the computing device <NUM>.

The process for converting the audio data to text can include a speech recognition algorithm, which can employ neural networks, and/or statistical models for identifying groups of audio data corresponding to phonemes, words, and/or phrases. The text converted from the audio data can be parsed by a data parsing engine <NUM> and made available to the automated assistant <NUM> as textual data that can be used to generate and/or identify command phrases from the user. In some implementations, output data provided by the data parsing engine <NUM> can be provided to a parameter engine <NUM> to determine whether the user provided an input that corresponds to a particular action and/or routine capable of being performed by the automated assistant <NUM> and/or an application or agent that is capable of being accessed by the automated assistant <NUM>. For example, assistant data <NUM> can be stored at the server device and/or the computing device <NUM>, as device data <NUM>, and can include data that defines one or more actions capable of being performed by the automated assistant <NUM> and/or client automated assistant, as well as parameters necessary to perform the actions.

In some implementations, the speech processing engine <NUM> can operate one or more engines for generating and/or processing speech recognition hypotheses. For example, the speech processing engine <NUM> can include a phoneme recognition engine <NUM>, which can process audio data in order to identify one or more phonemes, words, and/or phrases being characterized by the audio data. For example, the user can provide a spoken utterance, such as "Assistant, add bacon to my shopping list. " In response, audio data corresponding to the spoken utterance can be processed in order to identify particular phonemes within the audio data. Each phoneme, and/or a combination of phonemes, can be assigned a score, which can characterize a probability that the particular phoneme and/or combination of phonemes accurately characterizes to a particular portion of audio data. For example, a first set of phonemes corresponding to the portion of audio data that captured the user saying the word " bacon," can include "b," "ee," "ca," and "n," and a second set of phonemes can include "b," "a," "er," "ca," and "n.

In some implementations, one or more phonemes and/or sets of phonemes can be assigned a score that is based on speech recognition data characterizing speech patterns of the user and/or one or more other users. Alternatively, or additionally, the one or more phonemes and/or sets of phonemes can be assigned a score that is based on contextual data <NUM>, which can be processed by a context processing engine <NUM>. The context processing engine <NUM> can process contextual data <NUM> in order to determine a likelihood that a phoneme and/or a combination of phonemes would be provided by the user. The contextual data <NUM> can characterize data being rendered by one or more applications of the computing device <NUM> and/or another computing device associated with the user. For instance, when the user is accessing an application <NUM>, such as a recipe application, and the application <NUM> is rendering words or phrases at a graphical user interface of the computing device <NUM>, these words or phrases can be processed by the context processing engine <NUM> in order to assign scores to sets of phonemes based on the context in which the user provided the spoken utterance. When the contextual data characterizes a word that was provided by the user in the spoken utterance, phonemes, which can be combined to pronounce the word, can be prioritized over other phonemes that cannot be combined to pronounce the word when the combination is rendered audibly.

Based on the phonemes and/or the combination of phonemes generated based on the audio data, a speech recognition hypothesis engine <NUM> can generate one or more speech recognition hypotheses. For example, a first speech recognition hypothesis can include, "Assistant, add beer can to my shopping list," and a second speech recognition hypothesis can include, "Assistant, add bacon to my shopping list. " In some implementations, a hypothesis prioritizing engine <NUM> can prioritize one speech recognition hypothesis over other speech recognition hypotheses. Priority of a speech recognition hypothesis can be based on predicted accuracy of the speech recognition hypothesis relative to the spoken utterance and/or audio data, relevance of each speech recognition hypothesis to a context of the user, historical data characterizing previous interactions between the user and one or more applications <NUM>, application data <NUM>, device data <NUM>, and/or any other information that can be used to prioritize a particular speech recognition hypothesis.

In some implementations, the speech processing engine <NUM> can include a related term engine <NUM>. The related term engine <NUM> can identify one or more terms within one or more speech recognition hypothesis that are related to other terms and other speech recognition hypotheses. For example, and in accordance with the previous scenario, the related term engine <NUM> can determine that the first speech recognition hypotheses and the second speech recognition hypothesis are identical except for the term "beer can" and the term "bacon. " Based on this determination, the related term engine <NUM> can generate term correspondence data characterizing a relationship between the term "beer can" and the term "bacon. " Alternatively, or additionally, the related term engine <NUM> can determine that the first speech recognition hypothesis and a second speech recognition hypothesis are identical except for certain phonemes. For example, a phoneme term "ee" from the first speech recognition hypothesis is different from the phone in terms "a" and "er" from the second speech recognition hypothesis. Based on the related term engine <NUM> identifying differences between these phonemes, the related term engine <NUM> can generate term correspondence data characterizing a relationship between the phoneme term "ee" and the phoneme terms "a" and "er.

In some implementations, the speech recognition hypotheses engine <NUM> can use the term correspondence data in order to modify and/or supplement speech recognition hypotheses. For example, should the user provide the same spoken utterance, "Assistant, add bacon to my shopping list," and the speech recognition hypothesis engine <NUM> generate hypotheses that do not include the term bacon, but rather include the term beer can and/or the phoneme term "ee," the speech recognition hypothesis engine <NUM> can generate one or more supplemental speech recognition hypotheses. Specifically, the speech recognition hypotheses engine <NUM> can generate at least one speech recognition hypothesis that includes the term "bacon" and/or phoneme term "ee," at least based on the term correspondence data providing a relationship between the term "beer can" and/or "ee" and the term bacon and/or "a" and "er. " In some implementations, the term correspondence data can identify relative position of a related term in order that the speech recognition hypothesis engine <NUM> can generate supplemental speech recognition hypotheses according to relationships between terms and relative position of related terms within certain speech recognition hypotheses. As an example, the position data of the term correspondence data can indicate that a term (e.g., a phoneme "ee") is related to another term (e.g., a phoneme "a" and/or "er") when the term is directly adjacent to an existing term (e.g., "b" and/or "ca").

When one or more supplemental speech recognition hypotheses have been generated, the already generated speech recognition hypotheses-as well as the supplemental speech recognition hypotheses, can be prioritized via the hypothesis prioritizing engine <NUM>. In some implementations, a speech recognition hypothesis that is prioritized over all others can be the speech recognition hypothesis that is determined to be a most accurate rendering of the spoken utterance provided by the user. This determination can be based on contextual data <NUM>, such as a location of the user, historical interactions between the user and the computing device <NUM> and/or an automated assistant <NUM>, data that is accessible via the computing device <NUM>, data that has been rendered via one or more interfaces of the computing device <NUM> (e.g., a recipe that includes bacon), images rendered at a graphical user interface that is in communication with the computing device and processed by an image recognition engine (e.g., one or more images of bacon) and/or another device that is in communication with the computing device <NUM>, and/or any other information that can be relevant to ranking or scoring speech recognition hypotheses. An output generating engine <NUM> can initialize one or more actions based on the speech recognition hypothesis that is prioritized over all other speech recognition hypotheses. For example, when the speech recognition hypothesis, "Assistant, add bacon to my shopping list," is the highest prioritized speech recognition hypothesis, the automated assistant <NUM> can initialize a shopping application and cause the shopping application to edit a shopping list document to include "bacon.

<FIG> and <FIG> illustrate a method <NUM> and a method <NUM>, respectively, for generating and/or using term correspondence data for creating a variety of different speech recognition hypotheses for reducing frequency of misrecognition of spoken inputs. The method <NUM> and the method <NUM> can be performed by one or more computing devices, applications, and/or any other apparatus or engine capable of being responsive to a spoken input from a user. The method <NUM> can include an operation <NUM> of determining whether a spoken utterance was provided by a user. The determination can be performed at a computing device that is controlled by a user and that includes an audio interface, such as one or more microphones and/or one or more speakers. In some implementations, the computing device can provide access to an automated assistant, which can be responsive to spoken utterances from one or more users and can control one or more applications and/or devices in response to spoken utterances from the one or more users. When a spoken utterance is detected, the method <NUM> proceeds to an operation <NUM>. However, when a spoken utterance is not detected, audio data generated based on inputs to the audio interface can be further processed to determine whether a spoken utterance is directed to the computing device by one or more users.

The operation <NUM> includes generating a plurality of speech recognition hypotheses in furtherance of determining natural language content of the spoken utterance from the user. The speech recognition hypotheses are generated by processing audio data that characterizes the spoken utterance. The audio data can be separated into segments, which can be individually analyzed to identify one or more phonemes that, when spoken by the user, create audible sound waves that resemble a corresponding audio data segment. In some implementations, each phoneme and/or group of phonemes for a particular audio segment can be scored according to their respective similarity to the corresponding audio data segment. Based on the scoring, series of phonemes can be selected for each hypothesis, and the phonemes can be converted into characters, words, and/or phrases, thereby creating the natural language content for each hypothesis.

The method <NUM> proceeds from the operation <NUM> to an operation <NUM>, which includes determining whether a term of a first speech recognition hypothesis is different from another term of a second speech recognition hypothesis. As an example, the user can provide a spoken utterance to the computing device in order to direct a computing device to perform an action, such as controlling lights in the home of the user. In this instance, the spoken utterance can be, "Assistant, turn down the lights in the living room. " Speech recognition hypotheses generated based on the spoken utterance can include, "Assistant, turn in the lights in the living room," and "Assistant, turn on the lights in the living room. " In accordance with the operation <NUM>, the term and/or phoneme "on" can be identified as a term in a speech recognition hypothesis that is different from another term and/or phoneme "in" identified in the other speech recognition hypothesis. However, when the speech recognition hypotheses do not include variations in terms, the method <NUM> can proceed from the operation <NUM>, via continuation element "B," to an operation <NUM> at the method <NUM>.

When a determination is made that the speech recognition hypotheses include a term that is different from another term, the method <NUM> proceeds from the operation <NUM> to an operation <NUM>. The operation <NUM> includes determining whether term correspondence data identifies the term (e.g., "in") or the other term (e.g., "on"). The term correspondence data can include data that characterizes multiple different terms, as well as a relationship between multiple different terms, at least for purposes of generating supplemental speech recognition hypotheses. The term correspondence data can be generated over time as a user interacts with their respective computing devices. In this way, as the user continues to interact with their devices, a lexicon of hypotheses can be expanded for each subsequently provided spoken utterance.

When the term correspondence data does not identify the term or the other term, the method <NUM> proceeds from the operation <NUM> to an operation <NUM>. The operation <NUM> includes generating term correspondence data that characterizes a relationship between (i) the term and a related term, and/or (ii) the other term and another related term. For example, with respect to the aforementioned scenario, the term "in" can be characterized by the term correspondence data as having a relationship with a related term, such as "down. " In some implementations, other content of the hypotheses can also be identified by the term correspondence data and/or used to indicate the relationship between the two terms. For example, one or more words (e.g., "turn") adjacent to each term can be identified by the term correspondence data. In this way, when subsequent hypotheses include the term (e.g., "on"), as well as particular adjacent content (e.g., "turn"), supplemental hypotheses can be generated based on a related term in order to provide additional hypotheses (e.g., "turn [down]. "), which can be considered when performing speech recognition. The method <NUM> proceeds from the operation <NUM>, via continuation element "A," to an operation <NUM> of the method <NUM>.

However, when, at operation <NUM>, the term correspondence data is determined to identify the term or the other term, a comparative method <NUM> proceeds from the operation <NUM> to an optional operation <NUM>. The operation <NUM> includes determining whether the term or the other term is positioned according to the term correspondence data. For instance, the term correspondence data can identify the term as being related to the other term when one or more adjacent terms are present. Therefore, when the one or more adjacent terms are present with the term in a particular speech recognition hypothesis, the inquiry at the operation <NUM> is satisfied and the comparative method <NUM> proceeds to the operation <NUM> via a continuation element "A. " However, and optionally, when one or more adjacent terms are not present with the term in a particular speech recognition hypothesis, the inquiry at the operation <NUM> is determined to not be satisfied, and as a result, the comparative method <NUM> proceeds to the operation <NUM> via the continuation element "B.

The operation <NUM> includes generating one or more supplemental speech recognition hypotheses that include a related term. For example, a supplemental speech recognition hypothesis can include, "Assistant, turn down the lights in the living room. " This supplemental speech recognition hypothesis can be generated based on the term correspondence data defining a relationship between the term "in" and the term "down. " Alternatively, or additionally, this supplemental speech recognition hypothesis can be generated based on determining that the term "in" is adjacent to one or more other terms, which can be identified in position data of the term correspondence data. For example, the term correspondence data can define a relationship and/or condition such as: the term "in" is related to the term "down" when the term "in" is located in "N" speech recognition hypotheses directly adjacent to the term "turn," where N is any number.

The method <NUM> proceeds from the operation <NUM> to an operation <NUM>, which includes determining a priority for each speech recognition hypothesis of the plurality of speech recognition hypotheses, as well as the one or more supplemental speech recognition hypotheses. The priority assigned to each speech recognition hypothesis can be based on a frequency of occurrence of a respective speech recognition hypothesis in historical data that is generated based on interactions between the user and the computing device, and/or a population of users and a variety of different computing devices. In some implementations, a priority assigned to each speech recognition hypothesis can be determined based on contextual data that is associated with a context of the user. For example, the contextual data can characterize content being rendered at one or more graphical user interfaces of one or more computing devices that are associated with the user. For instance, the user can be viewing a website that includes information about turning down lights to preserve energy, thereby resulting in a priority of a hypothesis with the term "down" being prioritized over a different hypothesis not having the term "down.

The method <NUM> can proceed from the operation <NUM> to an optional operation <NUM>, which, if performed, includes causing one or more applications and/or devices to initialize performance of one or more actions according to a prioritized speech recognition hypothesis. For example, the supplemental speech recognition hypothesis, "Assistant, turn down the lights in the living room," can be selected as the most prioritized speech recognition hypothesis. Based on this selection, an automated assistant that is accessible via that computing device can be responsive to the content of the selected hypothesis. Specifically, the automated assistant can initialize an application for controlling one more lights within a living room of a home of the user in accordance with the selected speech recognition hypothesis. In this way, despite the speech recognition process not initially generating a hypothesis that matches the original natural language content of the spoken utterance, an action specifically requested by the user was nonetheless performed without requiring a user to reiterate their spoken utterance and causing the computing device to, again, process another spoken utterance.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the engines described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of method <NUM>, and/or to implement one or more of computing device <NUM>, computing device <NUM>, system <NUM>, automated assistant <NUM>, input processing engine <NUM>, and/or any other application, device, apparatus, and/or engine discussed herein.

These software engines are generally executed by processor <NUM> alone or in combination with other processors. The engines implementing the functionality of certain implementations may be stored by file storage subsystem <NUM> in the storage subsystem <NUM>, or in other machines accessible by the processor(s) <NUM>.

While several implementations have been described and illustrated herein, a variety of other means and/or structures for performing the function and/or obtaining the results and/or one or more of the advantages described herein may be utilized, and each of such variations and/or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and/or configurations will depend upon the specific application or applications for which the teachings is/are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims, implementations may be practiced otherwise than as specifically described. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and/or methods, if such features, systems, articles, materials, kits, and/or methods are not mutually inconsistent, is included within the scope of the present disclosure.

In an implementation, a method implemented by one or more processors for performing speech recognition on a spoken utterance from a user is set forth as including operations such as processing, at a computing device, audio data corresponding to the spoken utterance provided by a user. The method further include an operation of generating, based on processing the audio data, a plurality of speech recognition hypotheses, wherein each speech recognition hypothesis of the plurality of speech recognition hypotheses includes corresponding natural language content predicted to characterize original natural language content of the spoken utterance from the user. The method further include an operation of determining, based on processing the audio data, whether a first term, of a first speech recognition hypothesis of the plurality of speech recognition hypotheses, is different from a second term, of a second speech recognition hypothesis of the plurality of speech recognition hypotheses. The method further include, when the first term of the first speech recognition hypothesis is different from the second term of the second speech recognition hypothesis: generating, based on determining that the first term is different from the second term, term correspondence data that characterizes a relationship between the first term and the second term; and subsequent to generating the term correspondence data: processing the term correspondence data in furtherance of supplementing subsequent speech recognition hypotheses that identify the first term, but not the second term, and generating a supplemental speech recognition hypothesis for the subsequent speech recognition hypotheses, wherein the supplemental speech recognition hypothesis includes the second term. The method further includes determining a prioritized speech recognition hypothesis for the spoken utterance from the plurality of speech recognition hypotheses and the supplemental speech recognition hypothesis.

In a preferred implementation, the method further includes an operation of determining whether the first term and the second term are each predicted based at least in part on a same segment of the audio data, wherein generating the term correspondence data is performed when the first term and the second term are each predicted based at least in part on the same segment of audio data. In a preferred implementation, the method further includes an operation of determining whether the first term of the first speech recognition hypothesis shares a common position with the second term of the second speech recognition hypothesis, wherein generating the term correspondence data is performed when the first term of the first speech recognition hypothesis shares the common position with the second term of the second speech recognition hypothesis.

In a preferred implementation, determining whether the first term of the first speech recognition hypothesis shares the common position with the second term of the second speech recognition hypothesis includes: determining that the first term is directly adjacent to a particular natural language term within the first speech recognition hypothesis of the plurality of speech recognition hypotheses, and determining that the second term is also directly adjacent to the particular natural language term within the second speech recognition hypothesis of the plurality of speech recognition hypotheses. In a preferred implementation, determining whether the first term of the first speech recognition hypothesis shares the common position with the second term of the second speech recognition hypothesis includes: determining that the first term is directly between two natural language terms within the first speech recognition hypothesis of the plurality of speech recognition hypotheses, and determining that the second term is also directly between the two natural language terms within the second speech recognition hypothesis of the plurality of speech recognition hypotheses. In a preferred implementation, the method further includes an operation of determining, subsequent to generating the term correspondence data, a prioritized speech recognition hypothesis from the plurality of speech recognition hypotheses based on contextual data that characterizes a context in which the user provided the spoken utterance; and causing the computing device to render an output based on the prioritized speech recognition hypothesis.

In a preferred implementation, the contextual data characterizes graphical content being rendered at a graphical user interface of the computing device when the user provided the spoken utterance. In a preferred implementation, the contextual data further characterizes one or more applications that are accessible via the computing device, and determining the prioritized speech recognition hypothesis includes: prioritizing each speech recognition hypothesis of the plurality of speech recognition hypotheses according to whether each speech recognition hypothesis corresponds to an action that is capable of being initialized via the one or more applications that are accessible via the computing device. In a preferred implementation, the first term includes one or more phonemes that correspond to one or more words in the first speech recognition hypothesis of the plurality of speech recognition hypotheses. In a preferred implementation, the second term includes one or more other phonemes that correspond to one or more other words in the second speech recognition hypothesis of the plurality of speech recognition hypotheses.

Claim 1:
A method implemented by one or more processors for performing speech recognition on a spoken utterance from a user, the method comprising:
processing, at the computing device, audio data corresponding to the spoken utterance provided by the user;
generating (<NUM>), based on processing the audio data, a plurality of speech recognition hypotheses,
wherein each speech recognition hypothesis of the plurality of speech recognition hypotheses includes corresponding natural language content predicted to characterize original natural language content of the spoken utterance from the user;
determining (<NUM>), based on processing the audio data, whether a first term, of a first speech recognition hypothesis of the plurality of speech recognition hypotheses, is different from a second term, of a second speech recognition hypothesis of the plurality of speech recognition hypotheses; and
when the first term of the first speech recognition hypothesis is different from the second term of the second speech recognition hypothesis:
generating (<NUM>), based on determining that the first term is different from the second term, term correspondence data that characterizes a relationship between the first term and the second term; and
subsequent to generating the term correspondence data:
processing the term correspondence data in furtherance of supplementing subsequent speech recognition hypotheses that identify the first term, but not the second term,
generating (<NUM>) a supplemental speech recognition hypothesis for the subsequent speech recognition hypotheses, wherein the supplemental speech recognition hypothesis includes the second term; and
determining (<NUM>) a prioritized speech recognition hypothesis for the spoken utterance from the plurality of speech recognition hypotheses and the supplemental speech recognition hypothesis.