Patent Description:
Users frequently interact with voice-enabled devices, such as smart phones, smart watches, and smart speakers, through digital assistant interfaces. These digital assistant interfaces enable users to complete tasks and obtain answers to questions they have all through natural, conversational interactions. Exemplary voice-enabled devices are disclosed in <CIT> and in <CIT>.

Ideally, when conversing with a digital assistant interface, a user should be able to communicate as if the user were talking to another person, via spoken requests directed toward their voice-enabled device running the digital assistant interface. The digital assistant interface will provide these spoken requests to an automated speech recognizer to process and recognize the spoken request so that an action can be performed. In practice, however, it is challenging for a device to always be responsive to these spoken requests since it is prohibitively expensive to run speech recognition continuously on a resource constrained voice-enabled device, such as a smart phone or smart watch.

One aspect of the disclosure provides a method of decaying automated speech recognition processing. The method includes receiving, at data processing hardware of a voice-enabled device, an indication of a microphone trigger event indicating a possible user interaction with the voice-enabled device through speech where the voice-enabled device has a microphone that, when open, is configured to capture speech for recognition by an automated speech recognition (ASR) system. In response to receiving the indication of the microphone trigger event, the method also includes instructing, by the data processing hardware, the microphone to open or remain open for an open microphone duration window to capture an audio stream in an environment of the voice-enabled device and providing, by the data processing hardware, the audio stream captured by the open microphone to the ASR system to perform ASR processing over the audio stream. While the ASR system is performing the ASR processing over the audio stream captured by the open microphone, the method further includes decaying, by the data processing hardware, a level of the ASR processing that the ASR system performs over the audio stream based on a function of the open microphone duration window and instructing, by the data processing hardware, the ASR system to use the decayed level of the ASR processing over the audio stream captured by the open microphone.

In some examples, while the ASR system is performing the ASR processing over the audio stream captured by the open microphone, the method also includes determining, by the data processing hardware, whether voice activity is detected in the audio stream captured by the open microphone. In these examples, decaying the level of the ASR processing the ASR system performs over the audio stream is further based on the determination of whether any voice activity is detected in the audio stream. In some implementations, the method further includes obtaining, by the data processing hardware, a current context when the indication of the microphone trigger event is received. In these implementations, instructing the ASR system to use the decayed level of the ASR processing includes instructing the ASR system to bias speech recognition results based on the current context. In some configurations, after instructing the ASR system to use the decayed level of the ASR processing over the audio stream, the method additionally includes receiving, at the data processing hardware, an indication that a confidence for a speech recognition result for a voice query output by the ASR system fails to satisfy a confidence threshold and instructing, by the data processing hardware, the ASR system to increase the level of ASR processing from the decayed level and reprocess the voice query using the increased level ASR processing. In some implementations, while the ASR system is performing the ASR processing over the audio stream captured by the open microphone, determining, by the data processing hardware, when the decayed level of the ASR processing the ASR performs over the audio stream based on the function of the open microphone duration is equal to zero and when the decayed level of the ASR processing is equal to zero, instructing, by the data processing hardware, the microphone to close. Optionally, the method may also include displaying, by the data processing hardware, in a graphical user interface of the voice-enabled device where a graphical indicator indicates the decayed level of ASR processing performed by the ASR system on the audio stream.

Another aspect of the disclosure provides a voice-enabled device for decaying automated speech recognition processing. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform one of the methods described above.

Conversations with digital assistant interfaces are typically initiated with a convenient interaction by a user, such as a user speaking a fixed phrase (e.g., a hotword/keyword/wake word) or using some predefined gesture (e.g., raising or squeezing the voice-enabled device). However, once a conversation starts, requiring the user to speak the same fixed phrase or use the same predefined gesture for each successive spoken request/query can be cumbersome and inconvenient for the user. To mitigate this requirement, a microphone of the voice-enabled device can be kept open for some predefined amount of time immediately after an interaction to permit the microphone to capture immediate follow-up queries spoken by the user in a much more natural way. However, there are certain trade-offs for how long the microphone should be kept upon immediately following an interaction. For instance, leaving the microphone open for too long can lead to unnecessarily consuming power for performing speech recognition and increasing the likelihood of capturing unintended speech in the environment of the voice-enabled device. While, on the other hand, closing the microphone too soon creates a bad user experience since a user is required to re-initiate the conversation via the fixed phrase, gesture, or other means which may be inconvenient.

Implementations herein are directed toward initiating a speech recognizer to perform speech recognition responsive to an event by opening a microphone of a voice-enabled device and gradually decaying both the responsiveness and processing power of the speech recognizer. More specifically, implementations herein include decaying a speech recognition processing level based on a probability of user interaction, such as a follow-up query after an initial query, with the voice-enabled device. In contrast to making a binary decision to close the microphone and prevent further processing by the speech recognizer at some arbitrary point in time, decaying processing by the speech recognizer over time improves both the user experience, by keeping the microphone open longer to capture follow-up speech directed toward the voice-enabled device, as well as power consumption savings, by permitting the speech recognizer to run at different power modes depending on a level of confidence an upcoming user interaction.

Referring to <FIG> and <FIG>, in some implementations, a system <NUM> includes a user <NUM> providing a user interaction <NUM> to interact with a voice-enabled device <NUM> (also referred to as a device <NUM> or a user device <NUM>). Here, the user interaction <NUM> is a spoken utterance <NUM>, 12U corresponding to a query or a command to solicit a response from the device <NUM> or to have the device <NUM> execute a task specified by the query. In this sense, the user <NUM> may have conversation interactions with the voice-enabled device <NUM> to perform computing activities or find answers to questions.

The device <NUM> is configured to capture user interactions <NUM>, such as speech, from one or more users <NUM> within the speech environment. An utterance 12U spoken by the user <NUM> may be captured by the device <NUM> and may correspond to a query or a command for a digital assistant interface <NUM> executing on the device <NUM> to perform an operation/task. The device <NUM> may correspond to any computing device associated with the user <NUM> and capable of receiving audio signals. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music player, casting devices, smart appliances (e.g., smart televisions) and internet of things (IoT) devices, remote controls, smart speakers, etc. The device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations related to speech processing.

The device <NUM> further includes an audio subsystem with an audio capturing device (e.g., an array of one or more microphones) <NUM> for capturing and converting audio data within the speech environment into electrical signals. While the device <NUM> implements the audio capturing device <NUM> (also referred to generally as a microphone <NUM>) in the example shown, the audio capturing device <NUM> may not physically reside on the device <NUM>, but be in communication with the audio subsystem (e.g., peripherals of the device <NUM>). For example, the device <NUM> may correspond to a vehicle infotainment system that leverages an array of microphones positioned throughout the vehicle.

A speech-enabled interface (e.g., a digital assistant interface) <NUM> may field the query or the command conveyed in the spoken utterance 12U captured by the device <NUM>. The speech-enabled interface <NUM> (also referred to as interface <NUM> or an assistant interface <NUM>) generally facilitates receiving audio data <NUM> corresponding to an utterance 12U and coordinating speech processing on the audio data <NUM> or other activities stemming from the utterance 12U to generate a response <NUM>. The interface <NUM> may execute on the data processing hardware <NUM> of the device <NUM>. The interface <NUM> may channel audio data <NUM> that includes an utterance 12U to various systems related to speech processing. For instance, <FIG> illustrates that the interface <NUM> communicates with a speech recognition system <NUM>. Here, the interface <NUM> receives audio data <NUM> corresponding to an utterance 12U and provides the audio data <NUM> to the speech recognition system <NUM>. In some configurations, the interface <NUM> serves as an open communication channel between the microphone <NUM> of the device <NUM> and the speech recognition system <NUM>. In other words, the microphone <NUM> captures an utterance 12U in an audio stream <NUM> and the interface <NUM> communicates audio data <NUM> corresponding to the utterance 12U converted from the audio stream <NUM> to the speech recognition system <NUM> for processing. More specifically, the speech recognition system <NUM> processes the audio data <NUM> to generate a transcription <NUM> for the utterance 12U and may perform semantic interpretation on the transcription <NUM> to identify an appropriate action to perform. The interface <NUM> used to interact with the user <NUM> at the device <NUM> may be any type of program or application configured to execute the functionality of the interface <NUM>. For example, the interface <NUM> is an application programming interface (API) that interfaces with other programs hosted on the device <NUM> or in communication with the device <NUM>.

Referring specifically to the example of <FIG>, a first utterance 12U, 12Ua by the user <NUM> states "Hey computer, who is the president of France?" Here, the first utterance 12U includes a hotword <NUM> that when detected in the audio data <NUM> triggers the interface <NUM> to open the microphone <NUM> and relay subsequently captured audio data corresponding to the query "who is the president of France?" to the speech recognition system <NUM> for processing. That is, the device <NUM> may be in a sleep or hibernation state and run a hotword detector to detect the presence of the hotword <NUM> in the audio stream <NUM>. Serving as an invocation phrase, the hotword <NUM> when detected by the hotword detector, triggers the device <NUM> to wake-up and initiate speech recognition on the hotword <NUM> and/or one or more terms following the hotword <NUM>. The hotword detector may be a neural network-based model configured to detect acoustic features indicative of the hotword without performing speech recognition or semantic analysis.

In response to the hotword detector detecting the hotword <NUM> in the audio stream <NUM>, the interface <NUM> relays the audio data <NUM> corresponding to this utterance 12Ua to the speech recognition system <NUM> and the speech recognition system <NUM> performs speech recognition on the audio data <NUM> to generate a speech recognition result (e.g., transcription) <NUM> for the utterance 12Ua. The speech recognition system <NUM> and/or the interface <NUM> performs semantic interpretation on the speech recognition result <NUM> to determine that the utterance 12Ua corresponds to a search query for the identity of the president of France. Here, the interface <NUM> may submit the transcription <NUM> to a search engine <NUM> that searches for, and returns a search result <NUM> of "Emmanuel Macron" for the query of "Who is the president of France?" The interface <NUM> receives this search result <NUM> of "Emmanuel Macron" from the search engine <NUM> and, in turn, communicates "Emmanuel Macron" to the user <NUM> as a response <NUM> to the query of the first utterance 12Ua. In some examples, the response <NUM> includes synthesized speech audibly output from the device <NUM>.

To perform the functionality of the assistant interface <NUM>, the interface <NUM> may be configured to control one or more peripherals of the device <NUM> (e.g., one or more components of the audio subsystem). In some examples, the interface <NUM> controls the microphone <NUM> in order to dictate when the microphone <NUM> is open, or actively receiving audio data <NUM> for some speech processing purpose, or closed, or not receiving audio data <NUM> or receiving a restricted amount of audio data <NUM> for speech processing purposes. Here, whether the microphone <NUM> is "open" or "closed" may refer to whether the interface <NUM> communicates audio data <NUM> received at the microphone <NUM> to the speech recognition system <NUM> such that the interface <NUM> has an open channel of communication with the speech recognition system <NUM> to enable speech recognition for any utterance 12U included in an audio stream <NUM> received at the microphone <NUM> or a closed channel of communication with the speech recognition system <NUM> to disable the speech recognition system <NUM> from performing speech recognition over the audio stream <NUM>. In some implementations, the interface <NUM> designates or instructs whether such a channel is open or closed based on whether the interface <NUM> receives or has received an interaction <NUM> and/or an interaction <NUM> with a trigger <NUM>. For instance, when the interface <NUM> receives an interaction <NUM> with a trigger <NUM> (e.g., a spoken utterance 12U with a hotword), the interface <NUM> instructs the microphone <NUM> to open and relays audio data <NUM> converted from audio captured by the microphone <NUM> to the speech recognition system <NUM>. After the utterance is complete, the interface <NUM> may instruct the microphone <NUM> to close to prevent the speech recognition system <NUM> from processing additional audio data <NUM>.

In some implementations, the device <NUM> communicates via a network <NUM> with a remote system <NUM>. The remote system <NUM> may include remote resources <NUM>, such as remote data processing hardware <NUM> (e.g., remote servers or CPUs) and/or remote memory hardware <NUM> (e.g., remote databases or other storage hardware). The device <NUM> may utilize the remote resources <NUM> to perform various functionality related to speech processing. For instance, the search engine <NUM> may reside on the remote system <NUM> and/or some of the functionality of the speech recognition system <NUM> may reside on the remote system <NUM>. In one example, the speech recognition system <NUM> may reside on the device <NUM> for performing on-device automated speech recognition (ASR). In another example, the speech recognition system <NUM> resides on the remote system to provide server-side ASR. In yet another example, functionality of the speech recognition system <NUM> is split across the device <NUM> and the server <NUM>. For instance, <FIG> depicts the speech recognition system <NUM> and the search engine <NUM> in dotted boxes to indicate that these components may reside on-device <NUM> or server-side (i.e., at the remote system <NUM>).

In some configurations, different types of speech recognition models reside in different locations (e.g., on-device or remote) depending on the model. Similarly, end-to-end or streaming-based speech recognition models may reside on the device <NUM> due to their space-efficient size while larger, more conventional speech recognition models that are constructed from multiple models (e.g., an acoustic model (AM), a pronunciation model (PM), and a language model (LM)) are server-based models that reside in the remote system <NUM> rather than on-device. In other words, depending on the desired level of speech recognition and/or desired speed to perform speech recognition, speech recognition may reside on-device (i.e., user-side) or remotely (i.e., server-side).

When the user <NUM> conversationally engages with the interface <NUM>, it can be rather inconvenient for the user <NUM> to repeat speaking a same invocation phrase (e.g., hotword) <NUM> for each interaction <NUM> where the user <NUM> desires some feedback (e.g., a response <NUM>) from the interface <NUM>. In other words, requiring the user to speak the same hotword for each of multiple successive queries by the user <NUM> can be quite cumbersome and inconvenient. Yet unfortunately, it is also a waste of computing resources and can be computationally expensive for the device <NUM> to be continuously running speech recognition on any audio data <NUM> received at the microphone <NUM>.

In order to address the inconvenience of requiring the user <NUM> to repeat speaking the hotword <NUM> each time the user <NUM> wants to communicate a new query <NUM> while the user <NUM> is actively having a conversation (or session of interactions <NUM>) with the interface <NUM>, the interface <NUM> may allow the user <NUM> to provide follow-up queries after the interface <NUM> responds to a previous query without requiring the user <NUM> to speak the hotword <NUM>. That is, a response <NUM> to a query may serve as an indication of a microphone trigger event <NUM> (<FIG>) indicating a possible user interaction <NUM> with the voice-enabled device <NUM> through speech, and thereby cause the device <NUM> to instruct the microphone <NUM> to open or remain open for capturing the audio stream <NUM> and providing the captured audio stream <NUM> to the speech recognition system <NUM> to perform speech recognition processing on the audio stream <NUM>. Here, the microphone <NUM> may be open to accept a query from the user <NUM> without requiring the user <NUM> to first speak the hotword <NUM> again to trigger the microphone <NUM> to open to accept the query.

The microphone trigger event <NUM> (also referred to as a trigger event <NUM>) generally refers to the occurrence of an event that indicates that the user <NUM> may possibly interact with the device <NUM> via speech, and therefore requires activation of the microphone <NUM> to capture any speech for speech processing. Here, since the trigger event <NUM> indicates a possible user interaction <NUM>, the trigger event <NUM> may range from a gesture to a recognized user characteristic (e.g., pattern of behavior) to any action by user <NUM> that the device <NUM> may distinguish as a potential interaction <NUM>. For instance, a user <NUM> may have a routine during the work week where, when the user <NUM> enters the kitchen, the user <NUM> queries the device <NUM> for the weather and whether there are any events on the user's calendar. Due to this pattern of behavior, the device <NUM> may recognize the user <NUM> entering kitchen (e.g., hear the movement in the direction of a kitchen entryway) around a particular time and treat the action of the user <NUM> entering the kitchen in the morning as a trigger event <NUM> of a possible user interaction <NUM>. In the case of a gesture, the trigger event <NUM> may be an interaction <NUM> where the user <NUM> raises the device <NUM>, squeezes the device <NUM>, presses a button on the device <NUM>, taps a screen of the device <NUM>, moves hand in a predefined manner, or any other type of pre-programmed gesture to indicate that the user <NUM> may intend to engage in a conversation with the assistant interface <NUM>. Another example of a trigger event <NUM> is when the device <NUM> (e.g., the interface <NUM>) communicates a response <NUM> to a query from the user <NUM>. In other words, when the interface <NUM> relays a response <NUM> to the user <NUM>, the response <NUM> serves as a communication interaction to the user <NUM> on behalf of the device <NUM>; meaning that a follow-up query spoken by the user <NUM> may likely ensue after receiving the response <NUM> or simply occur because the user <NUM> is currently conversing with the device <NUM>. Based on this likelihood, the response <NUM> may be considered a trigger event <NUM> such that the microphone <NUM> either opens or remains open to capture and allow for speech processing of a follow-up query spoken by the user <NUM> after the device <NUM> outputs a response <NUM> without requiring the user <NUM> to prefix the follow-up query with a hotword <NUM>.

To implement the microphone trigger event <NUM> and to provide a conversation between the device <NUM> and the user <NUM> without compromising processing resources, the device <NUM> deploys an interaction analyst <NUM> (also referred to as the analyst <NUM>) that recognizes that the user <NUM> is addressing the assistant interface <NUM> while also maintaining speech recognition throughout the conversation with the user <NUM>. In other words, the analyst <NUM> may provide endpointing functionality by identifying when a conversation between the user <NUM> and the interface <NUM> begins and when it is best to assume that the conversation has ended to endpoint the interaction <NUM> and deactivate speech recognition. Furthermore, in addition to endpointing, the analyst <NUM> may also modify the process of speech recognition by reducing a processing level <NUM> (<FIG> and <FIG>) for speech recognition (i.e., using the speech recognition system <NUM>) depending on the nature of one or more interactions <NUM> by the user <NUM>. Specifically, the analyst <NUM> may be able to instruct the speech recognition system <NUM> to use a decayed level <NUM> of speech recognition over the audio stream <NUM> captured by the microphone <NUM>. For example, when the device <NUM> instructs the microphone <NUM> to open or remain open for an open microphone duration window to capture and provide an audio stream <NUM> to the speech recognition system <NUM> responsive to receiving an indication of a microphone trigger event <NUM>, the analyst <NUM> may decay the level <NUM> of speech processing that the speech recognition system <NUM> performs over the audio stream <NUM> as a function of the open microphone duration window <NUM>. Here, the analyst <NUM> operates in conjunction with the interface <NUM> to control speech recognition and/or speech-related processes.

Referring to <FIG>, when the device <NUM> (e.g., the interface <NUM>) receives an indication of the trigger event <NUM>, the interface <NUM> instructs the microphone <NUM> to open or remain and communicates audio data <NUM> associated with the audio stream <NUM> captured by the microphone <NUM> to the speech recognition system <NUM> for processing. Here, when the interface <NUM> instructs the microphone <NUM> to open, the analyst <NUM> may be configured to designate that the microphone <NUM> continues to stay open after commencement by the trigger event <NUM> for some open microphone duration window <NUM>. For example, the open microphone duration window <NUM> designates a period of time for which the interface <NUM> will communicate the audio stream <NUM> of audio data <NUM> captured by the microphone <NUM> to the speech recognition system <NUM>. Here, the microphone duration window <NUM> refers to a set duration of time that commences upon receipt of the trigger event <NUM> and ceases (e.g., undergoes a microphone closing event) after the set duration of time. In some examples, the analyst <NUM> may instruct the interface <NUM> to extend the microphone duration window <NUM> or to refresh (i.e., renew) the microphone duration window <NUM> when the interface <NUM> receives another trigger event <NUM> (e.g., a subsequent trigger event <NUM>) during the microphone duration window <NUM>. To illustrate, <FIG> and <FIG> depict that the user <NUM> generating a first utterance 12Ua stating "Hey computer, who is the president of France?" and a second utterance 12U, 12Ub of "how old is he?" as a follow-up question to the response <NUM> that the president of France is Emmanuel Macron. Referring specifically to <FIG>, the hotword <NUM> of "hey computer" corresponds to a set phrase that initiates speech recognition for the succeeding portion of the utterance, "who is the president of France?" When the interface <NUM> responds with "Emmanuel Macron," the analyst <NUM> establishes the response <NUM> as a microphone trigger event <NUM> that initiates a first microphone duration window <NUM>, 212a to commence. Here, the window <NUM> may have a duration defined by a start point <NUM> and an end point <NUM> where at the time of the end point <NUM>, the microphone duration window <NUM> expires and the interface <NUM> and/or the analyst <NUM> performs a microphone closing event. Yet in this example, the user <NUM> asks a follow-up question of "how old is he?" prior to the originally designated end point 216a, <NUM> of the first microphone duration window 212a. Due to the follow-up question (i.e., another user interaction <NUM>) of the second utterance 12Ub, the interface <NUM> generates a second response <NUM>, 122b that states that the age of Emmanuel Macron is "<NUM>. " Similar to the first response <NUM>, 122a of "Emmanuel Macron," the second response <NUM>, 122b is a second trigger event <NUM>, 202b that commences a new microphone duration window <NUM>, 212b as a second microphone duration window 212b even though the first microphone duration window 212a may or may not have expired (e.g., <FIG> illustrates that the first window 212a expired prior to the second response 122b). When the interface <NUM> initiates a new microphone duration window <NUM> while a microphone duration window <NUM> is still pending, this may be considered extending the pending window <NUM> or allowing the pending window <NUM> to remain open. In some implementations, when a duration window <NUM> is still pending, a trigger event <NUM> during that window <NUM> may not entirely renew the duration window <NUM> (i.e., begin the duration window anew), but rather extend the pending duration window <NUM> by some designated amount of time. For instance, if the duration window <NUM> is ten seconds and the duration window <NUM> is currently pending, a trigger event <NUM> during this pending duration window <NUM> extends the pending window <NUM> an additional five seconds rather than new additional full ten seconds. In <FIG>, the first window 212a expires prior to the second response 122b such that the second trigger event 202b commences an entirely new duration window <NUM>, 212b at the second start point <NUM>, 214b, which will end (unless, for example, one or more additional trigger events <NUM> occur) at the second end point <NUM>, 216b.

With continued reference to <FIG>, the analyst <NUM> is configured to additionally generate a decay state <NUM> during some portion of the window <NUM>. For instance, by transitioning to a decay state <NUM>, the processing level <NUM> for speech recognition may be modified (e.g., reduced by some amount) during the window <NUM>. In other words, the level <NUM> of speech processing performed by the speech recognition system <NUM> on the audio data <NUM> may be based on a function of the window <NUM>. By basing the level <NUM> of speech processing (e.g., for speech recognition) on the window <NUM>, the amount of speech processing performed over an audio stream <NUM> of audio data <NUM> becomes a function of time. Meaning that, as time passes and it appears less likely that a conversation between the interface <NUM> and the user <NUM> is still ongoing (e.g., the audio data <NUM> does not include user speech or speech not directed toward the device <NUM>), the analyst <NUM> recognizes this decreasing likelihood and reduces speech recognition accordingly. For instance, <FIG> illustrates that during the final half of the first microphone duration window 212a, speech processing transitions to a decay state <NUM> where the level <NUM> of speech processing is reduced. This approach may allow the analyst <NUM> and/or interface <NUM> to reduce the amount of computing resources being used to perform speech processing at the device <NUM>. Similarly, in this example of <FIG>, even though the user <NUM> generates a follow-up question of "how old is he?" during the final third of the first window 212a when the analyst <NUM> has transitioned speech recognition to the decay state <NUM>, the analyst <NUM> determines that the decay state <NUM> for the second window 212b should occur more quickly because speech recognition was already in a decay state <NUM> in the first window <NUM> and it was already somewhat unlikely that the user <NUM> was going to perform a second interaction 12Ub; therefore, the analyst <NUM> determines that the decay state <NUM> should occur for a longer period of time in the second duration window 212b, for example, about <NUM>% of the second duration window 212b. By utilizing the analyst <NUM>, the device <NUM> and/or the interface <NUM> strive to strike a balance between maintaining the ability to perform speech recognition without a hotword <NUM> while also trying to avoid actively performing speech recognition when it is unlikely that the user <NUM> intends to further interact with the assistant interface <NUM>.

Referring to <FIG> and <FIG>, the analyst <NUM> generally includes a window generator <NUM> and a reducer <NUM>. The window generator <NUM> (also referred to as a generator <NUM>) is configured to generate the open microphone duration window <NUM> that dictates a time duration for the microphone <NUM> to be open for capturing and providing an audio stream <NUM> to the speech recognition system <NUM> for processing. Each microphone duration window <NUM> includes a start point <NUM> designating a time for the beginning of the microphone duration window <NUM> when the audio stream <NUM> received at the microphone <NUM> begins to be sent to the speech recognition system <NUM> for processing and an end point <NUM> designating a time when the open microphone duration window <NUM> ends and the audio stream <NUM> thereafter is no longer communicated to the speech recognition system <NUM> for processing. The window generator <NUM> initially generates the open microphone duration window <NUM> when it is determined that the interface <NUM> receives a trigger event <NUM> from the user <NUM>.

In some implementations, the window generator <NUM> is configured to generate different sized windows <NUM> (i.e., windows <NUM> having different lengths of time) based on a configuration of the analyst <NUM> or intelligently based on the conditions occurring at the microphone <NUM>. For instance, an administrator of the analyst <NUM> sets a default duration for a window <NUM> generated by the window generator <NUM> (e.g., each window <NUM> is ten seconds long). In contrast, the generator <NUM> may recognize patterns of behavior by a user <NUM> or aspects/features of an utterance 12U by the user <NUM> and intelligently generate a window <NUM> having a size to match or to correspond to these recognized characteristics. For instance, it may be common for a particular user <NUM> to engage in multiple questions after each time that the particular user <NUM> generates the trigger <NUM> to start a conversation with the interface <NUM>. Here, a speech processing system associated with the device <NUM> may identify the identity of the user <NUM> and based on this identify, the generator <NUM> generates a window <NUM> with a size that corresponds or caters to the interaction frequency for that user <NUM> during the interaction session (i.e., conversation with the interface <NUM>). For instance, instead of the generator <NUM> generating a default window <NUM> having a five second duration, the generator <NUM> generates a window <NUM> having a ten second duration because the user <NUM> that submitted the hotword <NUM> tends to generate a higher frequency of interactions <NUM> with the interface <NUM>. On the other hand, if the user <NUM> that submitted the hotword <NUM> tends to only ask a single query whenever he or she interacts with the interface <NUM>, the generator <NUM> may receive this behavior information about the user <NUM> and shorten the default window <NUM> from five seconds to a custom window <NUM> of three seconds.

The generator <NUM> may also generate a custom open microphone duration window <NUM> based on aspects or features of an utterance 12U or a response <NUM> to the utterance 12U. As a basic example, the user <NUM> speaks an utterance 12U that states "hey computer, play the new Smashing Pumpkins album, Cyr. " Yet when the interface <NUM> attempts to respond to this command, the interface <NUM> determines that the new Smashing Pumpkins album, Cyr, is being released later in the month and provides a response <NUM> indicating that the album is not currently available. The response <NUM> may further ask if the user <NUM> wants to listen to something else. Here, following the response <NUM> by the interface <NUM>, the generator <NUM> may generate a larger window <NUM> (i.e., a window <NUM> lasting a longer duration of time) or may extend an initially generated window <NUM> automatically based on the fact that the interface <NUM> and/or device <NUM> determines there is a high likelihood that there is a follow-up interaction <NUM> by the user <NUM> when the interface <NUM> generates a response <NUM> that the requested album is not available. In other words, the interface <NUM> and/or device <NUM> intelligently recognizes it is likely the user <NUM> will submit another music request following the response <NUM> due to the results of the command by the user <NUM>.

As <FIG> illustrates, the generator <NUM> not only generates a window <NUM> when the trigger event <NUM> is initially received (e.g., when the interface <NUM> communicates the response <NUM>), but also may generate a new window <NUM> or extend a window <NUM> that is currently open when an trigger event <NUM> is received by the interface <NUM> during an open window <NUM>. By generating a new window <NUM> when a trigger event <NUM> is received during an open window <NUM> or extending an open window <NUM>, the generator <NUM> functions to continue keeping the window <NUM> open when the conversation between the user <NUM> and the interface <NUM> is ongoing.

In some examples, such as <FIG>, when the generator <NUM> extends a window <NUM> or generates a new window <NUM> in an ongoing conversation, the generator <NUM> is configured to discount the duration of the window <NUM>. For instance, the generator <NUM> is configured to generate subsequent windows <NUM> of shorter duration after each subsequent interaction <NUM> by the user <NUM>. This approach may account for the fact that a second or follow-up interaction <NUM> has a first probability of occurring after an initial interaction <NUM>, but thereafter the probability decreases that additional interactions <NUM> will occur. For instance, after an initial query in the first utterance 12Ua, a follow-up query in the second utterance 12Ub may occur about <NUM>% of the time, but a third utterance 12U after the follow-up query in the second utterance 12Ub only occurs about <NUM>% of the time during an interaction session between the user <NUM> and the interface <NUM>. Based on this pattern, the generator <NUM> may decrease the size of the window <NUM> or how long an open window <NUM> is extended as a function of this probability that another trigger event <NUM> (i.e., possible interaction <NUM>) occurs. For example, <FIG> illustrates three trigger events <NUM>, 202a-c where each trigger event <NUM> corresponds to a subsequent response <NUM> (e.g., three responses 122a-c) from the interface <NUM> such that the generator <NUM> generates a first window 212a for the first trigger event 202a followed by a second window 212b for the second trigger window 202b that occurs after the first trigger window 202a and then followed by a third window 212c for the third trigger event 202c that occurs after the second trigger event 202b. In this example, the generator <NUM> shortens the generated window <NUM> such that the third window 212c has a shorter duration than the second window 212b, which has a shorter duration than the first window 212a. Additionally or alternatively, the generator <NUM> may analyze the audio data <NUM> of the audio stream <NUM> to determine whether the voice activity level in the audio data <NUM> provides any indication that a trigger event <NUM> is likely to occur. With this information, the generator <NUM> may modify the size of the current open window <NUM> or modify the size of any subsequently generated/extended window <NUM>.

The reducer <NUM> is configured to designate the processing level <NUM> of speech recognition that should be performed on the audio stream <NUM> of audio data <NUM>. The processing level <NUM> may be based on several variables that include, but are not limited to, the type of speech recognition model that performs the speech recognition for the speech recognition system <NUM>, the location where speech recognition occurs, the speech recognition parameters used to perform speech recognition, whether the speech model is designated to operate at full capability or in some lesser degree of capability, etc. The processing level <NUM> may generally refer to the amount of computing resources (e.g., local resource such as the data processing hardware and memory hardware or remote resources) that are being dedicated or consumed by speech processing, such as speech recognition, at any given time. This means that a first processing level <NUM> in a first state is less than a second processing level <NUM> in a second state when, for example, the amount of computing resources or computing power being dedicated to speech processing in the first state is less than in the second state.

In some examples, when speech recognition occurs at a reduced processing level <NUM> (e.g., when compared to maximum processing capabilities), the reduced processing level <NUM> may function as a first pass to recognize speech, but then result in a second pass with a processing level <NUM> higher than the first pass. For instance, when the speech recognition system <NUM> is operating at a reduced processing level <NUM>, the speech recognition system <NUM> identifies a low confidence speech recognition result (e.g., low confidence hypothesis) that the audio data <NUM> contains an utterance 12U to command or query the interface <NUM>. Due to this low confidence speech recognition result, the reducer <NUM> may increase the processing level <NUM> such that it may be determined at a higher processing level <NUM> whether the low confidence speech recognition result actually corresponds to a higher confidence speech recognition result (e.g., high confidence hypothesis) that the audio data <NUM> contains an utterance 12U to command or query the interface <NUM>. In some examples, a low confidence speech recognition result is a speech recognition result that fails to satisfy a confidence threshold during speech recognition. In other words, the reducer <NUM> may change the processing level <NUM> based not only on the variables mentioned above, but also based on the results obtained during speech recognition at a particular processing level <NUM>.

In some implementations, the reducer <NUM> designates the processing level <NUM> as a function of the window <NUM>. That is, the reducer <NUM> is able to reduce the processing level <NUM> for speech recognition when an open microphone duration window <NUM> exists. For instance, when the window <NUM> corresponds to a duration of ten seconds (e.g., as shown in <FIG>), during the first five seconds of the window <NUM>, the reducer <NUM> instructs the speech recognition system <NUM> to operate at full power/processing (e.g., a first processing level <NUM>, 222a) for speech recognition. After these five seconds occur at full processing, the speech processing transitions to a decay state <NUM> where the processing level <NUM> is reduced to some degree less than full processing. As an example, from five seconds to seven seconds into the duration of the window <NUM>, the reducer <NUM> instructs the speech recognition system <NUM> to perform at a second processing level <NUM>, 222b that corresponds to <NUM>% of full processing for the speech recognition system <NUM>. Then, after the seventh second of the duration and until the end of the duration for the window <NUM>, the reducer <NUM> instructs the speech recognition system <NUM> to perform at a third processing level <NUM>, 222c corresponding to <NUM>% of full processing for the speech recognition system <NUM>. Thus, the reducer <NUM> controls the processing level <NUM> of the speech recognition system <NUM> during the open microphone duration window <NUM>.

In some implementations, the generator <NUM> and the reducer <NUM> work in conjunction such that the duration of the window <NUM> depends on the decay state <NUM>. The generator <NUM> may not generate an end point <NUM> at a designated time, but rather allow the reducer <NUM> to reduce the processing level <NUM> to a level that has the same effect as closing the microphone <NUM>. For example, after the third processing level 222c corresponding to <NUM>% of full processing for the speech recognition system <NUM>, the reducer <NUM> then closes the microphone <NUM> because another <NUM>% reduction in processing reduces the processing level <NUM> to <NUM>% of full processing for the speech recognition system <NUM> (i.e., no processing or "closed"). This particular example is a tiered approach that steps the processing level <NUM> down in discrete increments, but other types of decay are possible to result in a closed microphone <NUM>. For instance, the processing level <NUM> may decay linearly at any point during the open microphone duration window <NUM>. By allowing the reducer <NUM> to reduce the processing level <NUM> until the microphone <NUM> is closed, this technique may advance the continuous decaying of speech processing (e.g., when no interaction <NUM> is occurring and the microphone <NUM> is open).

In some configurations, such as <FIG>, the processing level <NUM> within a window <NUM> is based on a time when the interface <NUM> received the last trigger event <NUM> by the user <NUM>. In these configurations, the reducer <NUM> may determine whether a time period <NUM> from when the interface <NUM> received the last trigger event <NUM> to the current time satisfies a time threshold <NUM>. When the time period <NUM> satisfies a time threshold <NUM> (i.e., the no interaction <NUM> has occurred for a threshold amount of time), the reducer <NUM> may generate a processing level <NUM> at the current time. For instance, if the interface <NUM> sent a response <NUM> to an utterance 12U by the user <NUM>, the window <NUM> began when the interface <NUM> communicated the response <NUM>. When the time threshold <NUM> is set to five seconds, the reducer <NUM> determines whether five seconds have elapsed from when the interface <NUM> communicated the response <NUM>. When the reducer <NUM> determines that five seconds have elapsed, the reducer <NUM> may then designate some processing level <NUM> for speech recognition starting, for example, at the current time when the reducer <NUM> determined that the five seconds had elapsed or some particular time thereafter.

In some configurations, the processing level <NUM> changes by adjusting one or more parameters of the speech recognition system <NUM>. In one approach to change the processing level <NUM> of speech recognition at the speech recognition system <NUM>, speech recognition is changed as to the location where it occurs. The location of speech recognition may change from occurring server-side (i.e., remotely) to on-device (i.e., locally). In other words, a first processing level 222a corresponds to remote speech recognition using a server-based speech recognition model while a second processing level 222b corresponds to local speech recognition occurring on-device. When a speech recognition system <NUM> is hosted "on-device," the device <NUM> receives the audio data <NUM> and uses its processor(s) (e.g., data processing hardware <NUM> and memory hardware <NUM>) to execute the functionality of the speech recognition system <NUM>. Speech recognition that uses a server-based model may be considered to have a greater processing level <NUM> than an on-device speech recognition model because the server-based model may leverage a greater number of remote processing resources (e.g., along with other costs such as bandwidth and transmission overhead). With a greater amount of processing, the server-based model may be potentially larger in size than an on-device model and/or perform decoding using a larger search graph than an on-device model. For instance, a server-based speech recognition model may leverage multiple, larger models (e.g., an acoustic model (PM), a pronunciation model (PM), and a language model (PM)) that may be particularly trained for a dedicated speech recognition purpose whereas an on-device model often has to consolidate these different models into a smaller package to operate effectively and space efficiently on the finite processing resources of the device <NUM>. Therefore, when the reducer <NUM> reduces the processing level <NUM> to a decay state <NUM>, the reducer <NUM> may change speech recognition from occurring remotely using a server-based model to locally using an on-device model.

In some situations, there may be more than one device <NUM> nearby a user <NUM> who is generating an interaction <NUM> such as a spoken utterance 12U. When multiple devices <NUM> are located in the vicinity of the user <NUM>, each device <NUM> may be able to perform some aspect of speech recognition. Since multiple devices <NUM> performing speech recognition on the same spoken utterance 12U may be duplicative, the reducer <NUM> may reduce the processing level <NUM> at a particular device <NUM> by closing the microphone <NUM> for that device <NUM> knowing that another device <NUM> is handling or configured to handle the speech recognition. To illustrate, when the user <NUM> has a mobile device and a smart watch, both devices <NUM> may be able to perform speech recognition. Here, the reducer <NUM> may close the microphone <NUM> for speech recognition at the mobile device to conserve the mobile device's processing resources for the wide range of other computing tasks that the mobile device may need to perform. For instance, the user <NUM> may prefer to conserve battery power for his or her mobile device rather than his or her smart watch. In some examples, when there are multiple devices, the reducer <NUM> may try to determine characteristics (e.g., current processor consumption, current battery life, etc.) of each device <NUM> to identify which device <NUM> is most optimal to have a microphone <NUM> that stays open and which device(s) are most optimal to have their microphones <NUM> closed.

Besides there being a processing level difference between a remote speech recognition system <NUM> and an on-device speech recognition system <NUM>, there may be different versions of the on-device speech recognition model or the server-side speech recognition model. With different versions, the reducer <NUM> may change the processing level <NUM> by changing the model or the version of the model being used for speech recognition. Broadly speaking, a model may have a large version with a high processing level <NUM>, a medium version with a moderate processing level <NUM>, and a small version with a low processing level <NUM>. In this sense, if the reducer <NUM> wants to reduce the processing level <NUM>, the reducer <NUM> may transition the speech recognition from being performed at a first version of the model to a second version of the model with a lower processing level <NUM> than the first version of the model. In addition to changing between versions of either an on-device model or a server-side model, the reducer <NUM> may also change from one version of a server-side model to a particular version of an on-device model. By having models and versions of these models, the reducer <NUM> has a greater number of processing levels <NUM> at its disposal in order to decay the speech recognition processing level <NUM> during an open microphone window <NUM>.

In some implementations, versions of the on-device speech recognition model have different processing demands such that reducer <NUM> may designate such versions for different processing levels <NUM>. Some examples of on-device speech recognition models include sequence-to-sequence models such as recurrent neural network transducers (RNN-T) models, listen-attend-spell (LAS) models, neural transducer models, monotonic alignment models, recurrent neural alignment (RNA) models, etc. There may also be on-device models that are hybrids of these models, such as a two-pass model that combines an RNN-T model and a LAS model. With these different versions of on-device models, the reducer <NUM> may rank or identify the processing requirements of each of these versions in order to generate different processing levels <NUM>. For instance, a two-pass model includes a first-pass of an RNN-T network followed by a second-pass of a LAS network. Since this two-pass model includes multiple networks, the reducer <NUM> may designate the two-pass model as a large on-device model that has a relatively high processing level <NUM> for speech recognition. To reduce the processing level <NUM> of speech recognition from the processing level <NUM> of a two-pass model, the reducer <NUM> may change from the two-pass model to an LAS model. Here, an LAS model is an attention-based model that performs attention during its decoding process to generate a character sequence that forms the transcription <NUM>. Generally speaking, attention-based approaches tend to be more computationally intensive to focus attention on particular features for a given speech input. For comparison, an RNN-T model does not employ an attention mechanism and also performs its beam search through a single neural network instead of a large decoder graph; thus, the RNN-T model may be more compact than an LAS model and less computationally expensive. For these reasons, the reducer <NUM> may change between the two-pass model, the LAS model, and the RNN-T model for speech recognition in order to reduce processing levels <NUM>. That is, the two-pass model, by employing both the RNN-T network as a first pass and rescoring the first pass with a LAS network as a second pass, has a higher processing level <NUM> than either of the LAS model or the RNN-T models alone while the LAS model, as an attention-based model, has a higher processing level <NUM> than the RNN-T model. By identifying the processing requirements of speech recognition for different versions of on-device speech recognition models, the reducer <NUM> is able to decay speech processing (or increase processing) by switching among any of the different versions of on-device models. Moreover, when the reducer <NUM> combines the processing level options for on-device models with processing level options for server-side models, the decay of speech processing by the reducer <NUM> has a multitude of potential processing level gradations.

To expand the potential number of processing level gradations further, the reducer <NUM> may be configured to modify speech processing steps or speech processing parameters for a given model to change the processing level of that particular model. For instance, a particular model includes one or more layers of a neural network (e.g., a recurrent neural network with layers of long-short-term memory (LSTM)). In some examples, an output layer may receive information from past states (backwards) and future states (forward) in order to generate its output. When a layer receives backwards and forwards states, the layer is considered to be bidirectional. In some configurations, the reducer <NUM> is configured to modify the processing steps of a model to change a speech recognition model from operating bi-directionally (i.e., forward and backward) to simply uni-directionally (e.g., forward). Additionally or alternatively, the reducer <NUM> may reduce the number of neural network layers that a model uses to perform speech recognition in order to change the processing level <NUM> of a particular model.

The reducer <NUM> may also reduce the processing level <NUM> for a speech recognition model by altering beam search parameters (or other pruning/search mode parameters) for the model. Generally speaking, a beam search includes a beam size or beam width parameter that specifies how many of the best potential solutions (e.g., hypotheses or candidates) to evaluate in order to generate a speech recognition result. The beam search process, therefore, performs a type of pruning of potential solutions to reduce the number of solutions evaluated to form the speech recognition result. That is, the beam search process can limit the computation involved by using a limited number of active beams to search for the most likely sequence of words spoken in an utterance 12U to produce the speech recognition result (e.g., the transcription <NUM> of the utterance 12U). Here, the reducer <NUM> may adjust the beam size to reduce the number of best potential solutions to evaluate, which, in turn, reduces the amount of computation involved in the beam search process. For instance, the reducer <NUM> changes the beam size from five to a beam size of two to have the speech recognition model evaluate <NUM>-best candidates rather than <NUM>-best candidates.

In some examples, the reducer <NUM> performs quantization or sparsification on one or more parameters for a speech recognition model in order to generate a lower processing level <NUM> for the model. When generating speech recognition results, a speech recognition model typically generates a large number of weights. For example, the speech recognition model weights different speech parameters and/or speech-related features in order to output a speech recognition result (e.g., the transcription <NUM>). Due to this large number of weights, the reducer <NUM> may discretize the values of these weights by performing quantization. For instance, a quantization process converts floating point weights to weights represented as fixed point integers. Although this quantization process loses some information or quality, it allows the resources processing these quantized parameters to use less memory, and may allow for more efficient operations to be performed (e.g., multiplications) on certain hardware.

In a similar respect, sparsification also aims to reduce the amount of processing to run the model. Here, sparsification refers to a process of removing redundant parameters or features in a speech recognition model in order to focus on more relevant features. For example, while determining a speech recognition result, a speech model may determine probabilities for all speech-related features (e.g., characters, symbols, or words) even though not all speech-related features are relevant to a certain speech input. By using sparsification, the model may spend less computational resources by determining probabilities for speech-related features that are relevant to the input instead of all speech-related features; allowing the sparsification process to ignore speech-related features that are not relevant to the input.

Optionally, the reducer <NUM> may generate a lower processing level <NUM> for speech recognition by determining the context of an interaction <NUM> (e.g., spoken utterance 12U) that originally generated or led to the open microphone duration window <NUM>. Once the reducer <NUM> identifies the context, the reducer <NUM> may use the context to reduce the processing level <NUM> for a speech recognition system <NUM> by biasing a speech recognition result based on the context. In some implementations, the reducer <NUM> biases the speech recognition result based on context by limiting the speech recognition system <NUM> to vocabulary related to the context. As an example, the user <NUM> may ask the interface <NUM>, "how do you tie a prusik hitch?" From this question, the reducer <NUM> determines that prusik hitches are predominantly used in mountaineering or rock climbing. In other words, the reducer <NUM> identifies that the context of the interaction <NUM> is mountaineering. In this example, when the reducer <NUM> proceeds to reduce the processing level <NUM> for speech recognition, the reducer <NUM> limits the speech recognition output to vocabulary related to mountaineering. Therefore, if the user <NUM> subsequently asks a follow-up question about the Appalachian Mountains, the speech recognition system <NUM> may generate possible speech recognition results that include the term "Application" and "Appalachian," but since "Appalachian" is related to the context of "Mountaineering," the reducer <NUM> ensures that speech recognition system <NUM> is biased towards the term "Appalachian. " For instance, the reducer <NUM> instructs the speech recognition system <NUM> to increase the probability score of potential results that relate to the identified context (e.g., mountaineering). In other words, the speech recognition system <NUM> increases the probability score for potential results with vocabulary related to mountaineering.

When the speech recognition system <NUM> is operating on the device <NUM>, the speech recognition system <NUM> may generally use a system on chip-based (SOC-based) processor to perform speech recognition. A system on chip (SOC) processor refers to a general processor, a signal processor, as well as additional peripherals. The reducer <NUM> may generate a reduced processing level <NUM> when speech recognition uses SOC-based processing by instructing the speech recognition system <NUM> to change from SOC-based processing to a digital signal processor (DSP). Here, this change results in a lower processing level <NUM> because DSPs tend to consume lower power and memory than SOC-based processing.

As the reducer <NUM> decays the processing level <NUM> for speech recognition, it may be advantageous to provide some degree of indication to the user <NUM> that the decay is occurring. To provide this indication, a graphical user interface (GUI) associated with the device <NUM> may include a graphical indicator to indicate the current processing level <NUM> of the speech recognition system <NUM>. In some examples, the graphical indicator has a brightness level that is configured to fade proportionally with the decay of the processing level <NUM>. For instance, a screen of the device <NUM> includes a GUI that shows a red microphone dot to indicate that the microphone <NUM> is open (i.e., listening for interactions <NUM>) which gradually fades out as the reducer <NUM> decays the processing level <NUM> for speech recognition. Here, when the microphone <NUM> closes, the red microphone dot would extinguish. Additionally or alternatively, an indicator that indicates that the microphone <NUM> is open and/or the degree of decay of the processing level <NUM> for speech recognition may be a hardware indicator, such as a light on the device <NUM> (e.g., a light emitting diode (LED)). For instance, an LED fades to off or blinks at a decreasing rate (e.g., more and more slowly) until extinguishing as the processing level <NUM> decreases until the microphone <NUM> is closed.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of decaying speech processing. At operation <NUM>, the method <NUM> receives, at a voice-enabled device <NUM>, an indication of a microphone trigger event <NUM> indicating a possible user interaction with the voice-enabled device <NUM> through speech where the voice-enabled device <NUM> has a microphone <NUM> that, when open, is configured to capture speech for recognition by an automated speech recognition (ASR) system <NUM>. Operation <NUM> includes two sub-operations <NUM>, 304a-b that occur in response to receiving the indication of the microphone trigger event <NUM>. At operations 304a, the method <NUM> instructs the microphone <NUM> to open or remain open for an open microphone duration window <NUM> to capture an audio stream <NUM> in an environment of the voice-enabled device <NUM>. At operation 304b, the method <NUM> provides the audio stream <NUM> captured by the open microphone <NUM> to the ASR system <NUM> to perform ASR processing over the audio stream <NUM>. Operation <NUM> includes two sub-operations <NUM>, 306a-b that occur while the ASR system <NUM> is performing the ASR processing over the audio stream <NUM> captured by the open microphone <NUM>. At operation <NUM>, the method <NUM> decays a level <NUM> of the ASR processing that the ASR system <NUM> performs over the audio stream <NUM> based on a function of the open microphone duration window <NUM>. At operation 306b, the method <NUM> instructs the ASR system <NUM> to use the decayed level <NUM>, <NUM> of the ASR processing over the audio stream <NUM> captured by the open microphone <NUM>.

<FIG> is a schematic view of an example computing device <NUM> that may be used to implement the systems (e.g., the device <NUM>, the interface <NUM>, the remote system <NUM>, the speech recognition system <NUM>, the search engine <NUM>, and/or the analyst <NUM>) and methods (e.g., the method <NUM>) described in this document.

For example, it may be implemented as a standard server 400a or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>) of a voice-enabled device (<NUM>), an indication of a microphone trigger event (<NUM>) indicating a possible user interaction (<NUM>) with the voice-enabled device (<NUM>) through speech, the voice-enabled device (<NUM>) having a microphone (<NUM>) that, when open, is configured to capture speech for recognition by an automated speech recognition, ASR, system (<NUM>);
in response (<NUM>) to receiving the indication of the microphone trigger event (<NUM>):
instructing, by the data processing hardware (<NUM>), the microphone (<NUM>) to open or remain open for an open microphone duration window (<NUM>) to capture an audio stream (<NUM>) in an environment of the voice-enabled device (<NUM>); and
providing, by the data processing hardware (<NUM>), the audio stream (<NUM>) captured by the open microphone (<NUM>) to the ASR system (<NUM>) to perform ASR processing over the audio stream (<NUM>); and
while the ASR system (<NUM>) is performing the ASR processing over the audio stream (<NUM>) captured by the open microphone (<NUM>):
decaying, by the data processing hardware (<NUM>), a level of the ASR processing that the ASR system (<NUM>) performs over the audio stream (<NUM>) based on a function of the open microphone duration window (<NUM>); and
instructing, by the data processing hardware (<NUM>), the ASR system (<NUM>) to use the decayed level (<NUM>, <NUM>) of the ASR processing over the audio stream (<NUM>) captured by the open microphone (<NUM>).