Patent Description:
Users frequently interact with voice-enabled devices, such as smart phones, smart watches, and smart speakers, through digital assistant interfaces. These digital assistant interfaces enable users to complete tasks and obtain answers to questions they have all through natural, conversational interactions. Ideally, when conversing with a digital assistant interface, a user should be able to communicate as if the user were talking to another person, via spoken requests directed toward their voice-enabled device running the digital assistant interface. The digital assistant interface will provide these spoken requests to an automated speech recognizer to process and recognize the spoken request so that an action can be performed.

Exemplary implementations of voice-enabled devices are disclosed in <CIT>, <CIT> and <CIT>.

Digital assistant interfaces are moving onto mobile devices in which much of the speech recognition processing occurs on-device without incurring the added latency to connect to a server via a network and send audio data to the server to perform speech recognition using cloud-based speech recognition servers. In addition to improving latency, other benefits of on-device speech recognition include improved reliability and privacy. As such, digital assistant interfaces are becoming deeply integrated with various applications and operating systems running on mobile devices, thereby enabling a user to control their mobile device solely using their voice. However, these applications installed on user devices may themselves be slow, unreliable, or require network access to servers, thereby throttling the benefits that the on-device processing capabilities the digital assistant interface affords and can lead to a sluggish user experience.

One aspect of the disclosure provides a method for streaming action fulfillment. The method is defined by claim <NUM>.

Implementations of the disclosure may include the optional features of the dependent claims.

Another aspect of the disclosure provides a system for streaming action fulfillment. The system is defined by claim <NUM>.

Implementations of the disclosure may include the optional features according to the dependent claims.

Conventionally, digital assistant interfaces that enable users to complete tasks and obtain answers to questions they have through natural, conversational interactions have required the processing of servers to support speech recognition and language understanding models. These server-based models were not suitable for execution on device due to their size, which could exceed the available storage or memory on the device. However, recent advancements in recurrent neural networks have enabled the development of new speech recognition and language understanding models of drastically reduced size (e.g., less than half a gigabyte) suitable for storage and processing on-device. As such, digital assistant interfaces are moving onto mobile devices in which much of the speech recognition processing occurs on-device without incurring the added latency to connect to a server via a network and send audio data to the server to perform speech recognition using cloud-based speech recognition servers. User experience is drastically enhanced since transcriptions of speech can occur in real-time and without a network connection. In addition to improving latency, other benefits of on-device speech recognition include improved reliability and privacy. As such, digital assistant interfaces are becoming deeply integrated with various applications and operating systems running on mobile devices, thereby enabling a user to control their mobile device solely using their voice. For example, a user could multi-task across multiple applications running on the device, such as creating a calendar invite, finding and sharing a photo with friends, or dictating an email. However, these applications installed on user devices may themselves be slow, unreliable, or require network access to servers, thereby throttling the benefits that the on-device digital assistant interface affords and can lead to a sluggish user experience.

Generally, in speech recognition systems, speech endpointing is the process of determining which part of incoming audio contains speech by determining a beginning and an end of an utterance. The part of incoming audio corresponding to speech is provided to a speech recognizer to obtain a speech recognition result or a transcript of the audio. User perceived latency of speech recognition is the time from when the user stops speaking until the speech recognition result or transcript is output, often output for display on a screen of a user device. The recent advancements made to run speech recognition models on-device have allowed for real-time speech recognition results (e.g., streaming transcription) to display as the user speaks and before endpointing determines the end of the utterance.

Yet, while the user perceived latency of the actual speech recognition has been improved by processing speech on-device, a user perceived latency of fulfillment of user queries/commands still exists since an endpoint identifying the end of the utterance has to be made before the actual fulfillment can take place. For example, a command for "Play rock music playlist on YouTube Music" spoken by a user to a digital assistant interface executing on the user's phone requires performance of multiple sub-actions in order to fulfill the complete action of streaming the user's rock music playlist from the YouTube Music service for audible output through an acoustic speaker. These sub-actions include launching the YouTube Music application on the user's phone, accessing a search box in the YouTube Music to input a query for the rock music playlist or switching to a playlists tab in the YouTube Music to select the rock music playlist, and finally executing audible playback of the rock music playlist from the YouTube Music application. In this example, the system has to wait for the end pointer to endpoint the end of the utterance and obtain the final speech recognition result before initiating the first sub-action that needs to be fulfilled, e.g., launching the YouTube Music application.

Implementations herein are directed to generating intermediate speech recognition results from a user utterance commanding performance of an action and using the intermediate speech recognition results to perform multiple sub-actions related to the final action while the user is still speaking the utterance. The leveraging of available intermediate speech recognition results to identify and to perform sub-actions related to a final action before the user finishes speaking drastically reduces user perceived latency since waiting for a final endpoint is not required before initiating performance of a first sub-action in a sequence of sub-actions that need to be performed in order to fulfill the final action.

Yet, intermediate recognition results of "Play", "rock music", and "playlist" occur before the final endpoint and are available to identify possible sub-actions related to the query. For instance, the intermediate recognition result of "Play" can indicate that the user wants to play media content, which could include video content such as television programming, movies, or video clips, or audio content from one of multiple music applications on the user device. Next, the intermediate recognition result of "rock music" provides context that narrows the type of media content the user wants to play to audio content. Now, the possible available sub-actions are opening either a streaming radio application, a Spotify application, or a YouTube application on the user's phone that all can potentially output rock music. For instance, the streaming radio, Spotify, and YouTube applications all include their own rock music channels, while the Spotify and YouTube applications each include respective rock music playlists for the user. Once the intermediate recognition result of "playlist" becomes available, the available possible sub-actions are now more specific to indicate that a rock music playlist needs to be accessed on the Spotify application or the YouTube application.

Referring to <FIG>, in some implementations, the speech environment <NUM> includes a user <NUM> speaking an utterance <NUM> directed toward a voice-enabled device <NUM> (also referred to as a device <NUM> or a user device <NUM>) executing a digital assistant interface <NUM>. Here, the utterance <NUM> spoken by the user <NUM> may be captured by the device <NUM> in streaming audio <NUM> and may correspond to a query to perform an action <NUM>, and more specifically, a query <NUM> for the digital assistant interface <NUM> to perform an action <NUM>. The user <NUM> may prefix the query <NUM> with a hotword <NUM> (e.g., invocation phrase) to trigger the device <NUM> from a sleep or hibernation state when the hotword <NUM> is detected in the streaming audio <NUM> by a hotword detector running on the device <NUM> while in the sleep or hibernation state. The action <NUM> may also be referred to as an operation or task. In this sense, the user <NUM> may have conversational interactions with the digital assistant interface <NUM> executing on the voice-enabled device <NUM> to perform computing activities or to find answers to questions.

The device <NUM> may correspond to any computing device associated with the user <NUM> and capable of capturing audio from the environment <NUM>. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music players, casting devices, smart appliances (e.g., smart televisions) and internet of things (IoT) devices, remote controls, smart speakers, etc. The device <NUM> includes data processing hardware 112d and memory hardware <NUM> in communication with the data processing hardware 112d and storing instructions, that when executed by the data processing hardware 112d, cause the data processing hardware 112d to perform one or more operations related to speech processing.

The device <NUM> further includes an audio subsystem with an audio capturing device (e.g., an array of one or more microphones) <NUM> for capturing and converting audio within the speech environment <NUM> into electrical signals (e.g., audio data <NUM> (<FIG>). While the device <NUM> implements the audio capturing device <NUM> (also referred to generally as a microphone <NUM>) in the example shown, the audio capturing device <NUM> may not physically reside on the device <NUM>, but be in communication with the audio subsystem (e.g., peripherals of the device <NUM>). For example, the device <NUM> may correspond to a vehicle infotainment system that leverages an array of microphones positioned throughout the vehicle.

The device <NUM> may also include a display <NUM> to display graphical user interface (GUI) elements (e.g., windows, screens, icons, menus, etc.) and/or graphical content. For example, the device <NUM> may load or launch applications <NUM>, 118a-n that generate GUI elements or other graphical content for the display <NUM>. Moreover, the elements generated in the display <NUM> may be selectable by the user <NUM> and also serve to provide some form of visual feedback to processing activities and/or operations occurring on the device <NUM>. Furthermore, since the device <NUM> is a voice-enabled device <NUM>, the user <NUM> may interact with elements generated on the display <NUM> using various voice commands. For instance, the display <NUM> may depict a menu of options for a particular application <NUM> and the user <NUM> may use the interface <NUM> to select an option through speech.

In some implementations, the device <NUM> communicates via a network <NUM> with a remote system <NUM>. The remote system <NUM> may include remote resources <NUM>, such as remote data processing hardware <NUM> (e.g., remote servers or CPUs) and/or remote memory hardware <NUM> (e.g., remote databases or other storage hardware). The device <NUM> may utilize the remote resources <NUM> to perform various functionality related to speech processing. For instance, some portion of the assistant interface <NUM> may reside on the remote system <NUM>. In one example, a speech recognizer <NUM> executes locally on the device <NUM> to provide on-device automated speech recognition (ASR). In another example, the speech recognizer <NUM> resides on the remote system <NUM> to provide server-side ASR for the assistant interface <NUM>. In yet another example, functionality of the speech recognizer <NUM> is split across the device <NUM> and the server <NUM> (e.g., the device <NUM> and the server <NUM> perform parallel ASR processing).

The speech-enabled interface <NUM> may field the query/command <NUM> conveyed in the spoken utterance <NUM> captured in streaming audio <NUM> by the device <NUM>. The speech-enabled interface <NUM> (also referred to as the interface <NUM> or the assistant interface <NUM>) generally facilitates receiving audio data <NUM> corresponding to an utterance <NUM> captured by the audio capturing device <NUM>, coordinating speech processing on the audio data <NUM>, performing semantic interpretation on the audio data <NUM> to identify a query <NUM> to perform an action <NUM>, and performing a sequence of sub-actions <NUM>, 26a-n in order to fulfill the action <NUM>. The interface <NUM> may execute on the data processing hardware 112d of the device <NUM>. When the microphone <NUM> of the device <NUM> captures an utterance <NUM> in streaming audio <NUM> and converts the audio into audio data <NUM>, the audio data <NUM> corresponding to the utterance <NUM> is relayed to the interface <NUM> such that the interface <NUM> can perform speech recognition and semantic interpretation to identify and ultimately fulfill a query <NUM> conveyed in the spoken utterance <NUM>. Although traditionally, fulfillment systems have had to wait for an endpointer (e.g., part of the speech recognizer or a separate component) to endpoint the end of an utterance <NUM> to obtain the final speech recognition result before initiating performance of an action <NUM> specified by the query/command <NUM>, here, the interface <NUM> fulfills sub-actions <NUM> associated with the final action <NUM> as partial speech recognition results (e.g., partial hypotheses) are generated and before the user <NUM> is finished speaking the query <NUM> in the utterance <NUM>. By leveraging partial speech recognition results, the interface <NUM> may provide the user <NUM> with real-time or near real-time feedback as to sub-actions <NUM> the interface <NUM> is performing in order to fulfill an action <NUM> specified by the query <NUM> once the utterance <NUM> is endpointed. In other words, the interface <NUM> strives to perform sub-actions <NUM> interpreted from intermediate speech recognition results for the audio data <NUM> while the user <NUM> is actively speaking the utterance <NUM>.

In some implementations, to perform a sequence of sub-actions <NUM> in order to fulfill an action <NUM>, the interface <NUM> interfaces with a plurality of applications <NUM>, 118an on the device <NUM> or accessible to the device <NUM>. An application <NUM> generally refers to any application <NUM> that is configured to run on the device <NUM>. Some types of applications <NUM> include media applications (e.g., video streaming applications, audio streaming applications, media player applications, media gallery applications, etc.), word processing applications, navigation applications, social media applications, communication applications (e.g., messaging applications, email applications, etc.), financial applications, organizational applications (e.g., address book applications), retail applications, entertainment applications (e.g., news applications, weather applications, sport applications), casting applications, etc. The interface <NUM> may be integrated with these applications <NUM> to enable the user <NUM> to control applications on the device <NUM> using his or her voice. For example, the interface <NUM> is an application programming interface (API) or any other type of program or application configured to execute the functionality of the interface <NUM>.

With continued reference to <FIG>, the user <NUM> speaks an utterance <NUM> that states "hey computer, show photos from my holiday trip to Greece with my parents. " Here, the utterance <NUM> is a voice query/command <NUM> asking the interface <NUM> to perform the action <NUM> of displaying a particular set of the user's photos. While the interface <NUM> receives audio data <NUM> corresponding to this utterance <NUM>, portions of the utterance <NUM> are processed as they are received at the interface <NUM> in order to generate intermediate speech results <NUM> (<FIG>). The intermediate speech results <NUM> are interpreted by the interface <NUM> to identify sub-actions <NUM> associated with a candidate action <NUM> while the user <NUM> is speaking the utterance <NUM>. For instance, the complete action <NUM> specified by the query <NUM> once the utterance <NUM> is complete includes a sequence of four sub-actions <NUM>, 26a-d that may be performed as the user <NUM> speaks the utterance <NUM> rather than waiting for the user <NUM> to finish speaking the utterance <NUM>.

Referring to <FIG>, the assistant interface <NUM> generally includes a speech recognizer <NUM>, an interpreter <NUM>, and an executor <NUM>. The speech recognizer <NUM> receives audio data <NUM> as an input and processes the audio data <NUM> to generate a speech recognition result <NUM>. As the speech recognizer <NUM> performs speech recognition on the audio data <NUM>, the speech recognizer <NUM> may generate intermediate speech results <NUM>. Here, an intermediate speech recognition result <NUM> refers to a result that is generated over some subset of the audio data <NUM> (e.g., a portion of the audio data) instead of the entirety of the audio data <NUM>. Generally speaking, speech recognition may occur for an entire phrase (e.g., an entire utterance) or some subset of a phrase, such as characters, wordpieces, and/or words. Stated differently, each recognition result in a sequence of speech recognition results (e.g., a sequence of intermediate speech recognition results <NUM>) may correspond to a character, wordpiece, and/or word combined to form a final recognition result (e.g., transcription) of the utterance <NUM>. For instance, when speech recognition occurs over an audio stream, a speech recognition model performing speech recognition may generate an output (i.e., a result or hypothesis) at each time step in a streaming fashion. Here, the frame-by-frame outputs may correspond to intermediate speech recognition results that may be combined to form one or more sequences of speech recognition results <NUM> representative of respective portions of the audio data <NUM>. In some configurations, a speech recognition result corresponds to the top-N hypotheses at any given time step such that multiple hypotheses may be processed at once for generating all possible partial query interpretations. Since the audio data <NUM> may correspond to an utterance <NUM> spoken by the user <NUM>, a portion of the audio data <NUM> may correspond to a portion of the utterance <NUM>. For example, when the audio data <NUM> corresponds to the utterance <NUM> of "show photos from my holiday trip to Greece with my parents," a portion of the audio data <NUM> may be a sequence of characters forming one or more words. As shown <FIG>, a first sequence of intermediate speech recognition results 212a for a first portion <NUM>, 14a of the audio data <NUM> includes the phrase "show photos. " A second sequence of intermediate speech recognition results 212b for a second portion <NUM>, 14b of the audio data <NUM> includes the phrase "from my holiday trip to Greece. " A third sequence of intermediate speech recognition results 212c for a fourth portion <NUM>, 14c of the audio data <NUM> includes the phrase "with my parents.

The interpreter <NUM> receives one or more sequences of intermediate speech recognition results <NUM> and performs partial query interpretation on the one or more intermediate speech recognition results <NUM>. For instance, the interpreter <NUM> performs semantic interpretation (e.g., grammar interpretation) on a sequence of intermediate speech recognition results <NUM> to understand a portion of the utterance <NUM> and its context to identify any candidate sub-actions <NUM> that may be associated with a final action <NUM> to be specified once the query <NUM> is revealed when the user <NUM> finished speaking the utterance <NUM>. Here, because the interpreter <NUM> is interpreting a sequence of intermediate speech recognition results <NUM> that corresponds to only a portion of the query <NUM>, the interpreter <NUM> is able to derive the context of a sub-action <NUM> from the sequence of intermediate speech recognition results <NUM> corresponding to a portion of the utterance <NUM>. Accordingly, the result of the partial query interpretation performed on a sequence of intermediate speech results <NUM> includes an interpretation <NUM> that may characterize a sub-action <NUM> that the query/command <NUM> will involve in order to fulfill a final action <NUM> that remains unknown until the user is finished speaking the utterance <NUM>. When the interpreter <NUM> performs partial query interpretation, the interpretation <NUM> may suffer from some missing information due to the inherent fact that the interpretation is unable to contextualize the entirety of the utterance <NUM>. Stated differently, the sub-actions <NUM> characterized by interpretations <NUM> may become increasingly specific as the number of sequences of intermediate speech recognition results <NUM> generated from the audio data <NUM> increases. For this reason, the interpreter <NUM> may form as complete of an interpretation <NUM> as possible by deriving a missing intent from the available information from a sequence of intermediate speech recognition results <NUM>. For example, the interpreter <NUM> may perform partial query interpretation on an initial sequence of intermediate speech recognition results <NUM> to identify a particular application type needed to perform an action <NUM>, but fails to specify a slot value associated with naming a specific application <NUM> for use in fulfilling the action <NUM> since the interpreter <NUM> cannot confidently identify the specific application <NUM> from the initial sequence of intermediate speech recognition results <NUM>. In this example, the executor <NUM> may be configured to perform a first sub-action <NUM> by launching a default application <NUM> associated with the particular application type since the slot value associated with naming the specific application <NUM> is not specified (e.g., empty).

Referring to the example shown in <FIG>, when the interpreter <NUM> generates a first interpretation <NUM>, 222a for the phrase "show me photos" indicated by the first sequence of intermediate speech recognition results <NUM>, 212a, the interpreter <NUM> is clearly lacking, at the time of interpretation, the context of which particular photos to show. Because of this limited context, the interpreter <NUM> generates a more general interpretation <NUM>, 222a that indicates that the user <NUM> is likely issuing a query <NUM> that will likely require launching a photo application <NUM> on the device <NUM>. This means that the interpreter <NUM> may infer a particular application type (e.g., photo applications) from the first sequence of intermediate speech recognition results 212a, "show me photos," even though a name of a specific photo application cannot be derived from the partial query interpretation performed on the first sequence of intermediate speech recognition results 212a. Moreover, as the user <NUM> continues to speak the utterance <NUM>, the speech recognizer <NUM> generates additional sequences of intermediate speech recognition results <NUM>, 212b-d and the interpreter <NUM> performs partial query interpretation on each of the sequences of intermediate speech recognition results <NUM> to generate respective interpretations <NUM>, 222b-d that provide additional context to the query <NUM>. In other words, the interpreter <NUM> performs partial query interpretation on each sequence of intermediate speech recognition results <NUM> while the user <NUM> is speaking to identify and construct a sequence of sub-actions <NUM> that need to be performed in order to fulfill an action <NUM>. The executor <NUM> may execute some or all of these sub-actions <NUM> before the user is finished speaking and then ultimately complete fulfillment of the action <NUM> responsive to receiving an end of speech condition when user <NUM> is finished speaking the utterance <NUM>. At the time the end of speech condition is received, the executor <NUM> may have already performed each sub-action <NUM> in the sequence of sub-actions <NUM> that are needed in order to fulfill the query <NUM>, and thus, fulfilled the action <NUM> by the time the end of speech condition is received.

In <FIG>, the interpreter <NUM> performs partial query interpretation on the second sequence of intermediate speech recognition results <NUM>, 212b of "from my holiday trip to Greece" to generate a second interpretation <NUM>, 222b that identifies a search filter (e.g., search query) on a variable of location with respect the user <NUM> within the photo application <NUM>. Based on the second interpretation 222b, the executor <NUM> may perform a second sub-action <NUM>, 26c by filtering photos in the local photo gallery application on the variable of location). For instance, <FIG> illustrates the user's albums are filtered on the variable of location.

In some examples, a subsequent interpretation <NUM> may void a prior interpretation <NUM>. For instance, if the first interpretation <NUM>, 222a led to the executor <NUM> launching two plausible photo applications <NUM> in parallel where one was a local photo gallery application and the other was a third-party photo gallery accessible via a browser application, the second interpretation <NUM>, 222b that identifies that the filter of time relates specifically to the user <NUM> (e.g., based on the word "my" preceding the word "holiday) would void/rollback the action of opening the third-party photo gallery because that gallery will not include any user-specific photos.

Referring to <FIG>, the interpreter <NUM> performs partial query interpretation on the third sequence of intermediate speech recognition results <NUM>, 212c of "with my parents" to generate a third interpretation <NUM>, 222c that identifies another search filter (e.g., search query) for the presence of the user's parents within the photos. This means that the third interpretation 222c will result in the executor <NUM> performing a third sub-action <NUM>,26c of further filtering the photos in the local photo gallery application by subject matter (i.e., the subject matter of the user's parents).

In some configurations, the interpreter <NUM> uses an interpretation model that generates a confidence level for a given interpretation <NUM>. In some implementations, the interpreter <NUM> generates multiple possible interpretations <NUM> for the same sequence of intermediate speech recognition results <NUM> and each possible interpretation <NUM> may have a respective confidence level. Furthermore, the speech recognizer <NUM> may generate multiple different candidate sequences of intermediate speech recognition results <NUM> for a same portion of audio data <NUM> and the interpreter <NUM> may generate one or more possible interpretations <NUM> for each candidate sequence of intermediate speech recognition results <NUM>. Yet in some approaches, the interface <NUM> may only want to pursue a limited number of interpretations <NUM> (e.g., one interpretation <NUM>) or interpretations <NUM> that indicate a confidence level above some interpretation confidence threshold. Here, when the interpreter <NUM> generates multiple possible interpretations <NUM> for a given sequence of intermediate speech recognition results <NUM> and with confidence levels satisfying an interpretation confidence threshold, the executor <NUM> may process respective sub-actions <NUM> characterized by the possible interpretations <NUM> in parallel. With the sub-actions <NUM> processing in parallel, the interface <NUM> may graphically display each parallel track on the display <NUM> and enable the user <NUM> to select a particular track, or even modify his or her utterance <NUM> to change the behavior of the interpreter <NUM> and/or executor <NUM>.

Proceeding with the example of <FIG>, when the executor <NUM> receives the first interpretation 222a that identifies photo applications as a particular application type needed to perform an action <NUM>, the executor <NUM> performs a first sub-action 26a by launching a first photo application <NUM>. Here, while the partial query interpretation performed on the first sequence of intermediate ASR results 212a identifies the particular application type (e.g., photo applications), the partial query interpretation fails to specify a slot value associated with naming a specific photo application for use in fulfilling the action <NUM>. Accordingly, the first photo application <NUM> corresponds to a default photo application that the user <NUM> may have previously specified to use as default (e.g., for photos) or one the user <NUM> uses most frequently. As shown in <FIG>, the display <NUM> may depict the launching of the first photo application <NUM> by showing a graphical icon representing the first photo application <NUM> being selected and proceeding to display a graphical user interface (GUI) <NUM> for the first photo application <NUM>.

When, as shown in <FIG>, the executor <NUM> receives the second interpretation 222b that identifies the search filter relating to photos with the user during holiday time in Greece, the executor <NUM> performs a second sub-action 26b by instructing the first photo application <NUM> launched on the device <NUM> to perform a search query for photos of the user during holiday time in Greece. For instance, comparing <FIG> and <FIG>, originally the application <NUM>, when launched, displayed two photo albums sorted by location. One album depicting London photos and the other album depicting photos from Greece. Here, the executor <NUM> may instruct the application to input text characterizing the search query (e.g., text saying "Greece" as a search term <NUM>ST) into a search field <NUM> of the GUI <NUM> for the first photo application <NUM> displayed on the display <NUM>. By executing this search query, <FIG> depicts the nine photo thumbnails from the album "Greece. " Following the second sub-action 26b, the executor <NUM> receives the third interpretation 222c that identifies the search filter relating to photos with the user <NUM> in Greece that includes the user's parents. The executor <NUM> performs the third sub-action 26c by sorting the photos in the "Greece" album by people. Here, this search/filter selects four of the nine photos that resulted from the previous sub-step 26b and displays these four photos within the first photo application <NUM>. As illustrated by <FIG>, the GUI <NUM> for the first photo application <NUM> includes two different methods having different GUI elements for searching or filtering content within the first photo application <NUM>. In this example, the user <NUM> may use a first GUI element corresponding to a search field <NUM> to search the content of the application <NUM> (e.g., to search photos within the first photo application <NUM>). The application <NUM> also includes GUI element corresponding to a menu of sorting/featuring options. Specific to this example, the menu <NUM> depicts a list of photo tags that allow the user <NUM> to sort his or her photos by tags that have been associated with photos of the first photo application <NUM>. A tag generally refers to an identifier that may be shared across one or more content elements (e.g., photos). In this example, the executor <NUM> may select the option of "parents" within the menu <NUM> to instruct the executor <NUM> to filter the user's photos of Greece by whether the user's parents are present.

In some configurations, when the utterance <NUM> is complete or endpointed, the executor <NUM> performs complete fulfillment of the action <NUM>. Here, the utterance <NUM> may be endpointed when the speech recognizer <NUM> detects some designated minimum duration of time of non-speech in the audio data <NUM>. The executor <NUM> may perform a highest confidence action <NUM> based on the full speech recognition result for the entirety of the audio data <NUM> (or utterance <NUM>). In these configurations, the executor <NUM> may roll back (rescind or terminate) previous sub-actions <NUM> that occurred prior to the fulfillment of the entire action <NUM>. For instance, the executor <NUM> rolls back one or more sub-actions <NUM> that are inconsistent with the entire action <NUM>. Generally speaking, the sub-action execution process aims to be in a final state that matches the final state of the execution process of the full action <NUM>. However, depending on the query/command <NUM>, this may not always be the case. Hence, roll back(s) allow the interface <NUM> to flexibly accommodate for different scenarios.

In some implementations, the executor <NUM> may roll back previous sub-actions <NUM> based not only on the entire action <NUM>, but on other sub-actions <NUM> prior to the utterance <NUM> being endpointed. To illustrate, as described above, the interpreter's first interpretation <NUM> may have led to two plausible photo-related types of applications <NUM> where one was a local photo gallery application and the other was a third-party photo gallery in a browser application. Yet later interpretations <NUM> of sub-actions <NUM> confirmed that the application <NUM> could not have been the third-party photo gallery application because it would not contain photos of the user <NUM>. In this respect, the executor <NUM> would roll back the first sub-action <NUM> for the launching of the third-party photo gallery in favor of the local photo gallery application. If the executor <NUM> actually launched the third-party photo gallery, but did not also launch the local photo gallery application, the executor <NUM>, based on the later interpretations <NUM>, may roll back the first sub-action <NUM> for the launching of the third-party photo gallery by ceasing execution of the third-party photo gallery and instead re-perform the first sub-action <NUM> by launching the local photo gallery application.

Additionally or alternatively, there may be certain sub-actions <NUM> that the interface <NUM> is not able to roll back, or that rolling back the sub-action <NUM> would negatively impact the user's experience. For example, when the sub-action <NUM> is to purchase an item on a retail application <NUM>, the interface <NUM> may not be able to roll back such a purchase or do so without user intervention. Here, a sub-action <NUM> that the interface <NUM> is not able to roll back may be referred to as an irreversible sub-action <NUM>. In these irreversible sub-actions <NUM>, the executor <NUM> may prompt the user <NUM> for authorization or action confirmation while executing an irreversible sub-action <NUM>. Another approach to irreversible sub-actions <NUM> is to identify an irreversible sub-action <NUM> and, when a sub-action <NUM> is identified as irreversible, the executor <NUM> waits to perform complete fulfillment of the action <NUM>. That is, it may be safer to have the entire context of a full speech recognition result interpreted rather than a partial speech recognition result <NUM>. In yet another approach, before the utterance <NUM> is endpointed, but after the executor <NUM> launches an application <NUM>, the executor <NUM> may determine a rollback feasibility score for a sub-action <NUM>. Here, the rollback feasibility score indicates a likelihood that the user's experience will be degraded or detrimentally impacted if the executor <NUM> rolls back the sub-action <NUM>. When the rollback feasibility score satisfies a rollback feasibility threshold, the executor <NUM> may proceed to execute the sub-action <NUM>. On the other hand, when the rollback feasibility score fails to satisfy the rollback feasibility threshold, the executor <NUM> may roll back the sub-action <NUM> or delay the roll back of the sub-action <NUM> to determine whether the complete fulfillment of the action <NUM> indicates that the sub-action <NUM> should be rolled back or not, and rolling it back accordingly.

Because the user <NUM> may see results of sub-actions <NUM> being performed by the interface <NUM> while the user <NUM> is speaking, the user <NUM> may, in some circumstances, endpoint an utterance manually before the user finishes speaking the complete utterance. Namely, when the user <NUM> is seeking a particular result from the device <NUM>, if the particular result is displayed for the user <NUM> prior to completing the utterance <NUM>, the user <NUM> may abandon the completion of the utterance <NUM> since the purpose has already been achieved. For instance, using the example of <FIG>, if all of the user's photos in the Greece album included the user's parents, the second sub-action 26b and the third sub-action 26c would produce the same results. In this scenario, the user <NUM> may recognize that the device <NUM> has successfully displayed his or her photos with his or her parents from the Greece trip. Based on this recognition, the user <NUM> may endpoint the utterance <NUM> early (i.e., before stating the portion, "with my parents").

<FIG> show an example interface <NUM> while the executor <NUM> is executing multiple tasks in parallel based on an utterance <NUM> spoken by the user <NUM>. In this example, the user <NUM> speaks an utterance <NUM> directed toward the device <NUM> that includes a query <NUM> "play rock music playlist on Google Play music. " When the user <NUM> speaks the utterance <NUM>, the interface receives the audio data <NUM> corresponding to the utterance <NUM> and relays the audio data <NUM> to the speech recognizer <NUM>. For a first portion <NUM>, 14a of the audio data <NUM> including the word "play," the speech recognizer <NUM> provides a first sequence of intermediate speech recognition results <NUM>, 212a as shown in <FIG>. The interpreter <NUM> receives and performs partial query interpretation on the first sequence of speech recognition results 212a to obtain a first interpretation <NUM>, 222a. Based on the word "play," the first interpretation 222a indicates that the audio data <NUM> includes a media-related command to execute a media type of application <NUM> and the executor <NUM> performs a first set of four sub-actions <NUM>, 26a-d that launch four different applications <NUM>, 118a-d on the device <NUM>. The first application 118a includes a first music streaming application, Google Play. The second application 118b includes a second music streaming application, Spotify. The third application 118c includes a video streaming application, YouTube, and the fourth application 118d includes a local generic video player application called "video player. " These four applications 118a-d may be launched to execute in parallel on the device <NUM> as the sequence of sub-actions <NUM> continues and the query/command spoken in the utterance <NUM> becomes more specific. When the executor <NUM> launches these applications 118a-d, these applications 118a-d may appear on the device <NUM> in one or more windows (e.g., a cascade of windows or a panel of windows). Because the first portion of the utterance <NUM> lacks a clear and identifiable application <NUM> to "play" something, the interpreter <NUM> generates multiple candidate sub-actions 26a-d, which may include all of the media type of applications <NUM> rather than a specific application <NUM> at this point when the interface is beginning to perform the action <NUM> corresponding the command <NUM> of the utterance <NUM>. Furthermore, since the execution process for each sub-action <NUM> may be displayed on the display <NUM> of the device <NUM> for the user <NUM>, the user <NUM> (e.g., while speaking the utterance <NUM>), may engage with the display <NUM> to further facilitate or modify the sub-actions <NUM>. For instance, with four applications <NUM> displayed for the user <NUM>, the user <NUM> may close one or more applications <NUM> by interacting with a GUI element of a GUI <NUM> for the application <NUM> that has the functionality to terminate a selected application <NUM>. In contrast, the user <NUM> may additionally or alternatively select which applications <NUM> to keep rather than to terminate. In yet another example, the user <NUM> may select to keep an application <NUM> and also to terminate an application <NUM>.

As the user <NUM> continues to speak the utterance <NUM>, <FIG> depicts the speech recognizer <NUM> performing speech recognition over a second portion 14b of the audio data <NUM> relating to the words "rock music" to generate a second sequence of intermediate speech recognition results <NUM>, 212b. The interpreter <NUM> interprets that the second sequence of intermediate speech recognition results 212b to obtain two different interpretations <NUM>, 222b-c. A first interpretation 222b where "rock music" refers to a genre of music (i.e., the genre of rock music) or a second interpretation <NUM> where "rock music" refers to a playlist (e.g., playlist title) for the user <NUM>. Since both of these interpretations 222b-c may have a confidence level that satisfies the confidence threshold, the executor <NUM> may generate parallel processes to execute both interpretations 222b-c on each application <NUM> launched and open from the first set of sub-actions 26a-d.

The executor <NUM> may also use the second interpretation 222b and the third interpretation 222c as a validity check to determine if any of the sub-actions <NUM> in the first set of sub-actions 26a-d should be rolled back (e.g., terminated). Since two of the applications 118c-d were applications for video (e.g., streaming video or a local video player), performing the validity check against the second interpretation 222b and the third interpretation 222c results in the executor <NUM> terminating the video-based applications 118c-d that were launched based on a third and a fourth sub-action <NUM>, 26c-d. With this being the case, executing both interpretations 222b-c only has to occur at the first application 118a and the second application 118b. This execution process therefore forms a second set of sub-actions 26e-h where four parallel action sequences are occurring as shown in <FIG>.

In <FIG>, while the user <NUM> continues to speak the utterance <NUM>, the speech recognizer <NUM> performs speech recognition over a third portion 14c of the audio data <NUM> relating to the word "playlist" to generate a third sequence of intermediate speech recognition results <NUM>, 212c that the interpreter <NUM> performs partial query interpretation on to determine a fourth interpretation <NUM>, 222d. In this step of the processing, the interpreter <NUM> may be able to recognize that this sequence of intermediate speech recognition results <NUM>, 212c clarifies some of the ambiguity of the second and third interpretations 222b-c. Namely, that "rock music" should not have been interpreted as a genre search for music, but rather referred to a playlist called "rock music. " Based on these additional context clues, the executor <NUM> has terminated any of the processes that were executing on the premise that "rock music" may correspond to a genre. After the termination of these processes, this leaves two remaining action sequences, one for the first streaming music application <NUM> of Google Play music and one for the second streaming music application <NUM> of Spotify. At this point in the action sequence, each application <NUM> has been launched and has searched for the playlist "rock music" according to the sub-actions 26i-j corresponding to the fourth interaction 222d.

In <FIG>, the user <NUM> speaks the final portion of the utterance <NUM>. While the user <NUM> speaks the final portion of the utterance <NUM> and before an endpoint <NUM> (<FIG>) (or end of speech condition) of the utterance <NUM>, the speech recognizer <NUM> performs speech recognition over a fourth portion <NUM>, 14d of the audio data <NUM> relating to the words, "on Google Play music. " Here, the speech recognizer <NUM> generates a fourth sequence of intermediate speech recognition results <NUM>, 212d for the fourth portion 14d of the audio data <NUM>. The speech recognizer <NUM> relays the fourth sequence of intermediate speech recognition results 212d to the interpreter <NUM> and the interpreter <NUM> performs a partial query interpretation to generate a fifth interpretation <NUM>, 222e that identifies the fourth sequence of intermediate speech recognition results 212d as uniquely designating a particular application, Google Play music. In other words, although this specific application <NUM> was defined at the end of the utterance <NUM>, this application <NUM> was already executing while the user <NUM> was speaking because of the context of the utterance <NUM> preceding this portion of the words "on Google Play music. " Based on this fifth interpretation 222e, the executor <NUM> ceases the execution of the Spotify application <NUM>. Additionally, when the endpoint <NUM> of the utterance <NUM> occurs, the interpreter <NUM> may then compare the sub-action sequence to the action <NUM> to fulfill the full speech recognition result from the speech recognizer <NUM>.

<FIG> is a flowchart of an example arrangement of operations for streaming action fulfillment based on partial hypotheses. At operation <NUM>, the method <NUM> receives audio data <NUM> corresponding to an utterance <NUM> spoken by a user <NUM> of a user device <NUM> where the utterance <NUM> includes a query <NUM> to perform an action <NUM> where the query <NUM> requires performance of a sequence of sub-actions <NUM> in order to fulfill the action <NUM>. Operation <NUM> includes three sub-operations <NUM>, 304a-c that occur while the method <NUM> receives the audio data <NUM>, but before receiving an end of speech condition <NUM>. At operation 304a, the method <NUM> processes, using a speech recognizer <NUM>, a first portion <NUM>, 14a of the received audio data <NUM> to generate a first sequence of intermediate automated speech recognition (ASR) results <NUM>. At operation 304b, the method <NUM> performs partial query interpretation on the first sequence of intermediate ASR results <NUM> to determine whether the first sequence of intermediate ASR results <NUM> identifies an application type needed to perform the action. At operation 304c, when the first sequence of intermediate ASR results <NUM> identifies a particular application type, performing a first sub-action <NUM> in the sequence of sub-actions <NUM>, 26a-n by launching a first application <NUM> to execute on the user device <NUM> where the first application <NUM> is associated with the particular application type. At operation <NUM>, the method <NUM> includes, in response to receiving an end of speech condition <NUM>, fulfilling performance of the action <NUM>.

<FIG> is a schematic view of an example computing device <NUM> that may be used to implement the systems (e.g., the interface <NUM>) and methods (e.g., the method <NUM>) described in this document.

The processor <NUM> may include the data processing hardware 112d of the user device <NUM> or the data processing hardware <NUM> of the remote system <NUM>. Thus, the data processing hardware <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display <NUM> coupled to high speed interface <NUM>.

The memory <NUM> may include the memory hardware <NUM> of the user device <NUM> or the memory hardware <NUM> of the remote system <NUM>. The memory hardware <NUM> may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory hardware <NUM> may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device <NUM>.

For example, it may be implemented as a standard server 400a or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), audio data (<NUM>) corresponding to an utterance (<NUM>) spoken by a user of a user device (<NUM>), the utterance (<NUM>) comprising a query (<NUM>) to perform an action (<NUM>), the query requiring performance of a sequence of sub-actions (<NUM>) in order to fulfill the action (<NUM>);
while receiving the audio data (<NUM>) before receiving an end of speech condition (<NUM>):
processing, by the data processing hardware (<NUM>), using a speech recognizer (<NUM>), a first portion of the received audio data (<NUM>) to generate a first sequence of intermediate automated speech recognition, ASR, results;
performing, by the data processing hardware (<NUM>), partial query interpretation on the first sequence of intermediate ASR results (<NUM>) to determine whether the first sequence of intermediate ASR results (<NUM>) identifies an application type needed to perform the action (<NUM>);
when the first sequence of intermediate ASR results (<NUM>) identifies a particular application type, performing, by the data processing hardware (<NUM>), a first sub-action (<NUM>) in the sequence of sub-actions (<NUM>) by launching a first application (<NUM>) to execute on the user device (<NUM>), the first application (<NUM>) associated with the particular application type; and
after launching the first application (<NUM>):
processing, by the data processing hardware (<NUM>), using the speech recognizer (<NUM>), a second portion of the received audio data (<NUM>) to generate a second sequence of intermediate ASR results (<NUM>);
performing, by the data processing hardware (<NUM>), the partial query interpretation on the second sequence of intermediate ASR results (<NUM>) to identify a second sub-action (<NUM>) in the sequence of sub-actions (<NUM>);
determining, by the data processing hardware (<NUM>), a rollback feasibility score associated with the second sub-action (<NUM>), the rollback feasibility indicating a likelihood that a user experience will be degraded if executing the second sub-action (<NUM>) has to be rolled back; and
when the rollback feasibility score satisfies a rollback feasibility threshold, performing, by the data processing hardware (<NUM>), the second sub-action (<NUM>); and
in response to receiving an end of speech condition (<NUM>), fulfilling, by the data processing hardware (<NUM>), performance of the action (<NUM>).