Patent Description:
However, in certain circumstances, these operational metrics may not be entirely conclusive with respect to utility and/or efficiency of the hardware for certain applications. For example, a vehicle can have a computing device that employs network hardware of the vehicle in order to communicate over a cellular network. The network hardware can include an antenna that is more powerful than antennas employed by other computing devices, such as cellular phones. As a result, applications that effectively rely on operational metrics of cellular phones may be misled by similar operational metrics offered by vehicle network hardware or other apparatuses.

For instance, an application that relies on signal strength (dB) to determine whether to communicate with a server computing device may not operate effectively when executing in a vehicle computing device. This can be a result of vehicle antennas enabling lower signal strength network connections compared to cellular phone antennas. Therefore-regardless of certain connection characteristics, when an application determines that a vehicle antenna is exhibiting a threshold signal strength, the application may elect to send and receive data via the vehicle antenna. This can result in the sending and receiving of data (or at least an attempt at sending and receiving) even when certain connection characteristics are suboptimal (e.g., no connection, low speeds, etc.) As a result, the application may exhibit processing delays and/or other operational deficiencies, despite-from a perspective of the application-having selected a suitable modality for performing network communications.

<CIT> discloses an exemplary approach for a computing device in communication with a server over a network that transmits an audio input comprising an utterance from a user to the server and receives results of speech recognition performed in the server.

Implementations set forth herein relate to determining whether or not to offload automated assistant tasks to a server computing device based on one or more network metrics that are indicative of connection status of an ongoing or recent communication with the server computing device. The network metric(s) can include metric(s) that are different from other network metrics, such as signal strength. Such network metric(s) can be based on data that is provided by the server computing device and/or other information determined from interacting with the server computing device. For example, when the automated assistant receives a spoken utterance from a user during a dialog session between the user and the automated assistant, the automated assistant can choose to send audio data to the server computing device for speech-to-text processing. A processing rate by which the server computing device receives and/or processes portions of the audio data can be determined by the server computing device and provided to a client computing device(s). This can allow the server computing device and the client computing device(s) to make their own respective processing decisions based on the processing rate. In some implementations, the processing rate can be determined by a client computing device. In some implementations, the processing rate can be determined as a network metric, and can be used by the automated assistant to choose whether to continue using the server computing device to process additional audio data.

For instance, the processing rate can be determined by dividing a value that indicates an amount of audio data that the server computing device processed and/or received during an amount of time by the amount of time. The amount of time can be, for example, an amount of time since the client computing device began providing audio data, an amount of time since the server computing device began receiving the amount of audio data, or a fixed amount of time (e.g., the last <NUM> seconds). The value that indicates the amount of audio data that the server computing device processed and/or received can be a value indicating a quantity of packets of audio data that have been processed and/or received, a total size of audio data that has been processed and/or received, and/or other value. Details about how much audio data that the server computing device received and/or processed can be characterized by data that is provided to the client computing device over time.

As one example, the server computing device can send acknowledgements to the client computing device. For example, the server computing device can send an acknowledgement upon receipt of each data packet (e.g., audio data packet) from the client and/or at a regular interval so long as data (e.g., audio data) from the client continues to be received. A processing rate can be determined, locally by the client device, based on a quantity of the acknowledgments received over an amount of time. For example, the processing rate can be a function of dividing the quantity of the acknowledgements received in a period of time, by a duration of the period of time. In some implementations, the acknowledgments can be void of any data that directly indicates a quantity of data that the server has received and/or processed from the client computing device. Rather, they can merely be an acknowledgment that data is being received from the client computing device. In those implementations, the acknowledgments can nonetheless be utilized in determining a processing rate based on a quantity of the acknowledgments that are received over a time period. In other implementations, the acknowledgments can include data that directly indicates a quantity of data that the server has received and/or processed from the client computing device, and such data utilized in determining a processing rate. In some implementations, the processing rate can be based on a reference time determined using a delta between a timestamp provided by the server and another timestamp provided by the client computing device. For example, each timestamp can correspond to a start of an interaction from a perspective of a respective device. In some implementations, the processing rate can be a reference time determined using a time metric characterizing a duration of audio generated at, or received by, the client computing. This duration of audio can refer to a duration of a portion of audio, or a duration of an entire amount of audio to be processed.

Depending on the choice of the automated assistant, any additional audio data that is generated during the dialog session can be processed at the client computing device or the server computing device. As a result, when the automated assistant is capable of making such decisions based on the processing rate or other network metric(s)-rather than solely based on signal strength-the automated assistant can be more responsive to a user while also eliminating processing delays and preventing unnecessary resource-intensive transmission of data to the server device. For example, if the network metric(s) indicate that the connection status to the server computing device is poor or even non-existent, the client computing device can rely on local processing in processing and responding to a spoken utterance of a user. Even though the local processing may be less accurate and/or robust than processing by the server, selectively relying on the local processing based on the network metric(s) can still be successful for many spoken utterances, while preventing undue latency in waiting on the server processing when the connection status is poor or non-existent. This can shorten the overall duration of the dialog session and prevent excess usage of client computing device resources in a more prolonged dialog session. Further, continued transmission of audio data can be halted in response to the network metric(s) indicating that the connection status is poor or non-existent, thereby preventing unnecessary usage of client device network interface(s) involved in the transmission and preventing usage of network resource(s) involved in attempting routing of audio data to the server computing device.

In some implementations, the automated assistant can be accessible via (e.g., integrated as part of) a vehicle computing device (i.e., a client computing device) that is part of a vehicle that transports one or more users and that employs an antenna that is different than antennas of other devices, such as cell phones. While riding in the vehicle, a user can provide a spoken utterance to the client computing device in order to initialize a dialog session between the user and the automated assistant and to cause the automated assistant to perform an action. The client computing device can generate audio data from the ongoing spoken utterance from the user and transmit portions of the audio data (e.g., in a streaming fashion) to a server computing device via a network connection. The transmission of the audio data can be via the antenna of the vehicle (e.g., via communication channel(s) between the antenna and cell tower(s)). The server computing device can then, via one or more processes, convert the received audio data to text, perform natural language processing on the converted text, generate fulfillment data based on the natural language processing, and/or perform other processing of the audio data (including processing on the audio data itself, or processing on text or other data generated based on the audio data).

While the user is speaking to the automated assistant, natural language content provided by the server computing device can optionally be rendered at a display interface of the client computing device as a transcription of what the user has said thus far (e.g., "Hey, Assistant, search for nearby. During this time, the client computing device may still be providing other portions of audio data to the server computing device and/or the user may still be speaking to the automated assistant (e.g., ". restaurants with a kid's menu. The automated assistant may choose to continue providing the other portions of the audio data to the server computing device as long as one or more network metrics satisfy one or more respective thresholds. However, implementations set forth herein allow the automated assistant to temporarily cease providing other portions of audio data (e.g., from the same utterance or a subsequent utterance) to the server computing device responsive to one or more network metrics satisfying certain condition(s).

For example, the client computing device can determine an amount of time that has transpired between the client computing device providing a portion of audio data to the server computing device and receiving corresponding content from the server computing device. This amount of time can be a basis for deciding whether the automated assistant should continue providing portions of audio data to the server computing device for further processing. Additionally, or alternatively, the client computing device can determine an amount of audio data that has been received, and/or processed, by the server computing device, based on data provided by the server computing device. Additionally, or alternatively, the client computing device can determine an estimated rate (i.e., a processing rate) by which data is being received, and/or processed, by the server computing device, based on data provided by the server computing device. For example, the client computing device can receive N number of acknowledgments from the server computing device within a duration of M seconds for a particular dialog session. Therefore, the estimated rate for a particular duration can be a function of N/M.

One or more of these values can then be used as a basis for determining whether the automated assistant should rely on the server computing device for further processing associated with the ongoing dialog session. For example, one or more of these values can be used as a basis for determining whether the automated assistant should rely on speech-to-text processing at the server computing device for a dialog session or instead rely on speech-to-text processing at the client computing device for a dialog session. As another example, one or more of these values can additionally or alternatively be used as a basis for determining whether the automated assistant should rely on natural language processing at the server computing device for a dialog session or instead rely on natural language processing at the client computing device for a dialog session. As yet another example, one or more of these values can additionally or alternatively be used as a basis for determining whether the automated assistant should rely on fulfillment data to be generated and/or a fulfillment to be performed at the server computing device for a dialog session or instead rely on fulfillment data generation and/or fulfillment at the client computing device for a dialog session.

As one particular example, the client computing device can record an initial timestamp corresponding to a beginning of a dialog session between the user and the automated assistant. This initial time stamp can also mark a beginning of the audio data that is generated by the client computing device. When the server computing device receives an initial portion of the audio data, the server computing device can determine an offset between the initial time stamp and a subsequent time at which the server computing device receives the initial portion of the audio data. The server computing device can then determine a total amount of the audio data, in units of time, that the server computing device has received from the client computing device or has processed, and share this value with the client computing device. The server computing device can frequently generate this value and share the value with the client computing device via a series of acknowledgments transmitted over the network connection.

When the automated assistant determines that the server computing device has received or processed at least a particular amount of audio data or non-audio data within a period of time, the automated assistant can continue sending other portions of audio data, or non-audio data (e.g., image data, location data, other media data), to the server computing device (e.g., audio data for a current dialog session or a subsequent dialog session). However, when the automated assistant determines that the server computing device has not received or not processed at least a certain amount of audio data within a certain period of time, the automated assistant can cease sending other portions of audio data to the server computing device. Rather, the automated assistant can decide to have other portions of audio data corresponding to the dialog session locally processed at the client computing device in order to generate additional natural language content. As a result, the automated assistant may render, for the user, a portion of natural language content derived at the server computing device and another portion of natural language content derived at the client computing device.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations may include a system of one or more computers that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

<FIG>, <FIG>, and <FIG> illustrate a view <NUM>, a view <NUM>, and a view <NUM> of an automated assistant and/or a client computing device <NUM> that determines whether to offload computational tasks based on network metrics that are not limited to signal strength. The user <NUM> can initialize an automated assistant via a client computing device <NUM>, which can be located in a vehicle <NUM> that includes an antenna that is different from another antenna of a portable computing device <NUM>. Certain features of the vehicle <NUM> and/or the antenna of the vehicle <NUM> may cause the client computing device <NUM> to indicate that the client computing device <NUM> is exhibiting suitable signal strength for communicating to a server computing device <NUM> (e.g., as illustrated at an interface <NUM>). However, in order to ensure that a network connection and/or the server computing device <NUM> is satisfactory, the client computing device <NUM> and/or the server computing device <NUM> can rely on one or more network metrics-not limited to signal strength. This can allow the automated assistant and/or the client computing device <NUM> to more reliably execute network and/or local tasks when there is a disparity between signal strength computing devices (e.g., signal strength indicated by interface <NUM> of the portable computing device and signal strength indicated by interface <NUM> of the client computing device <NUM>).

For example, the user <NUM> can provide an ongoing spoken utterance <NUM> such as, "Assistant, could you call. ," which can be received as an audible input by the client computing device <NUM>. In response to receiving the spoken utterance <NUM>, the client computing device <NUM> can generate audio data <NUM>, which can be stored at the client computing device <NUM> and can characterize an initial portion of an ongoing spoken utterance from the user <NUM>. In order for the automated assistant to be responsive to the user <NUM>, the audio data <NUM> can be processed in order to identify one or more actions that the user <NUM> is requesting the automated assistant to perform. In some implementations, the client computing device <NUM> can perform such processing locally, as well as communicate with the server computing device <NUM> so that the server computing device <NUM> can assist with such processing. However, when the server computing device <NUM> and/or a network connection exhibits reliability issues, while still indicating suitable signal strength, the client computing device <NUM> and/or the automated assistant can rely on locally generated content to respond to the user <NUM>.

As an example, the client computing device <NUM> can provide a portion of audio data <NUM> to the server computing device <NUM> while also processing the portion of audio data <NUM> locally at the client computing device <NUM>. Although the client computing device <NUM> may indicate suitable signal strength (e.g., as indicated by the <NUM> filled-in bars of interface <NUM>), the client computing device <NUM> and/or the automated assistant can identify one or more network metrics to determine whether to continue to seek non-local processing to respond to the user <NUM>.

In some implementations, the server computing device <NUM> can provide status data <NUM> to the client computing device <NUM> and/or content data to the client computing device <NUM>. The status data <NUM> can indicate a quantity of the audio data that has been received and/or processed by the server computing device <NUM> from the client computing device <NUM>. In some implementations, the quantity of the audio data can be characterized by a volume of data (e.g., bytes), a length of time (e.g., milliseconds), and/or any other unit of measure that can quantify an amount of data received. In some implementations, multiple instances of the status data <NUM> can be received by the client computing device <NUM> from the server computing device <NUM> as the ongoing spoken utterance from the user <NUM> is received. The client computing device <NUM> can determine, based on one or more instances of the status data <NUM>, one or more network metrics.

In some implementations, and as the ongoing spoken utterance from the user <NUM> continues, the client computing device <NUM> can provided an additional portion of audio data <NUM>, and the server computing device <NUM> can provide additional status data <NUM>. For example, as illustrated in <FIG>, the user <NUM> can continue the ongoing spoken utterance by saying, ". the nearest post office?" as an additional spoken utterance <NUM>. As the user <NUM> is continuing to provide the ongoing spoken utterance, the client computing device <NUM> can provide the additional portion of audio data <NUM>, which can characterize a portion of the spoken utterance <NUM> and/or the spoken utterance <NUM>. Furthermore, while the user <NUM> is continuing to providing the ongoing spoken utterance, the client computing device <NUM> and/or the automated assistant can process the status data <NUM> and/or the status data <NUM>.

In some implementations, the client computing device <NUM> can determine a network metric that is based on a rate in which the server computing device <NUM> is receiving the portions of audio data. Additionally, or alternatively, the network metric can be based on a rate in which the server computing device <NUM> is processing the portions of audio data. For example, in some implementations, the client computing device <NUM> can generate a network metric using an amount of time that has transpired between, and/or including, an initial time stamp that a portion of audio data corresponds to and a receipt timestamp that indicates a time when the status data <NUM> is received. This network metric can characterize a receipt rate, and the automated assistant can determine to temporarily cease communicating with the server computing device <NUM> when the receipt rate does not satisfy a threshold.

Alternatively, or additionally, the client computing device <NUM> can generate a network metric using an amount of time that has transpired between, and/or including, when the client computing device <NUM> provided the portion of audio data <NUM> to the server computing device <NUM> and when the client computing device <NUM> received the status data <NUM>. This network metric can characterize a transmission rate, and the automated assistant can temporarily cease communicating with the server computing device <NUM> when the receipt rate does not satisfy a threshold.

Alternatively, or additionally, the client computing device <NUM> can generate a network metric using an amount of data has been provided to the server computing device <NUM> for a particular dialog session and an amount of data that has been processed by the server computing device <NUM>. This network metric can characterize a processing rate, and the automated assistant can temporarily cease communicating with the server computing device <NUM> when the processing rate does not satisfy a threshold.

In some implementations, when one or more network metrics do not satisfy one or more respective thresholds, the client computing device <NUM> and/or the automated assistant can determine rely on locally processing for responding to the spoken utterance from the user <NUM>. For example, the client computing device <NUM> can generate local content data <NUM> that characterizes natural language content of the spoken utterance. In some instances, some amount of audio data can be processed at the server computing device <NUM> to provide content data to the client computing device <NUM>. This can occur prior to the client computing device <NUM> temporarily ceasing communicating with the server computing device <NUM>. As a result, when the client computing device <NUM> is responding to the user <NUM>, the client computing device <NUM> may rely on content data from the server computing device <NUM> and the client computing device <NUM>. Additionally, or alternatively, a transcription of the ongoing spoken utterance from the user <NUM> can be provided at a display interface of the client computing device <NUM>. The transcription can include natural language content that is determined based on content from the server computing device <NUM> and/or content generated at the client computing device <NUM>.

In some implementations, the client computing device <NUM> can perform speech-to-text processing locally in order to generate the local content data <NUM>, which can characterize natural language content of the spoken utterance from the user <NUM>. Based on the local content data <NUM>, the automated assistant can determine whether the spoken utterance from the user <NUM> include one or more requests that are actionable without communicating with the server computing device <NUM> and/or any other computing device that is separate from the client computing device <NUM>. For example, the automated assistant can determine, based on content data, that the user <NUM> requested an action that can involve communicating with a remote computing device. Additionally, the client computing device <NUM> can determine, based on one or more network metrics, that a network connection may be unreliable. Based on these determinations, the automated assistant can render a message <NUM> at a display interface of the client computing device <NUM>. For example, a rendered message for the user <NUM> based on these circumstances can indicate that the automated assistant can remind the user <NUM> about the requested action at a later time (e.g., "Sorry. Would you like me to remind you about this call later?"). In response to a confirmation spoken utterance <NUM> (e.g., "Yes) from the user <NUM>, the automated assistant can generate reminder data, which can cause the client computing device <NUM> and/or another computing device to provide, at a later time, a reminder that identifies the requested action from the user <NUM>.

Additionally, or alternatively, when the user <NUM> provides an affirmative response to the message <NUM>, the automated assistant can generate data that causes the client computing device <NUM> or another computing device to initialize performance of the requested action when the one or more network metrics satisfies one or more respective thresholds. For example, the client computing device <NUM> can continue to ping the server computing device <NUM> to generate subsequent network metrics based on a responsiveness of the server computing device <NUM>. Therefore, when the one or more network metrics exhibit a suitable value, the automated assistant can initialize performance of a previously requested action that was not completed, and/or prompt the user <NUM> regarding whether they would like the previously requested action to be completed.

In some implementations, a threshold for a network metric can be based on one or more features of a context in which the user <NUM> provided the spoken utterance to the automated assistant. For example, a threshold can be based on a type of client computing device <NUM> (e.g., a vehicle computing device) that the user <NUM> is accessing. Additionally, or alternatively, the threshold can be based on a type of server computing device <NUM> that the client computing device <NUM> and/or the automated assistant is attempting to access. Additionally, or alternatively, the threshold for a network metric can be based on a type of action being requested by the user <NUM> for the automated assistant to initialize. Additionally, or alternatively, a threshold for a network metric can be based on a signal strength of a connection of the client computing device <NUM> and/or another signal strength of another connection of the portable computing device <NUM>. In some implementations, the threshold for a network metric can be based on whether the user is authenticated to access a user account via the client computing device. Alternatively, or additionally, a threshold for a network metric can be based on a particular language (e.g., Swahili, French, German, etc.) that the ongoing spoken utterance is provided in.

<FIG> illustrates a system <NUM> that determines whether to offload computational tasks based on network metrics that are not limited to signal strength, and that may be based on recent interactions between a user and an automated assistant <NUM>. The automated assistant <NUM> can operate as part of an assistant application that is provided at one or more computing devices, such as a computing device <NUM> and/or a server device. A user can interact with the automated assistant <NUM> via assistant interface(s) <NUM>, which can be a microphone, a camera, a touch screen display, a user interface, and/or any other apparatus capable of providing an interface between a user and an application. For instance, a user can initialize the automated assistant <NUM> by providing a verbal, textual, and/or a graphical input to an assistant interface <NUM> to cause the automated assistant <NUM> to initialize one or more actions (e.g., provide data, control a peripheral device, access an agent, generate an input and/or an output, etc.). Alternatively, the automated assistant <NUM> can be initialized based on processing of contextual data <NUM> using one or more trained machine learning models. The contextual data <NUM> can characterize one or more features of an environment in which the automated assistant <NUM> is accessible, and/or one or more features of a user that is predicted to be intending to interact with the automated assistant <NUM>.

The computing device <NUM> can include a display device, which can be a display panel that includes a touch interface for receiving touch inputs and/or gestures for allowing a user to control applications <NUM> of the computing device <NUM> via the touch interface. In some implementations, the computing device <NUM> can lack a display device, thereby providing an audible user interface output, without providing a graphical user interface output. Furthermore, the computing device <NUM> can provide a user interface, such as a microphone, for receiving spoken natural language inputs from a user. In some implementations, the computing device <NUM> can include a touch interface and can be void of a camera, but can optionally include one or more other sensors.

The computing device <NUM> and/or other third party client devices can be in communication with a server device over a network, such as the internet. Additionally, the computing device <NUM> and any other computing devices can be in communication with each other over a local area network (LAN), such as a Wi-Fi network. The computing device <NUM> can offload computational tasks to the server device in order to conserve computational resources at the computing device <NUM>. For instance, the server device can host the automated assistant <NUM>, and/or computing device <NUM> can transmit inputs received at one or more assistant interfaces <NUM> to the server device. However, in some implementations, the automated assistant <NUM> can be hosted at the computing device <NUM>, and various processes that can be associated with automated assistant operations can be performed at the computing device <NUM>.

In various implementations, all or less than all aspects of the automated assistant <NUM> can be implemented on the computing device <NUM>. In some of those implementations, aspects of the automated assistant <NUM> are implemented via the computing device <NUM> and can interface with a server device, which can implement other aspects of the automated assistant <NUM>. The server device can optionally serve a plurality of users and their associated assistant applications via multiple threads. In implementations where all or less than all aspects of the automated assistant <NUM> are implemented via computing device <NUM>, the automated assistant <NUM> can be an application that is separate from an operating system of the computing device <NUM> (e.g., installed "on top" of the operating system) - or can alternatively be implemented directly by the operating system of the computing device <NUM> (e.g., considered an application of, but integral with, the operating system).

In some implementations, the automated assistant <NUM> can include an input processing engine <NUM>, which can employ multiple different modules for processing inputs and/or outputs for the computing device <NUM> and/or a server device. For instance, the input processing engine <NUM> can include a speech processing engine <NUM>, which can process audio data received at an assistant interface <NUM> to identify the text embodied in the audio data. The audio data can be transmitted from, for example, the computing device <NUM> to the server device in order to preserve computational resources at the computing device <NUM> when a network connection is available. Additionally, or alternatively, the audio data can be exclusively processed at the computing device <NUM>.

The process for converting the audio data to text can include a speech recognition algorithm, which can employ neural networks, and/or statistical models for identifying groups of audio data corresponding to words or phrases. The text converted from the audio data can be parsed by a data parsing engine <NUM> and made available to the automated assistant <NUM> as textual data that can be used to generate and/or identify command phrase(s), intent(s), action(s), slot value(s), and/or any other content specified by the user. In some implementations, output data provided by the data parsing engine <NUM> can be provided to a parameter engine <NUM> to determine whether the user provided an input that corresponds to a particular intent, action, and/or routine capable of being performed by the automated assistant <NUM> and/or an application or agent that is capable of being accessed via the automated assistant <NUM>. For example, assistant data <NUM> can be stored at the server device and/or the computing device <NUM>, and can include data that defines one or more actions capable of being performed by the automated assistant <NUM>, as well as parameters necessary to perform the actions. The parameter engine <NUM> can generate one or more parameters for an intent, action, and/or slot value, and provide the one or more parameters to an output generating engine <NUM>. The output generating engine <NUM> can use the one or more parameters to communicate with an assistant interface <NUM> for providing an output to a user, and/or communicate with one or more applications <NUM> for providing an output to one or more applications <NUM>.

In some implementations, the automated assistant <NUM> can be an application that can be installed "on-top of" an operating system of the computing device <NUM> and/or can itself form part of (or the entirety of) the operating system of the computing device <NUM>. The automated assistant application includes, and/or has access to, on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition can be performed using an on-device speech recognition module that processes audio data (detected by the microphone(s)) using an end-to-end speech recognition machine learning model stored locally at the computing device <NUM>. The on-device speech recognition generates recognized text for a spoken utterance (if any) present in the audio data. Also, for example, on-device natural language understanding (NLU) can be performed using an on-device NLU module that processes recognized text, generated using the on-device speech recognition, and optionally contextual data, to generate NLU data.

NLU data can include intent(s) that correspond to the spoken utterance and optionally parameter(s) (e.g., slot values) for the intent(s). On-device fulfillment can be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU), and optionally other local data, to determine action(s) to take to resolve the intent(s) of the spoken utterance (and optionally the parameter(s) for the intent). This can include determining local and/or remote responses (e.g., answers) to the spoken utterance, interaction(s) with locally installed application(s) to perform based on the spoken utterance, command(s) to transmit to internet-of-things (IoT) device(s) (directly or via corresponding remote system(s)) based on the spoken utterance, and/or other resolution action(s) to perform based on the spoken utterance. The on-device fulfillment can then initiate local and/or remote performance/execution of the determined action(s) to resolve the spoken utterance.

In various implementations, remote speech processing, remote NLU, and/or remote fulfillment can at least selectively be utilized. For example, recognized text can at least selectively be transmitted to remote automated assistant component(s) for remote NLU and/or remote fulfillment. For instance, the recognized text can optionally be transmitted for remote performance in parallel with on-device performance, or responsive to failure of on-device NLU and/or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and/or on-device execution can be prioritized at least due to the latency reductions they provide when resolving a spoken utterance (due to no client-server roundtrip(s) being needed to resolve the spoken utterance). Further, on-device functionality can be the only functionality that is available in situations with no or limited network connectivity.

In some implementations, the computing device <NUM> can include one or more applications <NUM> which can be provided by a third-party entity that is different from an entity that provided the computing device <NUM> and/or the automated assistant <NUM>. An application state engine of the automated assistant <NUM> and/or the computing device <NUM> can access application data <NUM> to determine one or more actions capable of being performed by one or more applications <NUM>, as well as a state of each application of the one or more applications <NUM> and/or a state of a respective device that is associated with the computing device <NUM>. A device state engine of the automated assistant <NUM> and/or the computing device <NUM> can access device data <NUM> to determine one or more actions capable of being performed by the computing device <NUM> and/or one or more devices that are associated with the computing device <NUM>. Furthermore, the application data <NUM> and/or any other data (e.g., device data <NUM>) can be accessed by the automated assistant <NUM> to generate contextual data <NUM>, which can characterize a context in which a particular application <NUM> and/or device is executing, and/or a context in which a particular user is accessing the computing device <NUM>, accessing an application <NUM>, and/or any other device or module. While one or more applications <NUM> are executing at the computing device <NUM>, the device data <NUM> can characterize a current operating state of each application <NUM> executing at the computing device <NUM>. Furthermore, the application data <NUM> can characterize one or more features of an executing application <NUM>, such as content of one or more graphical user interfaces being rendered at the direction of one or more applications <NUM>.

In some implementations, the system <NUM> can include a status processing engine <NUM> that can process status data received from a server computing device that is in communication with the computing device <NUM>. The status processing engine <NUM> can determine, based on one or more instances of status data, an amount of data that has been processed by a server computing device. For example, the computing device <NUM> and/or the automated assistant <NUM> can provide a portion of assistant data <NUM> to a server computing device in response to a user providing an input to an assistant interface <NUM>. When the server computing device can receive the assistant data <NUM> and, in response, provide the status data indicating an amount of data that was received. Additionally, or alternatively, the server computing device can provide status data indicating an amount of data that has been processed by the server computing device. The status processing engine <NUM> can receive the status data and determine the amount of data that has been processed and/or received by the server computing device.

In some implementations, the status processing engine <NUM> can communicate with a network metric engine <NUM>, which can generate one or more network metrics based on data communicated from the status processing engine <NUM>. For example, a network metric can be generated by the network metric engine <NUM> to characterize a rate or a velocity by which the server computing device is receiving and/or processing assistant data <NUM> provided by the computing device <NUM>. The network metric can be different from a signal strength indicator, which can indicate a power of an antenna of the computing device <NUM> or a separate computing device(s). For example, the signal strength can be high in some instances when the network metric is not indicative of a reliable network connection between a server computing device and the computing device <NUM>.

In some implementations, the computing device <NUM> and/or the automated assistant <NUM> can include a metric threshold engine <NUM>, which can determine whether one or more network metrics generated by the network metric engine <NUM> satisfies one or more thresholds. A network metric threshold can be static or dynamic, depending on the contextual data <NUM>, device data <NUM>, application data <NUM>, and/or assistant data <NUM>. In some implementations, when the metric threshold engine <NUM> determines that one or more network metrics satisfy, or fail to satisfy, one or more network metric thresholds, the automated assistant <NUM> and/or the computing device <NUM> can temporarily cease communicating data over a network connection. Additionally, or alternatively, when the metric threshold engine <NUM> determines that one or more network metrics satisfy, or fail to satisfy, one or more network metric thresholds, the automated assistant <NUM> and/or the computing device <NUM> can temporarily cease communicating with a particular server computing device and/or separate computing device.

In some implementations, a network metric can be generated for a particular context of a user and when the automated assistant <NUM> determines that the user has returned to the particular context, the automated assistant <NUM> can initially operate according to the previously determined network metric. For example, a network metric can be stored in association with a particular user, a device, a location, an application, a time, and/or any other feature or combination of features that can characterize a context. For example, a network metric can be stored in association with a location and a particular time or range of time, and the network metric can fail to satisfy a corresponding network metric threshold. Therefore, when the automated assistant <NUM> is invoked at a computing device <NUM> that is at the location at the particular time, the automated assistant <NUM> can temporarily cease providing assistant data <NUM> (e.g., audio data characterizing a spoken utterance from a user) to a separate computing device for processing. Rather, the automated assistant <NUM> can rely on local processing while the computing device <NUM> is at the location at the particular time.

<FIG> illustrates a method <NUM> for determining whether or not to offload automated assistant tasks to a server computing device based on one or more network metrics that are indicative of progress of an ongoing, or recent communication, with the server computing device. The method <NUM> can be performed by one or more computing devices, applications, and/or any other apparatus or module that can be associated with an automated assistant. The method <NUM> can include an operation <NUM> of determining whether a spoken utterance has been detected at a client computing device. The client computing device can provide access to an automated assistant, which can interact with other computing devices and or applications that are accessible via the client computing device. In some implementations, the client computing device can be a vehicle computing device that is located within a vehicle and can connect with other personal computing devices, such as cellular phones. In some implementations, the client computing device can be associated with a first network provider and the personal computing devices can be associated with a second network provider. Alternatively, or additionally, the client computing device can include an antenna that is different from an antenna that is incorporated into the personal computing device. As such, the client computing device can indicate signal strength that is different from a signal strength that is indicated by a separate personal computing device. Therefore, an automated assistant that is accessible via the personal computing device can rely on different network metrics than an automated assistant that is accessible via the client computing device.

When the client computing device has determined that the spoken utterance has been received from a user, the method <NUM> can proceed to an operation <NUM>. Otherwise, the automated assistant can continue to determine whether any user is providing an input to the automated assistant. The operation <NUM> can include generating audio data as the spoken utterance is being received. The audio data can be stored as one or more instances of data at the client computing device. In some implementations, local processing of the audio data for determining an action being requested by the user can be initialized in response to receiving the spoken utterance. Additionally, the client computing device can further determine whether to rely on a server computer device for processing. For example, the method <NUM> can include an operation <NUM> determining whether a network connection of the client computing device is suitable for providing a portion of audio data to the server computing device.

The determination of whether a network connection of the client computing device is suitable for providing a portion (e.g., instance) of audio data to the server computing device can be based on one or more network metrics. For example, in some implementations an initial determination of whether the network connection is suitable (e.g., without exhibiting a certain amount of latency, packet loss, etc.) for the automated assistant to use to communicate audio data to the server computing device. In some implementations, a network metric that can be determined using data associated with an ongoing or recent interaction between the client computing device and a server computing device. For example, during a recent interaction, the client computing device may have received one or more data packets from the server computing device at different points in time. A rate of data packets received during the interaction can be generated by dividing a total number of data packets by a total amount of time in which the data packets were being received. When this rate satisfies a rate threshold, the network connection can be considered suitable for providing a portion of the audio data to the server computing device, and the method <NUM> can proceed to the operation <NUM>. Otherwise, when the client computing device and/or the automated assistant determines that the network connection is not suitable, the method <NUM> can proceed to an operation <NUM>. In some implementations, the rate threshold can be based on whether the user is authenticated to access a user account via the client computing device. Alternatively, or additionally, the rate threshold can be based on a particular language (e.g., Swahili, French, German, etc.) that the ongoing spoken utterance is provided in.

In some implementations, the determination at operation <NUM> can be based on whether a network connection is at all available to the client computing device. For example, as a result of a location of the client computing device (e.g., the client computing device is integrated into a vehicle that is traversing an underground tunnel), the client computing device may not be able to ping any nearby cell towers. As a result, the method <NUM> may proceed to the operation <NUM>, since a network connection is at least temporarily unavailable.

The operation <NUM> can include relying on local processing of the portion of the audio data. In some implementations, the client computing device can perform speech processing (e.g., speech-to-text processing, NLU processing, etc.) at the client computing device simultaneous to determining whether to communicate audio data to the server computing device. Therefore, when the client computing device determines to not proceed to the operation <NUM>, the client computing device can use any natural language content generated at the client computing device to respond to the spoken utterance from the user. However, when the client computing device determines that there is a suitable network connection, the method <NUM> can proceed to the operation <NUM>.

The operation <NUM> can include providing the portion of the audio data to the server computing device for processing. For example, the client computing device can provide one or more portions of the audio data that characterizes a first N milliseconds of the spoken utterance from the user (where "N" is any positive number). The first N milliseconds of audio data can be processed by the server computing device in furtherance of generating natural language content data characterizing speech from the user in the first N milliseconds of the spoken utterance. In some implementations, the server computing device can provide status data to the client computing device. In some implementations, the status data can indicate an amount of data that has been received by the server computing device, a duration of audio data that has been received by the server computing device, an amount of data that has been processed by the server computing device, and/or a duration of audio data that has been processed by the server computing device.

The method <NUM> can proceed from the operation <NUM> to an operation <NUM> of determining whether additional audio to be processed for the spoken utterance. For example, the spoken utterance may have a duration of N + M milliseconds, where M is any number. Therefore, some amount of additional audio data, characterizing M milliseconds of the spoken utterance, may not have been sent to the server computing device. When there is additional audio data to be processed for the spoken utterance, the method <NUM> can proceed to an operation <NUM>. However, when there is no additional audio data to be processed for the spoken utterance and/or a dialog session between the user and the automated assistant has ended, the method <NUM> can proceed to an operation <NUM>.

The operation <NUM> can include determining whether a network metric(s) is indicative of a reliable network connection. In some implementations, the network metric can be determined by the client computing device or the server computing device. In some implementations, a network metric can be based on status data received from the server computing device and/or other data (e.g., natural language content data) received from the server computing device. The status data can indicate an amount of client data received and/or processed at the server computing device. Additionally, or alternatively, the client computing device can generate a network metric based on a velocity of the status data. Additionally, or alternatively, the client computing device can generate the network metric based on a total number of instances of status data that are received for a duration of time. For example, a network metric can be generated as a rate in which instances of data (e.g., status data, content data, etc.) is received from, or generated by, the server computing device. When this network metric satisfies a threshold, the method <NUM> can proceed from the operation <NUM> back to the operation <NUM>. Otherwise, when the network metric does not satisfy the threshold, or is otherwise not indicative of a reliable network connection, the method <NUM> can proceed from the operation <NUM> to the operation <NUM>.

Thereafter, when there is no additional audio data to be processed for the spoken utterance, NLU processing can be performed in order to identify one or more actions to be initialized, at the operation <NUM>, based on content generated from processing the audio data. In some implementations, the one or more actions can include rendering natural language content of an ongoing spoken utterance is being received by the automated assistant during a dialog session between the user and the automated assistant. In some instances, as the network metric(s) vary with environmental changes (e.g., the client computing device is a vehicle computing device that is traveling along a highway where network availability varies), the natural language content that is rendered at a display interface of the computing device can be based on natural language content data generated at the server computing device and/or other natural language content data generated at the client computing device. When the client computing device has initialized and/or performed the one or more actions, the client computing device and/or the automated assistant can continue to determine whether a spoken utterance is being provided by a user, according to operation <NUM>.

<FIG> is a block diagram <NUM> of an example computer system <NUM>.

User interface input devices <NUM> may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of method <NUM>, and/or to implement one or more of system <NUM>, server computing device <NUM>, client computing device <NUM>, and/or any other application, device, apparatus, and/or module discussed herein.

Claim 1:
A method implemented by one or more processors, the method comprising:
receiving, at an interface of a client computing device, an input from a user, wherein the input is directed to an automated assistant that is accessible via the client computing device;
generating, in response to receiving the input, input data characterizing a portion of the input as the user continues to provide additional input to the automated assistant;
providing the portion of the input to a server computing device via a network connection between the client computing device and the server computing device,
wherein the server computing device processes the input data to generate content data;
receiving, by the client computing device and from the server computing device, the content data that is based on the input data;
determining, based on an amount of input data that has been processed by the server computing device, whether to provide additional data to the server computing device for processing; and
when the client computing device determines to not provide the additional data to the server computing device:
causing the client computing device to at least temporarily cease transmitting the additional data to the server computing device,
generating, at the client computing device and using the additional data, additional content data that characterizes another portion of the input, and
causing the client computing device to render the content data and the additional content data.