Patent Description:
Users frequently interact with voice-enabled assistant interfaces on smart devices such as phones, watches, and smart speakers/displays. These assistant interfaces enable users to get things done and find answers to questions they might have, all through natural, conversational interactions. Developers are creating assistant services that leverage voice-enabled assistant interfaces. For example, automatic speech recognition (ASR) models may recognize queries spoken by users and text-to-speech (TTS) models may generate synthetic speech for output to the users that conveys responses to the spoken queries. Generally, these assistant services execute in cloud computing environments that afford flexibility and provide extensive query processing capabilities. The drawbacks to cloud-based assistant services include consuming network bandwidth, increased latency, and reduced privacy since audio data characterizing the spoken queries must be transferred from a user device to the cloud-based service.

"<NPL> et al. proposes a collaborative learning approach (COLLA) for behavior prediction that allows cloud and devices to learn collectively and continuously. COLLA finds a middle ground to build tailored model for each device, leveraging local data and computation resources to update the model, while at the same time exploits cloud to aggregate and transfer device-learned knowledge across the network to solve the cold-start problem and prevent overfitting.

One aspect described herein provides a computer-implemented that when executed on data processing hardware causes the data processing hardware to perform operations that include receiving, from a plurality of client devices each associated with a respective user, user queries directed toward a cloud-based assistant service executing on the data processing hardware. For each received user query directed toward the cloud-based assistant service, the operations also include extracting one or more attributes from the user query and logging the user query into one or more of a plurality of category buckets based on the one or more attributes extracted from the user query. The operations also include determining when at least one of the plurality of category buckets includes a threshold number of the user queries logged into the at least one category bucket, and when the at least one of the plurality of category buckets includes the threshold number of the user queries, generating a distilled model of the cloud-based assistant service. Here, the distilled model of the cloud-based assistant service is configured to execute on one or more target client devices of the plurality of client devices.

Implementations described herein may include one or more of the following optional features. In some implementations, the operations further include prompting a developer of the cloud-based assistant service to accept the generated distilled model for execution on the one or more target client devices and deploying the distilled model to the one or more of the target client devices when the developer accepts the generated distilled model. In these implementations, the operations may also include determining whether accuracy of the generated distilled model on an evaluation data set is within a threshold range of an accuracy of a teacher model on the evaluation data set. Here, prompting the developer of the cloud-based assistant service may include prompting the developer of the cloud-based assistant service when the accuracy of the generated distilled model on the evaluation data set is within the threshold range of the accuracy of the teacher model on the evaluation data set.

In some examples, the operations also include, for each received user query directed toward the cloud-based assistant service, processing, using an automatic speech recognition (ASR) model of the cloud-based assistant service, audio data characterizing the user query to generate a transcription of the user query. In these examples, extracting the one or more attributes from the user query includes performing query interpretation on the transcription of the user query to identify a query vertical type for the user query and logging the user query includes logging the user query into a corresponding one of the plurality of category buckets associated with the query vertical type identified for the user query. The one or more attributes extracted from the user query may include at least one of a device category and/or a device type associated with the client device the user query was received from, a query vertical type identified for the user query, a language and/or locale associated with a user that spoke the user query, a text-to-speech response generated by the cloud-based assistant service after fulfillment of the user query, or a transcription of the user query.

In some implementations, generating the distilled model of the cloud-based assistant service includes selecting a model configuration for the distilled model that satisfies memory and/or processing constraints of each of the one or more target client devices. In some additional implementations, generating the distilled model of the cloud-based assistant service may include: obtaining a set of training queries having attributes associated with the at least one of the plurality of category buckets that includes the threshold number of the user queries; generating, using a teacher model of the cloud-based assistant service, corresponding training labels for the training queries in the set of training queries; and training the distilled model on the set of training queries and the corresponding training labels generated for the training queries in the set of training queries. Here, at least a portion of the training queries in the set of training queries may include previous user queries selected from among the threshold number of the user queries logged into each of the at least one of the plurality of category buckets. Optionally, at least a portion of the training queries in the set of training queries may include new incoming queries having the attributes associated with the at least one of the plurality of category buckets that includes the threshold number of the user queries. Moreover, at least a portion of the training queries in the set of training queries may be selected from offline data samples having the attributes associated with the at least one of the plurality of category buckets that includes the threshold number of the user queries.

In some examples, after deploying the generated distilled model for execution on each of the one or more target devices, the operations further include: receiving, from each target client device executing the distilled model, federated analytics indicating attributes associated with new incoming queries processed by the distilled model executing on the corresponding target client device; logging the new incoming queries into one or more of the plurality of category buckets based on the federated analytics; determining when at least another one of the plurality of category buckets includes a threshold number of the user queries and the new user queries; and when the at least the other one of the plurality of category buckets includes the threshold number of the user queries and the new user queries, generating another distilled model of the cloud-based assistant service, the another distilled model of the cloud-based assistant service configured to execute on one or more target client devices of the plurality of client devices.

In some implementations, after deploying the generated distilled model for execution on each of the one or more target devices, the operations further include: receiving, from each target client device executing the distilled model, federated analytics indicating attributes associated with new incoming queries processed by the distilled model executing on the corresponding target client device; generating an updated distilled model by updating parameters of the distilled model based on the federated analytics received from each target device executing the distilled model; and deploying the updated distilled model for execution on each of the one or more target client devices. In these implementations, the federated analytics may be received from each target client device without receiving audio data characterizing any of the new incoming queries processed by the distilled model and without receiving transcriptions of the new incoming queries processed by the distilled model. The distilled model may include a speech recognition model, a text-to-speech model, or a natural language understanding (NLU) model.

Another aspect described herein provides a system that includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware causes the date processing hardware to perform operations that include receiving, from a plurality of client devices each associated with a respective user, user queries directed toward a cloud-based assistant service executing on the data processing hardware. For each received user query directed toward the cloud-based assistant service, the operations also include extracting one or more attributes from the user query and logging the user query into one or more of a plurality of category buckets based on the one or more attributes extracted from the user query. The operations also include determining when at least one of the plurality of category buckets includes a threshold number of the user queries logged into the at least one category bucket, and when the at least one of the plurality of category buckets includes the threshold number of the user queries, generating a distilled model of the cloud-based assistant service. Here, the distilled model of the cloud-based assistant service is configured to execute on one or more target client devices of the plurality of client devices.

This aspect may include one or more of the following optional features. In some implementations, the operations further include prompting a developer of the cloud-based assistant service to accept the generated distilled model for execution on the one or more target client devices and deploying the distilled model to the one or more of the target client devices when the developer accepts the generated distilled model. In these implementations, the operations may also include determining whether accuracy of the generated distilled model on an evaluation data set is within a threshold range of an accuracy of a teacher model on the evaluation data set. Here, prompting the developer of the cloud-based assistant service may include prompting the developer of the cloud-based assistant service when the accuracy of the generated distilled model on the evaluation data set is within the threshold range of the accuracy of the teacher model on the evaluation data set.

Implementations herein are directed toward a cloud-based assistant service that aggregates user queries from client devices and is capable of detecting when any components/models of the cloud-based assistant service are capable of existing on some or all of the client devices based on patterns/attributes identified in the user queries. For instance, when user queries reveal that a vast majority of the user queries belong to a particular query vertical type, the cloud-based assistant service may determine to generate a distilled speech recognition model and/or a distilled natural language understanding (NLU) module that is optimized to recognize and/or interpret queries within that query vertical type. To illustrate, a developer may create a voice- and cloud-based assistant service tailored to run on a smart watch product, whereby spoken queries captured by the smart watch are recognized via a cloud-based speech recognition model. By extracting attributes from queries received and processed by the cloud-based assistant service, the cloud-based assistant service may learn that almost all the queries are fitness-related (e.g., belong to a fitness query vertical type). As such, the service may distill custom speech recognition and NLU models tailored for recognizing and understanding fitness-related queries. Accordingly, the service may deploy these distilled custom models for execution directly on the smart watches to lead to improved latency and privacy for customers/users of the developer.

Even further, multiple distilled speech recognition models may be generated that each have a configuration suitable for a different respective client device type (a particular make/model of smart phone) the model will execute on. Client device type and category (e.g., phone, smart speaker, smart watch, etc.) associated with a client device issuing a query to the cloud-based assistant service may be extracted as an attribute of the query. The client devices now executing distilled speech recognition models provide an improved user experience in terms of latency, bandwidth usage, and privacy since potentially high-dimensional and sensitive audio data characterizing the queries can now be processed locally on the client devices without the need to use a cloud-based speech recognition model associated with the cloud-based assistant service to process the queries. As used herein, client devices may include any user computing device as well as on-premises devices of customers of the cloud-based assistant service.

Referring to <FIG>, in some implementations, and example system <NUM> includes multiple client devices <NUM> associated with one or more users <NUM> and in communication with, via a network <NUM>, a cloud-based assistant service <NUM> executing on a remote system <NUM>. The client devices <NUM> may correspond to user computing devices 102Ua-b and edge devices 102E. Each user computing devices 102U may include a mobile phone, computer (laptop or desktop), tablet, smart speaker/display, smart appliance, smart headphones, wearable, vehicle infotainment system, etc., and is equipped with data processing hardware <NUM> and memory hardware <NUM>. Each user computing device 102U includes or is in communication with one or more microphones <NUM> for capturing utterances from the respective user <NUM>. Each edge device 102E may include any on-premises device (e.g., router, routing switch, integrated access device, multiplexer, private server, etc.) associated with an enterprise or entity that provides the user computing devices 102U access to the remote system <NUM> via the network. The remote system <NUM> may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g., memory hardware).

The cloud-based assistant service <NUM> provides a digital assistant interface to the client devices <NUM> for performing actions specified by user queries <NUM> captured by the client devices <NUM>. While examples herein describe the user queries <NUM> as spoken utterances captured in streaming audio by the microphones <NUM> of the client devices <NUM>, some of the user queries <NUM> may similarly include textual queries input to the client devices by the respective users without departing from the scope of the present disclosure. Here, a query <NUM> spoken by a user <NUM> may be captured by the client device <NUM> in streaming audio and specify an action/operation/task for the cloud-based assistant service <NUM> to perform. In this sense, the users <NUM> may have conversational interactions with the cloud-based assistant service <NUM> executing on the remote system to perform computing activities or to find answers to questions.

The cloud-based assistant service <NUM> generally facilitates receiving audio data <NUM> corresponding a spoken query <NUM> captured by a microphone <NUM> of the user computing devices 102U, coordinating speech processing on the audio data <NUM>, performing semantic interpretation on the audio data <NUM> to identify the query <NUM> to perform an action, and fulfilling the action. When the microphone <NUM> of the device <NUM> captures an utterance in streaming audio and converts the audio into audio data <NUM>, the audio data <NUM> corresponding to the user query is transmitted over the network <NUM> to the cloud-based assistant service <NUM> for performing speech recognition and semantic interpretation to identify and ultimately fulfill the query <NUM> conveyed in the spoken utterance. In the example shown, the assistant service <NUM> executes a speech recognition (ASR) model <NUM> configured to generate speech recognition results on received audio data <NUM> characterizing a query <NUM> spoken by a user <NUM>, a natural language understanding (NLU) model <NUM> configured to perform semantic interpretation on the speech recognition results to identify the query <NUM>, and a text-to-speech (TTS) model <NUM> configured to generate synthesized speech representations from input text conveying responses <NUM> to the query <NUM> fulfilled by the assistant service <NUM>. The user devices 102U may share federated analytics that aggregate the audio data <NUM> and/or transcription characterizing the query <NUM> so that the audio data <NUM> and/or transcription is not attributable to any specific user device 102U or user associated therewith.

In some implementations, a developer <NUM> creates the cloud-based assistant service <NUM> to provide a digital assistant interface that interfaces with one or more applications on the devices <NUM> or accessible to the devices <NUM>. An application generally refers to any application that is configured to run on the devices <NUM>. Some types of applications include media applications (e.g., video streaming applications, audio streaming applications, media player applications, media gallery applications, etc.), word processing applications, navigation applications, social media applications, communication applications (e.g., messaging applications, email applications, etc.), financial applications, organizational applications (e.g., address book applications), retail applications, entertainment applications (e.g., news applications, weather applications, sport applications), casting applications, etc. The assistant service <NUM> may be integrated with these applications to enable the users <NUM> to control applications on the device <NUM> using his or her voice. For example, the assistant service <NUM> may provide an application programming interface (API) or any other type of program or application configured to execute the functionality of the applications.

In the example shown, the developer <NUM> creates the cloud-based assistant service <NUM> for a dental practice where the cloud-based assistant service <NUM> provides an appointment booking assistant interface for the dental practice. Here, at least some of the users <NUM>, 10a-n correspond to patients of the dental practice that use their corresponding client devices <NUM>, 102Ua-Un to access the cloud-based assistant service <NUM> to schedule dental-related appointments for procedures at the dental practice. The user computing devices 102U may access the cloud-based assistant service <NUM> via a corresponding application that the users <NUM> download on to their device 102U, a general assistant application pre-installed on the device, or a web-based application by entering a uniform resource locator (URL) associated with the dental practice. In some examples, some of the users <NUM> correspond to employees/staff of the dental practice that also access the assistant service <NUM> to review/confirm appointments booked by patients <NUM> and/or communicate secure messages with the patients. As such, the edge device 102E may correspond to a private server/computer of the dental practice that the employees/staff connect with to gain access to the cloud-based assistant service <NUM>.

Continuing with the example, a first patient 10a speaks (or optionally types) a query 119a "Schedule a root canal with Dr. Zematol in February" that is captured in streaming audio by the microphone <NUM> of the client device 102Ua and converted into corresponding audio data <NUM> that the client device 102Ua transmits to the cloud-based assistant service <NUM> via the network <NUM>. Optionally, the edge device 102E may first receive the audio data <NUM> and facilitate transmission of the audio data <NUM> to the assistant service <NUM>. As such, the cloud-based ASR model <NUM> performs speech recognition on the audio data <NUM> to generate a transcription <NUM> and the NLU model <NUM> performs semantic interpretation on the transcription <NUM> to identify the query 119a, and more particularly, identify an action that the query 119a specifies for the assistant service <NUM> to perform. Of course, the NLU model <NUM> may receive textual queries <NUM> input by users directly without the need of the ASR model <NUM>. Here, the NLU model <NUM> identifies the first query 119a indicating that a particular patient 10a would like to see if a schedule for the dentist "Dr. Zematol" has any openings in the month of February to perform a root canal. The assistant service <NUM> may thereby access the dentist's schedule, retrieve available time slots in February for performing root canal procedures, and provide a response <NUM> back to the client device 102Ua of the patient 10a indicating the available time slots Dr. Zematol has in February for performing root canals. The response <NUM> may include a text-to-speech response that the client device 102Ua outputs (via an acoustic speaker <NUM>) as synthesized speech conveying the available time slots and prompting the patient 10a to select one of the time slots. In this scenario, the TTS model <NUM> may convert input text for the response <NUM> into the synthesized speech representation and the assistant service <NUM> may transmit the synthesized speech representation as a corresponding audio file to the client device 102Ua for audible output via a speaker <NUM>. Additionally or alternatively, the response <NUM> may include a textual representation that is graphically displayed on a graphical user interface of the client device <NUM> that enables the patient 10a to select one of the available time slots to book the root canal appointment.

<FIG> also shows another patient 10n speaking another query 119n "I need to reschedule dental exam with Dr. Zematol on March <NUM>" that is captured in streaming audio by the microphone <NUM> of the client device 102Un and converted into corresponding to audio data <NUM> that the client device 102Un transmits to the cloud-based assistant service via the network <NUM>. The cloud-based assistant service <NUM> similarly executes the cloud-based ASR and NLU models <NUM>, <NUM> to transcribe the audio data <NUM> and identify the other query 119n indicating that the other patient 10n would like to schedule his/her dental exam with Dr. Zematol on March <NUM>. In this scenario, the assistant service <NUM> may cancel the existing appointment the patient 10n has with Dr. Zematol on March <NUM> and provide a corresponding response <NUM> confirming that the appointment has been canceled. The response <NUM> may additionally include available dates/times for the patient 10n to select from to reschedule the dental exam with the dentist, Dr.

For each received user query <NUM>, the cloud-based assistant service <NUM> extracts one or more attributes <NUM> from the user query <NUM>, and based on the one or more extracted attributes <NUM>, logs the user query <NUM> into one or more of a plurality of category buckets <NUM> stored on data storage <NUM>. The data storage <NUM> may reside on the storage resources (e.g., memory hardware) of the remote system <NUM>. Attributes <NUM> extracted from the user query <NUM> may include the audio data <NUM> characterizing the query <NUM>, the transcription <NUM> of the query <NUM>, a query vertical type identified for the query <NUM>, and/or one or more other properties associated with the query <NUM>. For instance, the NLU model <NUM> may perform semantic interpretation on the transcription <NUM> of the query <NUM> generated by the ASR model <NUM> to identify a query vertical type for the user query. As a result, logging the user query <NUM> includes logging the user query <NUM> into a corresponding one of the plurality of category buckets <NUM> associated with the query vertical type identified for the user query <NUM>. In the example, the query vertical type attribute <NUM> extracted from each of the queries 119a, 119n indicates a vertical associated with appointment/scheduling booking, and may even be more specific to indicate that the vertical is associated with scheduling dentist visit appointments.

The attributes <NUM> extracted from each query may further include a device category and/or a device type associated with the client device the user query was received from. For instance, the device category associated with the client device 102Un the query 119n was received from may include a smart speaker while the device category associated with the client device <NUM> the first query 119a was received from may include a smart phone. Moreover, the attribute <NUM> may specify the device type indicating a make/model of the client device. For instance, the make and model of the type of smart phone the user 10a used to issue the query 119a may be specified by the device type attribute <NUM>.

In some examples, the one or more attributes <NUM> extracted from the query <NUM> include a language and/or locale associated with the user <NUM> that spoke the user query <NUM>. Here, the language and/or locale may be extracted from any combination of the audio data <NUM>, the transcription <NUM>, or some identifier indicating the language and/or locale of the spoken query <NUM>. In additional examples, a front-end audio processing component and/or the ASR model <NUM> extracts background noise levels from the audio <NUM> containing the user query <NUM> as one of the query attributes <NUM>.

The query attributes <NUM> may further include attributes associated with the TTS response <NUM> generated by the assistant service after fulfillment of the user query <NUM>. The attributes <NUM> associated with the TTS response <NUM> may include at least one of: the text input conveying the response <NUM> that is to be converted by the TTS model <NUM> into the corresponding synthesized speech representation; an audio file of the synthesized speech representation, or TTS modeling parameters such as prosody/style features, language or voice characteristics the TTS model <NUM> was conditioned on for generating the synthesized speech representation.

As incoming user queries <NUM> are logged into the corresponding category buckets <NUM>, the assistant service <NUM> maintains a query categorization log <NUM> containing the number of queries <NUM> logged into each of the category buckets <NUM>. A distilled model generation process <NUM> analyzes the query categorization log <NUM> to identify patterns/similarities among the user queries <NUM> for opportunistically generating one or more distilled assistant models <NUM> for execution on one or more target client devices among the plurality of client devices <NUM>. In some examples, the distilled model generation process <NUM> continuously analyzes the query categorization log <NUM> on an ongoing basis as the log <NUM> dynamically updates each time a new query <NUM> is logged into one or more category buckets <NUM> based on the attributes <NUM> extracted therefrom. In other examples, the process <NUM> analyzes the query categorization log <NUM> during periodic intervals (e.g., every hour, daily, weekly, etc.).

Implementations herein are directed toward the distilled model generation process <NUM> inspecting the query categorization log <NUM> to determine when at least one of the plurality of category buckets <NUM> includes a threshold number of the user queries <NUM> logged into the at least one category bucket <NUM>. When the at least one of the plurality of category buckets <NUM> includes the threshold number of user queries <NUM>, the distilled model generation process <NUM> may generate the one or more distilled models <NUM> of the cloud-based assistant service <NUM>. For example, the process <NUM> may generate a distilled ASR model <NUM> trained to recognize common terms/phrases associated with the query vertical type (e.g., appointment booking) and/or vocabulary (e.g., dentist terminology) and proper nouns (e.g., Dr. Zematol) associated with a customer (e.g., dentist office) of the assistant service. While a threshold number of user queries <NUM> is used as a condition, other metrics such as a threshold number of queries over a designated time window or some fraction of queries. In some examples, the process <NUM> may generate multiple distilled ASR models <NUM> each having a respective model configuration that satisfies memory and/or processing constraints for the device category and/or device type associated with the target client devices <NUM> that will execute the distilled model <NUM> For instance, the process <NUM> may generate a first distilled ASR model <NUM> having a first model configuration for target client devices that include smart phones and generate a second distilled ASR model <NUM> having a different second model configuration for target client devices that include smart speakers.

In some scenarios, the distilled model generation process <NUM> transmits a distillation request <NUM> to the developer <NUM> requesting approval from the developer <NUM> before generating the distilled assistant model <NUM>. In other scenarios, the process <NUM> transmits the distillation request <NUM> to the developer <NUM> after generating the distilled assistant model <NUM>, whereby the distillation request <NUM> prompts the developer <NUM> of the cloud-based assistant service <NUM> to accept the generated distilled assistant model <NUM> for execution on the one or more target client devices <NUM>. Here, the developer <NUM> may return a distillation approval <NUM> that indicates the developer accepts the generated distilled assistant model <NUM> for execution on the target client devices <NUM> specified in the distillation request <NUM>. By the same notion, the developer <NUM> may reject deploying the generated distilled assistant model <NUM> for execution on the target devices.

In the scenario when the distilled assistant model <NUM> includes a distilled ASR model, each target client device <NUM> may perform speech recognition on audio data characterizing queries spoken by the respective user <NUM> of the client device <NUM> without having to transmit the audio data over the network <NUM> for processing by the cloud-based ASR model <NUM>. In addition to improved latency and bandwidth reduction, executing the distilled ASR model <NUM> on each of the target client devices <NUM> also preserves user privacy since no potentially sensitive audio recordings of the user <NUM> are transmitted over the network <NUM> and shared with the cloud-based assistant service <NUM>. Distilled NLU and TTS models may also be generated and deployed for execution on the target client devices <NUM> to potentially eliminate the need for the cloud-based assistant service <NUM> to execute the cloud-based ASR, NLU, and/or TTS models <NUM>, <NUM>, <NUM> for processing user queries. In some scenarios, when a distilled model <NUM> executing on a target client device is unable to process an incoming user query, the target client device <NUM> may hand-off the query <NUM> (i.e., transmit the audio data <NUM> and/or transcription <NUM> of the user query) to the cloud-based assistant service <NUM> that is capable of running more and much larger cloud-based models <NUM>, <NUM>, <NUM> to process the query <NUM>.

In some examples, after deploying the generated distilled model <NUM> for execution on each of the one or more target devices <NUM>, the cloud-based assistant service <NUM> receives, from each target client device <NUM> executing the distilled assistant model <NUM>, federated analytics indicating attributes associated with new incoming queries <NUM> processed by the distilled assistant model <NUM> executing on the corresponding target client device <NUM>. Here, the federated analytics may indicate the same attributes extracted from the new queries <NUM> as the attributes <NUM> extracted from the queries <NUM> as discussed above. However, the federated analytics received from each target device aggregate the audio data and transcriptions characterizing any of the new incoming queries processed by the distilled model so that the audio data and transcriptions are not attributable to any specific user associated with the target devices. The assistant service <NUM> may now log the new incoming queries into one or more of the plurality of category buckets <NUM> based on the federated analytics and the distilled model generation process <NUM> may analyze the query categorization log <NUM> to determine when at least another one of the plurality of category buckets <NUM> includes a threshold number of the user queries and the new user queries. As discussed previously, the distilled model generation process <NUM> may generate another distilled model <NUM> of the cloud-based assistant service <NUM> for execution on the one or more target client devices of the plurality of client devices <NUM>.

The cloud-based assistant service <NUM> may additionally or alternatively use the federated analytics received from the target client devices for generating an updated distilled model by updating parameters of the distilled model. Here, the federated analytics may additionally include performance metrics for the distilled assistant model during execution on the target client devices <NUM>. In these scenarios, the cloud-based assistant service <NUM> may collect the federated analytics shared by each target client device and determine when the distilled model can be updated/improved. Accordingly, the assistant service <NUM> may deploy the updated distilled model for execution on each of the one or more target client devices. In some examples, the assistant service <NUM> sends the parameter updates to each of the target client devices and the target client devices <NUM> generate the updated distilled model locally by using the parameter updates to update the parameters of the distilled model executing thereon.

<FIG> shows a schematic view of logging an example query <NUM> into one or more category buckets <NUM>, 210a-n based on one or more query attributes <NUM> extracted from the query <NUM>. The category buckets <NUM> may be pre-populated, defined by the developer <NUM>, dynamically generated by the assistant service <NUM> based on observed patterns as incoming user queries are received, or some combination thereof. As described above, the one or more query attributes extracted from the user query <NUM> may include at least one of the following possible attribute types: a device category and/or a device type associated with the client device <NUM> the query was received from; a query vertical type identified for the user query; a language and/or locale associated with the user that spoke the user query; background noise levels in the audio data <NUM> containing the user query <NUM>; a TTS response <NUM> generated by the cloud-based assistant service <NUM> after fulfillment of the user query <NUM>; the audio data <NUM> characterizing the user query; or a transcription <NUM> of the user query.

Each category bucket <NUM> in the plurality of category buckets 210a-n not only represents a respective category among the different possible attribute types that can be extracted from a user query, but also represents a particular classification within the respective category. For instance, a first group of the category buckets <NUM> depicted along the top row of <FIG> includes category buckets <NUM> representing different device categories such as, but not limited to, smart phones, smart speakers, smart watches, edge devices, smart headphones (not shown), or vehicle infotainment devices (not shown). Additionally, some of the category buckets <NUM> in this group are associated with particular device types indicating different makes/models of smart phones (e.g., Phone A through Phone N) that all fall into a same device category (e.g., smart phone) to thereby provide a more granular classification for logging user queries <NUM> received by the cloud-based assistant service <NUM>. Moreover, the particular device types could further classify particular operating systems or versions of operating systems. Each device category may be associated with different constraints on available computing/memory resources. Similarly, specific device types within a given device category may have different constraints in terms of disk space, memory, and or processing capacity. As will described in greater detail below, generating distilled assistant models <NUM> for execution on target client devices requires selecting model configurations (i.e., model architecture, number of weights/parameters assigned to the model, etc.) for the distilled assistant models <NUM> based on memory and/or processing constraints of the client target devices.

With continued reference to <FIG>, a second group of the category buckets <NUM> depicted along the second row from the top of <FIG> includes category buckets <NUM> associated with different query vertical types. For instance, the category buckets <NUM> within this second group may include a bucket for logging user queries directed to the assistant service <NUM> that are media related (e.g., "Assistant, stream my music playlist" or "Assistant, pause the movie"), as well as other buckets for logging user queries that are fitness related and for logging user queries related to scheduling. The number of category buckets representing different query vertical types is non-limiting and may include buckets associated with query vertical types related to navigation (e.g., "Navigate to Uncle John's house in Sebastopol, CA"), word processing, messaging (e.g., "Send message to Mom, 'I'm running late'"), and shopping (e.g., "Re-order cold brew coffee for delivery") to name a few. The developer <NUM> may further create additional custom category buckets based on custom query vertical types defined by the developer that may be of particular interest to the developer <NUM> for logging incoming user queries. By the same notion, the assistant service <NUM> may dynamically create custom query vertical types on the fly. For instance, while logging queries into the category bucket related to the query vertical type of scheduling/appointment booking, the assistant service <NUM> may observe that the transcripts <NUM> in a large portion of the these queries include dentistry terminology as well as an uncommon proper noun (e.g., the name "Dr. In fact, the assistant service <NUM> may simply pass the transcripts <NUM> for all the queries <NUM> through a language model to ascertain frequencies of terminology/proper nouns and identify specific terms/phrases/proper nouns unique to the assistant service <NUM> that have high frequencies. Accordingly, the assistant service <NUM> may dynamically create one or more custom category buckets associated with learned query vertical types and/or terminology unique to the assistant service <NUM>.

A third group of the category buckets <NUM> depicted along the third row from the top of <FIG> includes category buckets <NUM> representing different languages and/or accents associated with users that spoke the incoming user queries <NUM> directed toward the cloud-based assistant service. For example, the third group may include category buckets for different languages A-N, as well as different accents/dialects within each of the different languages. For instance, there may include multiple category buckets <NUM> representing language A corresponding English, whereby each category bucket <NUM> represents a particular accent/dialect of English (e.g., American English, American English with southern accent, British English, British English with Manchester accent, etc.).

A fourth group of the category buckets <NUM> represented along the bottom row of <FIG> includes category buckets <NUM> representing different background noise levels in the audio data <NUM> containing the user queries <NUM>. For instance, this fourth group of category buckets <NUM> may include three buckets for classifying background noise levels in incoming queries as low, medium, or high. That is, the category bucket representing the low background noise level may include any user queries <NUM> having audio data with background noise levels less than a minimum noise level threshold, while the category bucket representing the high background noise level may include any user queries <NUM> having audio data with background noise levels greater than a maximum noise level threshold. Here, the maximum noise level threshold is greater than the minimum noise level threshold. Similarly, the category bucket representing medium background noise level may include any user queries <NUM> having audio data with background noise levels greater than or equal to the minimum noise level threshold and less than or equal to the maximum noise level threshold. There may exist more/less than three category buckets for representing different ranges of background noise levels.

<FIG> depicts a user query <NUM> received by the cloud-based assistant service <NUM> and having query attributes <NUM> extracted therefrom that include the audio data <NUM>, the transcript <NUM> generated by the cloud-based ASR model <NUM>, a device type indicator indicating the make/model of the client device the query <NUM> was received from, a language/accent identifier (e.g., British English) indicating the language and/or accent associated with the user that spoke the user query, a query type vertical indicator indicating the query vertical type (e.g., Scheduling vertical), and a noise level indicator indicating a background noise level (e.g., in decibels (Db)) of the audio data <NUM> containing the user query <NUM> captured by the client device <NUM>. Based on the extracted query attributes <NUM>, the assistant service <NUM> logs the user query <NUM> into multiple category buckets <NUM> each representing a particular classification within a respective category among the different attribute types extracted from the user query <NUM>. For example, the query <NUM> logs into each of the following category buckets: the category bucket representing the device type of Phone A; the category bucket representing the query vertical type related to scheduling/appointment booking; the category bucket representing Language A/Accent N associated with British English speakers; and the category bucket representing medium background noise levels. Notably, the category bucket representing the device type of Phone A also represents the device category of smart phones and the category bucket representing British English also broadly represents an English language query. Solid rectangles within each category bucket <NUM> may denote a logged query <NUM>, while dashed rectangles may denote slots available for logging queries. A bucket with all solid rectangles may indicate that the category bucket includes a threshold number of queries <NUM>.

<FIG> illustrates an example of the distilled model generation process <NUM> of <FIG> for generating one or more distilled assistant models <NUM> for execution on one or more target client devices <NUM>. The process <NUM> includes a distilled model candidate identification stage ('candidate identification stage') <NUM>, a distilled model training stage ('training stage') <NUM>, and an evaluation stage <NUM>.

During the identification stage <NUM>, the process <NUM> receives the query categorization log <NUM> from the data storage <NUM> indicating a number of user queries <NUM> received by the assistant service <NUM> that are logged into each corresponding category bucket among the plurality of category buckets <NUM>. For instance, <FIG> shows the query categorization log <NUM> having a query count indicator <NUM> for each category bucket that indicates the number of queries <NUM> logged into the corresponding category bucket <NUM>. The process <NUM> analyzes/inspects the query categorization log <NUM> to determine when at least one of the plurality of category buckets <NUM> includes a threshold number of the user queries <NUM> logged into the corresponding at least one category bucket <NUM>. The fully-shaded query count indicator <NUM> indicates that a threshold number of queries are logged into the corresponding category bucket <NUM> while an entirely unshaded query count indicator <NUM> indicates that zero queries have been logged into the corresponding category bucket <NUM>. In the example shown, the query categorization log <NUM> shows that the category buckets <NUM> representing the device type N, the query vertical type related to scheduling/appointment booking, and the spoken query language A each include the threshold number of queries as indicated by the fully-shaded query count indicators <NUM>. Notably, the entirely unshaded query count indicators <NUM> for the category buckets <NUM> representing the media query vertical type and Language N reveal that none of the user queries <NUM> received at the cloud-based assistant service were spoken in Language N or included media-related queries.

Based on the process <NUM> determining that the category buckets <NUM> representing the device type N, the query vertical type related to scheduling/appointment booking, and the spoken query language A each include the threshold number of queries <NUM>, the distilled model candidate identification stage <NUM> identifies one or more distilled assistant model candidates <NUM>, 315a-n of the cloud-based assistant service <NUM>. The distilled assistant model candidates <NUM> identified during the identification stage <NUM> may include any combination of one or more distilled ASR model candidates, one or more NLU model candidates, one or more TTS model candidates, or any other types of models or components used by the cloud-based assistant process during the query processing pipeline. For instance, other types of models/components that can be distilled to target devices may include language models and/or speech processing models for processing background noise levels and/or noise cancellation.

The identification stage <NUM> may identify multiple distilled assistant model candidates related to a same type of model (e.g., ASR model) where each candidate is adapted for use on a respective set of target devices associated with a respective device category or particular device type. Optionally, multiple candidates related to the same type of model and intended for use on the same set of target devices may be identified and eventually evaluated to select a top candidate for distillation to target client devices. As such, the identification stage may select a respective model configuration for each distilled assistant model candidate <NUM> that satisfies memory and/or processing constraints of each of the one or more target client devices intended to execute the distilled model candidate <NUM>. For instance, the identification stage <NUM> may obtain (e.g., look-up) memory and/or processing constraints for the device type N and select a model configuration for a distilled assistant model candidate <NUM> intended to execute on the device type N. Here, the selecting of the model configuration may include assigning a number of model weights to the corresponding model candidate <NUM> based on available memory of the device type N and/or assigning a number of operations that can be performed by the corresponding model candidate <NUM> based on processing capacity of the device type N. Additionally or alternatively, selecting model configurations for model candidates <NUM> to be distilled to target devices may include assigning model weights and/or operation constraints based on a lowest common denominator in terms of available memory and/or processing constraints for the target devices.

During the training stage <NUM>, the distilled model generation process <NUM> generates a corresponding distilled assistant model <NUM> for each distilled assistant model candidate <NUM> identified during the identification stage <NUM>. In some implementations, for each distilled assistant model candidate <NUM>, the process <NUM> obtains a set of training queries <NUM> having query attributes <NUM> associated with one or more of the category buckets <NUM> that include the threshold number of queries <NUM>. For example, when the distilled assistant model candidate <NUM> includes a distilled ASR model candidate adapted for recognizing terms/phrases related to scheduling/booking and spoken in Language A, the training stage <NUM> may issue a data retrieval request <NUM> to retrieve a set of training queries <NUM> stored on the data storage <NUM> that were spoken in Language A and belong to the query vertical type related to scheduling/appointment booking. Here, at least a portion of the training queries in the set of training queries <NUM> may include previous user queries selected from among the threshold number of user queries logged into each of the category bucket s210 representing the query vertical type related to scheduling/appointment booking and queries spoken in Language A. At least a portion of the training queries in the set of training queries <NUM> may additionally or alternatively include new incoming queries received at the cloud-based assistant service <NUM> that have the query attributes <NUM> associated with the one or more of the category buckets <NUM> that include the threshold number of queries <NUM>. For instance and continuing with the example, the training stage <NUM> may obtain at least a portion of the training queries in the set of training queries <NUM> by filtering out new incoming queries <NUM> that belong to the query vertical type related to scheduling/appointment booking and that are spoken in Language A. Further, at least a portion of the training queries obtained in the set of training queries <NUM> may be selected from offline data samples stored on the data storage <NUM> that have the attributes <NUM> associated with the one or more of the category buckets <NUM> that include the threshold number of queries <NUM>.

Each training query in the set of training queries <NUM> may include corresponding audio data characterizing the training query spoken by a respective user. Each training query <NUM> associated with a previous query <NUM> processed by the assistant service may be paired with the corresponding transcription <NUM> generated by the cloud-based ASR model <NUM> of the assistant service <NUM>. As such, the training stage <NUM> may use the cloud-based ASR model <NUM> as a teacher model for generating corresponding training labels (i.e., ground-truth transcriptions) for each sample of audio data characterizing the training queries in the set of training queries. Similarly, each training query <NUM> associated with a previous query <NUM> processed by the assistant service may be paired with the corresponding TTS response <NUM> including at least one of input text to the cloud-based TTS model <NUM> or a synthesized speech representation generated by the cloud-based TTS model <NUM>.

Once the set of training queries and corresponding training labels are obtained/generated, the training stage <NUM> trains the corresponding distilled assistant model <NUM> on the set of training queries and the corresponding training labels generated for the training queries in the set of training queries. Training may include training for a specific amount of time, training with a specified number of training steps, training on a specified number of training queries in the set of training queries, and/or until a training loss for training the model stabilizes. Notably, in the example where the distilled assistant model <NUM> includes the distilled ASR model, the distilled ASR model may include a drastic reduction in size and processing requirements compared to the corresponding cloud-based ASR model <NUM> since the distilled ASR model <NUM> is customized for recognizing queries spoken in Language A and including terms/phrases related to scheduling/appointment booking. The cloud-based ASR model may include a very large multilingual model trained to recognize speech in a multitude of different languages and spanning a multitude of different query vertical types. However, since the query categorization log <NUM> identifies threshold numbers of queries sharing common attributes <NUM>, the distilled ASR model may be trained as a reduced-size custom model having a configuration and parameters suited for processing queries having a limited set of attributes. The training stage <NUM> repeats these steps to train and generate a corresponding distilled assistant model candidate for each distilled assistant model candidate <NUM> identified during the identification stage <NUM>.

After the training stage <NUM> generates and trains each distilled assistant model <NUM>, the evaluation stage <NUM> may evaluate each distilled assistant model <NUM> with respect to a corresponding teacher model <NUM>. In some examples, the teacher model <NUM> includes the corresponding cloud-based model of the cloud-based assistant service <NUM> that is related to the distilled assistant model <NUM>. For instance, for any distilled assistant models <NUM> that include distilled ASR model, the evaluation stage <NUM> uses the cloud-based ASR model <NUM> of the assistant service <NUM> as the teacher model <NUM> for evaluating each of the distilled ASR models.

In some implementations, the evaluation stage <NUM> determines whether accuracy of the generated distilled assistant model <NUM> on an evaluation data set <NUM> is within a threshold range of accuracy of the corresponding teacher model <NUM> on the evaluation data set <NUM>. In these implementations, the evaluation stage prompts the developer <NUM> of the cloud-based assistant service <NUM> to accept the generated distilled assistant model <NUM> for execution on the one or more target client devices <NUM> when the accuracy of the generated distilled model <NUM> on the evaluation data set is within the threshold range of the accuracy of the corresponding teacher model <NUM>. In some examples, the evaluation set <NUM> includes audio data characterizing a live set of incoming queries and the distilled assistant and teacher models <NUM>, <NUM> correspond to ASR models. In these examples, a word error rate (WER) of the speech recognition results output by the models <NUM>, <NUM> for the incoming queries may be used as the metric to indicate the accuracy/performance of each of the models <NUM>, <NUM>. Other metrics may be used to indicate the accuracy/performance of the models <NUM>, <NUM>.

The evaluation stage <NUM> may prompt the developer <NUM> to accept the generated distilled assistant model <NUM> by transmitting the distillation request <NUM> to the developer <NUM>. In some examples, the distillation request <NUM> includes a message describing characteristics of the distilled assistant model <NUM>, target devices <NUM> intended to execute the distilled assistant model <NUM>, and/or evaluation results of the distilled assistant model. For instance, the distillation request <NUM> transmitted to the developer <NUM> includes the message: "We have a distilled model available which can run on devices X. It is within <NUM>% accuracy of the cloud-based model but will reduce latency and improve privacy". As shown in <FIG>, the developer <NUM> may return a distillation approval <NUM> that indicates the developer accepts the generated distilled assistant model <NUM> for execution on the target client devices <NUM> specified in the distillation request <NUM>. By the same notion, the developer <NUM> may reject deploying the generated distilled assistant model <NUM> for execution on the target devices.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of distilling assistant models <NUM> to client devices <NUM> based on user queries <NUM> directed toward a cloud-based assistant service <NUM> executing on the data processing hardware <NUM> of the remote system <NUM>. The method <NUM> may include a computer-implemented method that executes on the data processing hardware <NUM> of the remote system 111to cause the data processing hardware <NUM> to perform the operations. At operation <NUM>, the method <NUM> includes receiving, from a plurality of client devices <NUM> each associated with a respective user, user queries <NUM> directed toward the cloud-based assistant service <NUM>. At operation <NUM>, for each received user query <NUM> directed toward the cloud-based assistant service <NUM>, the method <NUM> also includes extracting one or more attributes <NUM> from the user query <NUM> and logging the user query <NUM> into one or more of a plurality of category buckets <NUM> based on the one or more attributes <NUM> extracted from the user query <NUM>.

At operation <NUM>, the method <NUM> also includes determining when at least one of the plurality of category buckets <NUM> includes a threshold number of the user queries logged into the at least one category bucket <NUM>. At operation <NUM>, when the at least one of the plurality of category buckets <NUM> includes the threshold number of user queries, the method <NUM> also includes generating a distilled model <NUM> of the cloud-based assistant service <NUM>. Here, the distilled model <NUM> of the cloud-based assistant service <NUM> is configured to execute on one or more target client devices <NUM> of the plurality of client devices <NUM>.

Claim 1:
A computer-implemented method (<NUM>) when executed on data processing hardware causes the data processing hardware (<NUM>) to perform operations comprising:
receiving, from a plurality of client devices (<NUM>) each associated with a respective user, user queries (<NUM>) directed toward a cloud-based assistant service (<NUM>) executing on the data processing hardware (<NUM>);
for each received user query (<NUM>) directed toward the cloud-based assistant service (<NUM>):
extracting one or more attributes (<NUM>) from the user query (<NUM>); and
logging the user query (<NUM>) into one or more of a plurality of category buckets (<NUM>) based on the one or more attributes (<NUM>) extracted from the user query (<NUM>);
determining when at least one of the plurality of category buckets (<NUM>) includes a threshold number of the user queries (<NUM>) logged into the at least one category bucket (<NUM>); and
when the at least one of the plurality of category buckets (<NUM>) includes the threshold number of the user queries (<NUM>), generating a distilled model (<NUM>) of the cloud-based assistant service (<NUM>), the distilled model (<NUM>) of the cloud-based assistant service (<NUM>) configured to execute on one or more of the plurality of client devices (<NUM>).