Patent ID: 12230257

DETAILED DESCRIPTION

There is a proliferation of smart network connected devices (also referred to herein as smart devices or Internet of Things (IoT) devices) such as smart home alarms, smart door locks, smart cameras, smart lights, smart thermostats, smart weight scales, smart beds, smart irrigation systems, smart garage door openers, smart plugs, smart appliances, smart baby monitors, smart televisions (TVs), smart fire alarms, smart moisture detectors, smart routers, etc. Often, multiple smart devices are located within the confines of a structure, such as a home—or located within multiple related structures, such as a user's primary residence and the user's secondary residence and/or work location.

Further, there is a proliferation of assistant client devices that can each include an assistant client that can optionally interact with one or more remote automated assistant components to form a logical instance of an automated assistant. An assistant client device can be devoted solely to assistant functionality (e.g., a standalone speaker and/or standalone audio/visual device including only an assistant client and associated interface, and devoted solely to assistant functionality) or can perform assistant functionality in addition to other functions (e.g., a mobile phone or tablet that includes an assistant client as one of multiple applications). Moreover, some smart devices can also be assistant client devices. For example, some smart devices can include an assistant client and at least speaker(s) and/or microphone(s) that serve (at least in part) as user interface output and/or input devices for an assistant interface of the assistant client.

Various techniques have been proposed for associating smart devices with corresponding logical instances of automated assistants (and optionally with individual assistant client devices). For example, a user, group of users, an assistant client device, and/or a group of assistant client devices (e.g., all within a structure) can be linked (e.g., in one or more databases) with a plurality of disparate smart devices to enable interaction with (e.g., control of) the smart devices via automated assistants. For instance, each of multiple assistant client devices in a household can be linked to each of multiple disparate smart devices in the household to enable any user (or a restricted group of users) to interface with any one of the assistant client devices to interact with any one of the multiple disparate smart devices.

Such linkings of smart devices and assistant client devices is referred to herein as a device topology and can be represented by various data structures. A device topology can be user created, and/or automatically created, and can define various assistant client devices, various smart devices, identifier(s) for each, and/or attribute(s) for each. For example, the identifier(s) for a device can specify a room (and/or other area(s)) of a structure in which the device is located (e.g., living room, kitchen) and/or can specify nickname(s) and/or alias(es) for the device (e.g. couch lamp, front door lock, bedroom speaker, kitchen assistant, etc.). In this manner, the identifiers of devices can be names, aliases, and/or locations of the respective devices that the user is likely to associate with the respective devices.

The device topology representation can further specify one or more device attributes associated with the respective devices. The device attributes for an assistant client device can include, for example, one or more input and/or output modalities supported by the assistant client device and/or preferred smart device(s) to be controlled by assistant client device(s) (e.g., ambiguous smart TV commands received at assistant client device 1 should be assumed to be directed to smart TV 1, whereas they can be assumed to be directed to smart TV 2 for assistant client device 2). For instance, a device attribute for a standalone speaker-only assistant client device can indicate that it is capable of providing audible output, but incapable of providing visual output. Also, for instance, a device attribute for the same standalone speaker-only assistant client device can indicate that lighting control requests that don't specify particular light(s), should be interpreted to pertain to the light(s) assigned to the same room as that assistant client device. The device attributes of a smart device can, for example, identify one or more states, of the smart device, that can be controlled; identify a party (e.g., a 3P) that manufactures, distributes, and/or creates the firmware for the smart device; and/or identify a unique identifier for the smart device, such as an address of the smart device and/or a 1P or 3P provided fixed identifier. According to various implementations disclosed herein, the device topology representation can optionally further specify: which smart devices can be controlled locally by which assistant client devices; local addresses for locally controllable smart devices (or local addresses for hubs that can directly locally control those smart devices); local signal strengths and/or other preference indicators between assistant client devices and smart devices. Further, according to various implementations disclosed herein, the device topology representation (or a variation thereof) can be stored remotely at a remote assistant system and/or locally stored at each of a plurality of assistant client devices for utilization in locally controlling smart devices. Optionally, each locally stored device topology may be less robust (e.g., define less devices, attributes, and/or identifiers) than the remotely stored device topology due to, for example, assistant client device hardware constraints.

Now turning toFIG.1, an example environment in which techniques disclosed herein may be implemented is illustrated. The example environment includes a plurality of assistant client devices1101-N(also referred to herein simply as “client devices”), cloud automated assistant component(s)120, smart device systems140A-N, smart devices145A-N, and a device topology152for the client devices1101-N. The client devices1101-Nand smart devices1451-NofFIG.1represent client devices and smart devices that are at least selectively associated with one another (e.g., via the device topology152). For example, the smart devices1451-Ncan all be at a home (e.g., in the interior and/or exterior of the home), the client devices1101-Ncan be at least occasionally in the same home, and the smart devices1451-Nand the client devices1101-Ncan be linked to one another utilizing one or more techniques, such as those described herein. Through such linking, the client devices1101-Ncan be utilized to control the smart devices1451-Naccording to implementations described herein.

One or more (e.g., all) of the client devices1101-Ncan execute a respective instance of an automated assistant client. However, in some implementations one or more of the client devices1101-Ncan optionally lack an instance of an automated assistant client and still include engine(s) and hardware components for controlling or more smart devices. An instance of the automated assistant client can be an application that is separate from an operating system of the corresponding client device1101-N(e.g., installed “on top” of the operating system)—or can alternatively be implemented directly by the operating system of the corresponding client device1101-N. Each instance of the automated assistant client can optionally interact with cloud automated assistant component(s)120in responding to various requests provided by a user via I/O components111of any one of the client devices1101-N. Further, other engine(s) of the client devices1101-Ncan optionally interact with cloud automated assistant component(s)120.

One or more (e.g., all) of the client devices1101-Ncan include a respective speech-to-text (STT) engine1121-Nthat utilize respective locally stored STT models1521-Nto process audio data that captures spoken utterances, to generate corresponding recognized text for the spoken utterances. Each instance of audio data can be detected via microphone(s) of I/O component(s)1111-Nof a respective client device. In some implementations, the STT engines1121-Nonly process audio data following an explicit automated assistant invocation detected at a corresponding client device1101-N. In some implementations, the STT engines1121-Ncan at least selectively process audio data, even absent an explicit automated assistant invocation, but discard (without further processing) any audio data and/or recognized text determined not to include any “hot phrase” as described herein. In situations where a “hot phrase” is detected, one or more responsive action(s) can be taken based on the “hot phrase”. For example, where the “hot phrase” matches text of a cache entry, of a corresponding local cache1581-N, semantic representation(s) of the cache entry can be further processed as described herein.

One or more (e.g., all) of the client devices1101-Ncan also optionally include a respective natural language understanding (NLU) engine1131-Nthat utilize respective locally stored NLU models1531-Nto process text, to attempt to locally generate a local semantic representation of the text. The text can be recognized text from STT engine112, or can be typed text (e.g., input via a virtual keyboard of a touchscreen), or can be text that is associated with a selected virtual and/or hardware button (e.g., text that is mapped to selection of a particular virtual button displayed on a touch screen). The NLU engines1131-Nmay be able to generate valid semantic representations for various textual phrases. However, as described herein, the NLU engines1131-Nmay also fail to generate valid semantic representations for various other textual phrases such as, for example, various textual phrases that relate to control of smart device(s) (e.g., relative control of smart device(s)).

One or more (e.g., all) of the client devices1101-Ncan also include respective fulfillment engine1141-N. Each fulfillment engine1141-Ncan include a respective cloud module1151-N, a respective local module1161-N, and/or a respective text-to-speech (US) engine1171-N.

The cloud modules1151-Ncan fulfill various requests through interaction with cloud automated assistant component(s)120. For example, cloud module1151can, responsive to determining text of received input (e.g., recognized text STT engine1121-N) matches text of a cache entry of local cache1581, transmit a cloud semantic representation(s), mapped to the text in the cache entry, to cloud automated assistant component(s)120as described herein. Cloud module1151can also optionally transmit cloud semantic representations, generated by NLU engine1131, to cloud automated assistant component(s)120. Cloud fulfillment engine127, of cloud automated assistant component(s)120, can then utilize the received cloud semantic representation(s) in fulfilling the received input. For example, for smart device control inputs, the cloud fulfillment engine127can utilize the cloud semantic representation to generate corresponding control request(s), to transmit to corresponding smart device system(s)140A-N, which then generate and transmit corresponding control command(s) to corresponding smart device(s)145A-N.

The cloud modules1151-Ncan also, for received input(s) that are not resolvable with a respective NLU engine1131-Nand/or that do not match any cache entries of a respective local cache1581-N, transmit data representative of the received input(s) to cloud based automated assistant component(s)120. For example, if the received input is a spoken utterance captured in audio data, the audio data and/or local recognized text therefor (e.g., from a respective SU engine1121-N) can be transmitted to the cloud based automated assistant component(s)120. For at least some of those transmission(s), the cloud-based component(s)120can provide, in response, cache request(s) to a respective cache engine1181-N, to enable future occurrences of the input(s) to be fulfilled more efficiently and/or with reduced latency. Further, the cloud-based component(s)120can additionally or alternatively, responsive to the received input(s), perform a corresponding cloud fulfillment and/or provide locally interpretable semantic representations and/or local command(s) to a respective client device1101-Nfor utilization by the respective client device1101-Nin performing a local fulfillment.

The local modules1161-Ncan fulfill various requests locally and without interaction with cloud automated assistant component(s)120. For example, local module1161can, responsive to determining text of received input (e.g., recognized text from STT engine1121-N) matches text of a cache entry of local cache1581, utilize locally interpretable semantic representation(s), mapped to the text in the cache entry, in locally fulfilling a request. For instance, the local module1161can utilize the locally interpretable semantic representation(s) in generating local control command(s) to locally transmit to one or more corresponding smart devices145A-Nover respective ones of the radio(s)1191. Optionally, and as described herein, when the locally interpretable semantic representation(s) are for smart device control, the local modules1161-Ncan utilize one or more respective adapter(s)1561-Nin generating the local control command(s). Local module1161can also optionally locally interpret any locally interpretable semantic representations generated by NLU engine1131.

Text-to-speech (US) engines1171-Ncan optionally be utilized to generate synthesized speech for providing responsive to requests. For example, TTS engine1171can, responsive to determining text of received input (e.g., recognized text from SU engine1121-N) matches text of a cache entry of local cache1581, utilize responsive text, mapped to the text in the cache entry, to generate corresponding synthesized speech and cause the synthesized speech to be rendered response to the received input. The synthesized speech can be rendered in addition to other action(s) performed based on semantic representation(s) (cloud and/or locally interpretable) also mapped to the text in the cache entry. Further, the responsive text can additionally or alternatively be visually rendered and/or alternative content can additionally or alternatively be rendered by the fulfillment engine responsive to it being mapped to the text in the cache entry.

One or more (e.g., all) of the client devices1101-Ncan also optionally include a respective cache engine1181-Nand a respective local cache1581-N, mentioned above. The cache engines1181-Ncan each receive respective cache requests from cache generation engine125of cloud automated assistant component(s)120, and store respective cache entries in a respective local cache1581-N. The local caches1581-Ncan be stored in, for example, RAM and/or ROM of the respective client devices1101-N. As described herein, in some implementations a received cache request, from cache generation engine125includes the entirety of the cache entry to be stored in a respective local cache1581-N. In other implementations, the cache request may lack the text to include in a cache entry and map to semantic representation(s) in the cache entry. In such implementations, a respective cache engine1181-Ncan determine the text based on it being the text most recently provided to the cloud automated assistant component(s)120(or recognized text for audio data most recently provided to the cloud automated assistant component(s)120.

The cloud automated assistant component(s)120can be implemented on one or more computing systems (collectively referred to as a “cloud” or a “remote” assistant system) that are communicatively coupled to client devices1101-Nvia one or more wide area networks (e.g., the Internet), indicated generally by1051ofFIG.1. For example, cloud automated assistant component(s)120can be implemented by one or more clusters of high-performance servers. It is noted that the client devices1101-Ncan utilize one or more local area networks in accessing the wide-area networks1051and/or in locally communicating with one another. Such local area networks can include a Wi-Fi network and/or a mesh network between the client devices1101-N.

The cloud automated assistant components120can also be communicatively coupled with smart device systems140A-Nvia one or more wide area networks. The communicative coupling of the cloud automated assistant components120with the smart device systems140is indicated generally by1052ofFIG.1. Further, the smart device systems140can each be communicatively coupled to a corresponding group of one or more smart devices145A-Nvia one or more wide area networks, generally indicated generally by1104ofFIG.1. It is noted that the smart devices145A-Ncan utilize one or more local area networks in accessing the wide-area networks1053.

Each of the smart device systems140A-Ncan be either a first party (1P—i.e., manufactured and/or distributed by the same party that controls the automated assistant) or a third party (3P—i.e., manufactured and/or distributed by a different party) system, and each can be communicatively coupled with one or more corresponding smart devices145A-N. For example, a first smart device system140A-Ncan be controlled by a first 3P and communicatively coupled with a first smart device145A1, a second smart device system140can be controlled by a second 3P and communicatively coupled with a second smart device145B1and a third smart device145B2, etc.

The smart device systems140A-Ncan communicate with the devices145A-Nvia the wide-area networks1053to control their respective smart devices145A-N, to deliver firmware updates to their respective smart devices145A-N, to ascertain the status of their respective smart devices145A-N, etc. For example, a given one of the smart device systems140can communicate with one of the smart devices145A-Nto control the smart device in response to user input being received via a mobile application, for the smart device system, that enables control of the smart device.

Also, for example, a given one of the smart device systems140can communicate with one of the smart devices145A-Nto control the smart device in response to a request from cloud automated assistant component(s)120. For example, according to some techniques a user can provide, via one or more I/O components1111of a client device1101, a request to control a smart device, such as spoken input of “turn off the couch light” provided via microphone(s) of I/O components1111. The request (e.g., audio data that captures the spoken input, or locally generated text) can be transmitted by the client device1101to the cloud automated assistant component(s)120via the wide-area networks1051. The cloud automated assistant component(s)120can process the request to determine a smart device to be controlled based on the request, and transmit, via the wide-area networks1052, a control request to a respective one of the smart device systems140A-Nwhich, in turn transmits, via wide-area networks1053, corresponding command(s) to the smart device. However, as described herein such techniques present drawbacks such as high latency and/or excessive consumption of network resources.

In some implementations, the plurality of client computing devices1101-Nand smart devices145A-Ncan be associated with each other in various ways in order to facilitate performance of techniques described herein. For example, in some implementations, the plurality of client devices1101-Nand smart devices145A-Nmay be associated with each other by virtue of being communicatively coupled via one or more LANs and/or via one or more peer-to-peer networks. This may be the case, for instance, where plurality of client computing devices1101-Nare deployed across a particular area or environment, such as a home, a building, and so forth. Additionally or alternatively, in some implementations, plurality of client devices1101-Nand smart devices145A-Nmay be associated with each other by virtue of them being members of a coordinated ecosystem of client devices1101-Nand smart devices145A-Nthat are at least selectively accessible by one or more users (e.g., an individual, a family, employees of an organization, other predefined groups, etc.). In some of those implementations, the ecosystem of client devices1101-Nand smart devices1451-Ncan be manually and/or automatically associated with each other in the device topology152.

An instance of an automated assistant client of a client device1101-N, by way of its interactions with one or more cloud automated assistant components120, may form what appears to be, from a user's perspective, a logical instance of an automated assistant with which the user may engage in a human-to-computer dialog. For example, a user can engage with the same logical instance of an automated assistant using either client device1101and automated assistant client1171or using client device110Nand automated assistant client117N. While the particular instances of the automated assistant client1171and117Ncan vary (e.g., provide different smart device control for the same commands) and/or can provide user interface output via different I/O components1111and111Nand/or accept different user interface input via different I/O components1111and111N(e.g., I/O components1111can include a touch-screen, while I/O components111Ndo not), the user may still effectively engage with the same logical instance of the automated assistant. For the sakes of brevity and simplicity, the term “automated assistant”, as used herein will refer to the automated assistant client executing on a client device110and optionally to one or more cloud automated assistant components120(which may be shared amongst multiple automated assistant clients). Although two client devices1101and110Nof a coordinated ecosystem are illustrated inFIG.1, it is understood that many additional client devices can be included in the ecosystem. Further, it is understood that separate coordinated ecosystems of client devices will also be provided, each associated with different user(s) (e.g., account(s)) and/or environments, and that such separate coordinated ecosystems can also interact with cloud automated assistant component(s)120(but with interactions tailored to the account(s) of those separate ecosystems).

The client devices1101-Nmay include, for example, one or more of: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone assistant-centric interactive speaker, a standalone assistant-centric interactive display with speaker(s), a smart appliance such as a smart television, and/or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided.

As mentioned above, one or more of the client devices1101-Ncan at least selectively interface with the cloud automated assistant component(s)120in processing inputs and/or in generating outputs based on the inputs and/or in generating smart device control commands based on the inputs. The cloud automated assistant component(s)120can include a STT engine121, an NLU engine122, a TTS engine123, a cache generation engine125, and/or a cloud fulfillment engine127.

As described above, for a received request, at a client device1101-N, that are not resolvable by a respective NLU engine1131-Nand/or that lack a matching cache entry in a respective local cache1581-N, text and/or audio data corresponding to the request can be transmitted to the cloud automated assistant component(s)120. The cloud automated assistant component(s)120can utilize its more robust NLU engine122to generate semantic representation(s) based on processing text for the request. The processed text can be transmitted by the client device, or can be recognized text generated by STT engine121utilizing audio data transmitted by the client device. The generated semantic representation(s) can include cloud semantic representation(s) and/or locally interpretable semantic representation(s) as described herein. As also described herein, in generating the semantic representation(s) the NLU engine122can leverage the device topology152for the client devices1101-Nand smart devices145A-N.

The cache generation engine125can, responsive to the transmission from one of the client devices1101-N, generate a cache request that includes a cache entry that includes the generated semantic representation(s), and optionally includes the text (or representation thereof) and a mapping of the text to the semantic representation(s). The cache entry of the generated cache request can include additional and/or alternative content, such as responsive content to be rendered, protocol suites, adapters, and/or radio(s) to be utilized with locally interpretable semantic representation(s), and/or other content. The responsive content to be rendered can include, text, audio data (e.g., synthesized speech for responsive text, generated using TTS engine123), and/or graphics. The cache generation engine125transmits a corresponding cache request to the respective cache engine1181-Nof the respective client device1101-Nto cause storage of the cache entry in a respective local cache1581-N.

When the generated semantic representation(s) include cloud semantic representation(s), the cloud fulfillment engine127can process the cloud semantic representation(s) to generate corresponding control request(s), that it then transmit(s) to corresponding smart device system(s)140A-N. The smart device systems140A-Ncan, responsive to the control request(s), generate and transmit corresponding control commands to corresponding smart device(s)145A-N.

When the generated semantic representation(s) include locally interpretable semantic representation(s), the respective client device110A-Ncan utilize the locally interpretable semantic interpretation(s) of the cache request in generating corresponding local control commands and locally transmitting those local control commands (e.g., using a respective local module116A-N). Alternatively, the cloud automated assistant component(s)120can provide the locally interpretable semantic representation(s) in a separate transmission, and the respective client device110A-Ncan utilize the locally interpretable semantic interpretation(s) of the separate transmission in generating corresponding local control commands and locally transmitting those local control commands (e.g., using a respective local module116A-N). As yet another alternative, the cloud automated assistant component(s)120can themselves generate the local control command(s), and transmit the local control command(s) to the respective client device110A-Nfor locally transmitting those control command(s).

As also described above, for a received request, at a client device1101-N, that matches a cache entry, in a respective local cache1581-N, that includes a cloud semantic representation, the cloud semantic representation can be transmitted to the cloud automated assistant component(s)120(e.g., in lieu of corresponding text and/or audio data). The cloud fulfillment engine127can process the received cloud semantic representation(s) to generate corresponding control request(s), that it then transmit(s) to corresponding smart device system(s)140A-N. The smart device systems140A-Ncan, responsive to the control request(s), generate and transmit corresponding control commands to corresponding smart device(s)145A-N.

Additional description of various components ofFIG.1is now provided with reference to the additional figures.FIG.2depicts a home floorplan that includes a plurality of rooms,250,252,254,256,258,260, and262. A plurality of client devices1101-3are deployed throughout at least some of the rooms. Each of the client devices1101-3can optionally implement an instance of an automated assistant client configured with selected aspects of the present disclosure and can optionally include one or more input devices, such as microphones, touch-screens, etc. and/or one or more output devices such as speakers, displays, etc. For example, a first client device1101taking the form of an interactive standalone speaker is deployed in room254, which in this example is a bedroom. A second client device1102taking the form of a standalone interactive speaker and display device (e.g., display screen, projector, etc.) is deployed in room252, which in this example is a living room. A third client device1103, also taking the form of an interactive standalone speaker, is deployed in room256.

The plurality of client devices1101-3may be communicatively coupled with each other and/or other resources (e.g., smart devices and the Internet) via a wireless router101, depicted in room252, and/or a local mesh network. Additionally, other client devices—particularly mobile devices such as smart phones, tablets, laptops, wearable devices, etc.—may also be present, e.g., carried by one or more persons (e.g., user103) in the home and may or may not also be connected to the same LAN. It should be understood that the configuration of client devices depicted inFIG.2is just one example; more or fewer and/or different client devices may be deployed across any number of other rooms and/or areas other than a home.

Further depicted inFIG.2are a plurality of smart devices. The smart devices include a smart light145A1. The smart light145A1is controllable by a first remote smart device system140A. The smart devices further include smart lights145B1and145B2that are controllable by a second remote smart device system140B. Second remote smart device system140Bcan be controlled by a part that is separate from a party that controls first remote smart device system140A. The smart devices further include smart thermostat145C1that is locally controllable, at least by the second client device1102. For example, the smart thermostat145C1can be controlled via control commands that conform to a protocol suite of the smart thermostat145C1, and that are provided by the second client device1102via a Bluetooth connection between the smart thermostat145C1and the second client device1103. It should be understood that the configuration of smart devices145depicted in FIG. is just one example; more or fewer and/or different smart devices may be deployed across any number of other rooms and/or areas other than a home.

FIG.2and the above description ofFIG.2will now be utilized in describing various aspects ofFIGS.3A,3B,4A,4B,5A, and5B.

FIG.3Aillustrates an example of how a spoken utterance352of “brighten lights a bit”, received at the second assistant client device1102ofFIG.2, can be processed when the second assistant client device1102lacks a cache entry that matches recognized text352A of the spoken utterance352(where recognized text352A is generated using an on-device STT engine)—and optionally when a local NLU engine of the second assistant client device1102is unable to validly process the recognized text352A. InFIG.3A, the second assistant client device1102transmits, to the cloud automated assistant component(s)120, the recognized text352A. The transmission of the recognized text352A can be responsive to the second assistant client device1102determining it lacks a locally stored cache entry that matches recognized text352A of the spoken utterance352—and optionally responsive to a local NLU engine of the second assistant client device1102failing to validly process the recognized text352A.

In response to receiving the recognized text352A, the cloud automated assistant component(s)120generate a cloud semantic representation354of the recognized text352A. The cloud semantic representation354can be generated based on the recognized text being received from the second assistant client device1102and based on reference to a remotely stored device topology. In generating the cloud semantic representation, the cloud automated assistant component(s)120can resolve “the lights” in the recognized text352A to particular lights. For example, the cloud automated assistant component(s)120can utilize an account identifier, received with the transmission of text352A, to identify a corresponding remotely stored device topology for the account identifier. Further, the cloud automated assistant component(s)120can utilize an identifier of the second assistant client device1102, received with the text352A, to identify the second assistant client device1102in the device topology. Yet further, the cloud automated assistant component(s)120can resolve “lights”, in the text352A, to lights145B1and145B2based on those lights being mapped, in the device topology, as default lights for the second assistant client device1102. Lights145B1and145B2can be mapped as defaults for the second assistant client device1102based on prior user interface input and/or based on them all being assigned to an identifier of room252, in the device topology.

In generating the cloud semantic representation, the cloud automated assistant component(s)120can also resolve “brighten . . . a bit” to mean “2 of 5” (e.g., 40%) brighter than current brightness intensity level(s)”. This can be based on reference to one or more remotely stored model(s) and/or rule(s) accessible to the cloud automated assistant component(s)120. The cloud semantic representation354can be generated based on these resolutions and can be, for example and as illustrated in cache request356, [device(s)=145B1and145B2; brighter, intensity relative 2 of 5], where “145B1and145B2” are unique identifiers for the corresponding lights, and where “brighter, intensity relative 2 of 5” indicates the lights are to be brightened to a degree that is 2 of 5 more relative to their current intensity.

The cloud automated assistant component(s)120utilize the generated cloud semantic representation354to generate a corresponding control request. The cloud automated assistant component(s)120transmit the control request to a smart device system140Bthat corresponds to the lights145B1and145B2. In response, the smart device system140Bcan generate corresponding control commands, and provide those control commands to the lights145B1and145B2to cause them to “brighten a bit” (i.e., increase relative brightness by 40%, as indicated by the cloud semantic representation)

Further, the cloud automated assistant component(s)120generate and transmit, to the client device1102, a cache request356that includes the cloud semantic representation354, and that optionally includes the recognized text352A and/or a mapping of the recognized text to the cloud semantic representation354. Responsive to receiving the cache request356, the client device1102stores a corresponding cache entry that includes a mapping of the recognized text352A to the cloud semantic representation354included in the cache request356.

FIG.3Billustrates an example of how another instance of the same spoken utterance352ofFIG.3A, also received at the second assistant client device1102, can be processed differently after the cache entry with the cloud semantic representation354(FIG.3A) is stored at the at the second assistant client device1102inFIG.3A.

InFIG.3B, the second assistant client device1102processes audio data, that captures the spoken utterance352, to generate recognized text. Further, the second assistant client device1102determines that the recognized text matches text of the cache entry stored in the local cache of the second assistant client device1102inFIG.3A. In response, the second assistant client device1102transmits, to the cloud automated assistant component(s)120, the cloud semantic representation of the matching cache entry. The cloud automated assistant component(s)120can then process the cloud semantic representation354directly, to generate a corresponding control request and transmit the control request to smart device system140B. In response, the smart device system140Bcan generate corresponding control commands, and provide those control commands to the lights145B1and145B2to cause them to “brighten a bit”. In these and other manners, resources at the cloud automated assistant component(s) can be conserved by preventing another occurrence of generating the cloud semantic representation. Further, the control request can be generated and transmitted more quickly, reducing latency in effectuating the corresponding change to the lights145B1and145B2. It is noted that the control request (and as a result, the control commands) ofFIG.3Bcan vary from that ofFIG.3A, despite being generated based on the same semantic representation. This can be based on the cloud state change of the cloud semantic representation being a relative state change, and the “current” state of the lights145B1and145B2, at a time of generating the control request, potentially varying betweenFIGS.3A and3B.

FIG.4Aillustrates an example of how the same spoken utterance (“brighten the lights a bit”) ofFIGS.3A and3B, received at first assistant client device1102ofFIG.2, can be processed when the first assistant client device1102lacks a cache entry that matches text of the spoken utterance. InFIG.4A, the user103provides the spoken utterance452of “brighten the lights a bit”). Although spoken utterance452is the same spoken utterance as spoken utterance352ofFIGS.3A and3B, it is labeled as452inFIGS.4A and4Bfor numbering consistency.

InFIG.4A, the first assistant client device1101transmits, to the cloud automated assistant component(s)120, recognized text452A that is generated by processing, using a local STT engine, audio data that captures the spoken utterance452. The transmission of the recognized text452A can be responsive to the first assistant client device1101determining it lacks a locally stored cache entry that matches recognized text452A of the spoken utterance452—and optionally responsive to a local NLU engine of the first assistant client device1101failing to validly process the recognized text452A.

In response to receiving the recognized text452A, the cloud automated assistant component(s)120generate a cloud semantic representation454of the recognized text452A. The cloud semantic representation454can be, for example, and as illustrated in cache request456, [device(s)=145A1; brighter, intensity relative 2 of 5], where “145A1” is a unique identifier for the corresponding light, and where “brighter, intensity relative 2 of 5” indicates the lights are to be brightened to a degree that is 2 of 5 more relative to their current intensity. The cloud semantic representation454differs from the cloud semantic representation354ofFIG.3Ain that it includes an identifier of the light145A1in lieu of identifiers of the lights145B1and145B2. This can be based on the cloud automated assistant component(s)120utilizing an identifier of the first assistant client device1101, received with the text452A, to identify the first assistant client device1101, and resolving “lights”, in the text452A, to light145A1based on that light being mapped, in the device topology, as a default light for the first assistant client device1101. Accordingly, despite the same spoken utterance being received inFIGS.3Aand4A, different cloud semantic representations can be generated based at least in part on the spoken utterance being received from different assistant client devices.

The cloud automated assistant component(s)120utilize the generated cloud semantic representation454to generate a corresponding control request. The cloud automated assistant component(s)120transmit the control request to a smart device system140Athat corresponds to the light145A1of the semantic representation454. In response, the smart device system140Acan generate corresponding control command(s), and provide those control command(s) to the light145A1to cause it to “brighten a bit”.

Further, the cloud automated assistant component(s)120generate and transmit, to the first client device1101, a cache request456that includes the cloud semantic representation454, and that optionally includes the recognized text452A and/or a mapping of the recognized text to the cloud semantic representation454. Responsive to receiving the cache request456, the first client device1101stores a corresponding cache entry that includes a mapping of the recognized text452A to the cloud semantic representation454included in the cache request456.

FIG.4Billustrates an example of how another instance of the same spoken utterance452ofFIG.4A, received at the first assistant client device1101, can be processed differently after a cache entry with the cloud semantic representation454(FIG.4A) is stored at the first assistant client device1101inFIG.4A.

InFIG.4B, the first assistant client device1101processes audio data, that captures the spoken utterance452, to generate recognized text. Further, the first assistant client device1101determines that the recognized text matches text of the cache entry stored in the local cache of the first assistant client device1101inFIG.4A. In response, the first assistant client device1101transmits, to the cloud automated assistant component(s)120, the cloud semantic representation454of the matching cache entry. The cloud automated assistant component(s)120can then process the cloud semantic representation454directly, to generate a corresponding control request and transmit the control request to smart device system140A. In response, the smart device system140Acan generate corresponding control command(s), and provide those control commands to the light145A1to cause it to “brighten a bit”.

FIG.5Aillustrates an example of how another spoken utterance552of “decrease temp a bit”, received at the at the second assistant client device1102ofFIG.2, can be processed when the second assistant client device1102lacks a cache entry that matches recognized text552A (generated using an on-device STT engine) of the spoken utterance552—and optionally when a local NLU engine of the second assistant client device1102is unable to validly process the recognized text452A.

InFIG.5A, the second assistant client device1102transmits, to the cloud automated assistant component(s)120, the recognized text552A. The transmission of the recognized text552A can be responsive to the second assistant client device1102determining it lacks a locally stored cache entry that matches recognized text552A of the spoken utterance552—and optionally responsive to a local NLU engine of the second assistant client device1102failing to validly process the recognized text552A.

In response to receiving the recognized text552A, the cloud automated assistant component(s)120generate a locally interpretable semantic representation554of the recognized text552A. In some implementations, the cloud automated assistant component(s)120generate the locally interpretable semantic representation554(e.g., in lieu of or in addition to a cloud based one), responsive to determining that the smart device inferentially referenced by the recognized text552A (thermostat145C1) is locally controllable by the second assistant client device1102and/or another client device in network communication with the second assistant client device1102.

The locally interpretable semantic representation554can be generated based on reference to a remotely stored device topology to determine a “decrease temperature” request is one directed to thermostat145C1(e.g., it may be the only smart device in the device topology that has “decrease temperature” as a supported state change). Accordingly, the text552A can be resolved to be inferentially referencing the thermostat145C1as a target smart device. In generating the cloud semantic representation, the cloud automated assistant component(s)120can also resolve “decrease . . . a bit” to mean “2 degrees less than a current set point”. This can be based on reference to one or more remotely stored model(s) and/or rule(s) accessible to the cloud automated assistant component(s)120. The locally interpretable semantic representation554can be generated based on these resolutions and can further be generated to enable the client device to include a relative state change representation that dictates how to locally resolve the target set point for “2 degrees less than a current set point”. For example, the relative state change representation can be “temp set=(current temp−2 degrees)”.

The locally interpretable semantic representation556can optionally additionally include: an indication of the adapter that is to be utilized in generating corresponding local control command(s) (e.g., an adapter specific to a manufacturer of the smart device); an indication of the channel (e.g., Bluetooth or Wi-Fi) that is to be utilized to transmit the local control command(s); and/or an indication of the protocol suite that is to be utilized in generating control command(s) based on the locally interpretable semantic representation556. For example, and as illustrated in cache request556, the locally interpretable semantic representation can be [device(s)=145C1; temp set=(current temp−2 degrees); adapter/channel=C], where “145C1” is a unique identifiers for the thermostat145C1(e.g., a local address for the thermostat145C1), where “temp set=(current temp−2 degrees)” is a current-state dependent state change representation that is locally interpretable to resolve current state changes; and where “adapter/channel=C” indicates the adapter and/or protocol suite to be utilized in generating local control command(s) (e.g., after determining a current state change) and/or the channel to be utilized to transmit the local control command(s).

The cloud automated assistant component(s)120generate and transmit, to the second client device1102, a cache request556that includes the cloud semantic representation554, and that optionally includes the recognized text552A and/or a mapping of the recognized text to the cloud semantic representation554. Responsive to receiving the cache request556, the second client device1102stores a corresponding cache entry that includes a mapping of the recognized text552A to the cloud semantic representation554included in the cache request556.

In some implementations, and as indicated by dashed lines between the second client device1102and the thermostat145C1, the second client device1102can utilize the locally interpretable semantic representation, of the cache request (or from a separate transmission from the cloud automated assistant component(s)120), to locally generate local control commands for the thermostat145C1, and to locally transmit them to the thermostat145C1in response to the spoken utterance ofFIG.5A.

In some alternative implementations, the cloud automated assistant component(s)120generate a corresponding cloud semantic representation based on the recognized text552A. Further, the cloud automated assistant component(s)120can utilize the separate cloud semantic representation to generate a corresponding control request. The cloud automated assistant component(s)120transmit the control request to a smart device system140Cthat corresponds to the thermostat145C1. In response, the smart device system140Ccan generate corresponding control command(s), and provide those control command(s) to the thermostat145C1to cause its set point to “decrease a bit” (i.e., decrease by 2 degrees, as indicated by the cloud semantic representation).

FIG.5Billustrates an example of how another instance of the same spoken utterance552ofFIG.5A, received at the second assistant client device1102, can be processed differently after a cache entry with the locally interpretable semantic representation554(FIG.5A) is stored at the second assistant client device1102inFIG.5A.

InFIG.5B, the second assistant client device1102processes audio data, that captures the spoken utterance552, to generate recognized text. Further, the second assistant client device1102determines that the recognized text matches text of the cache entry stored in the local cache of the second assistant client device1102inFIG.4A. In response, the second assistant client device1102identifies the locally interpretable semantic representation554of the cache entry, and uses the locally interpretable semantic representation554to generate corresponding control commands. It is noted that the generated control commands ofFIG.5Bcan differ from those ofFIG.5Asince the state representation, of the locally interpretable semantic representation554, is a current-state dependent state representation—and the “current” temperature set point of the thermostat145C1can vary betweenFIGS.5A and5B. For example, inFIG.5Bthe second assistant client device1102can resolve a target set point by subtracting 2 degrees from the current set point, as dictated by the relative state change of the locally interpretable semantic representation554. The second assistant client device1102can then process the target set point, utilizing a corresponding adapter, to generate the control commands, and transmit the control commands to an address of the thermostat145C1, that is optionally included in the locally interpretable semantic representation554. In response to receiving the control commands, the thermostat145C1can decrease its set point by 2 degrees.

FIG.6is a flow chart illustrating an example method600that can be implemented by one or more processors of assistant client devices, according to various implementations disclosed herein.

At block602, an assistant client device processes audio data, that captures a spoken utterance, to generate recognized text for the spoken utterance. In other implementations, instead of processing audio data to generate recognized text, the assistant client device can identify text based on it being typed directly (e.g., using a virtual keyboard) or being provided based on it corresponding to a hardware or software button press (e.g., text of “dim the lights” being mapped to a graphical software button labeled “dim”).

At block604, the assistant client device determines whether the text matches a local cache entry. If so, the client device proceeds to block606, and selects semantic representation(s) that are from the matching local cache entry. At block608, the system then uses the semantic representation(s). Block608can include sub-blocks608A and/or608B. At sub-block608A, the client device locally processes any of the selected semantic representations, that are locally interpretable semantic representations, to generate local control command(s), and transmit(s) the control command(s) via local channel(s). At sub-block608B, the client device transmits any of the selected semantic representations, that are cloud semantic representations, to a remote system. For example, they can be transmitted to a remote system that can then perform blocks812and814of method800(FIG.8, described below). In some implementations block606can include additional sub-block(s), such as a sub-block where the assistant client device renders any responsive content that is included in the local cache entry. For example, the local cache entry can include responsive text, and the assistant client device can perform local TTS processing on the responsive text, and audibly render the resulting synthesized speech. As another example, the local cache entry can additionally or alternatively include responsive graphic(s), and the graphic(s) can be visually rendered on a display of the assistant client device.

If the decision at block604is no, the assistant client device proceeds to optional block610and determines whether the text is resolvable with local NLU (e.g., using a local NLU engine and local model(s)). If so, the system proceeds to block612, generates semantic representation(s) (locally interpretable and/or cloud), then proceeds to block608and uses the generated semantic representation(s).

If the decision at block610is no (or the decision at604is no and610is omitted), the client device proceeds to block614and transmits the audio data and/or the text to a remote assistant system. Optionally, block614can occur before and/or during performance of blocks604and/or610—and further processing at the remote system optionally halted (e.g., responsive to a halt transmission from the client device) responsive to a “yes” determination at either of blocks604and610. For example, if block614occurs before and/or during performance of blocks604and/or610, and the assistant client device makes a “yes” determination at blocks604and/or610, it can transmit a halt request to the remote assistant system to cause halting of any further processing, by the remote assistant system, related to the transmitted audio data and/or text. In these and other manners processing at the remote assistant system can be initiated more quickly to reduce latency if the determinations at blocks604and/or610is “no”, while also enabling halting of such processing if the decision is “yes”, to conserve remote assistant system resources.

At block616, the client device receives, from the remote assistant system, a cache request that includes semantic representation(s) for the text, and optionally the text itself (and/or a representation thereof). Block616optionally includes sub-block616A, where the assistant client device optionally proceeds to block608A to process one or more locally interpretable semantic representation(s) of the cache request (or a separate transmission), if any. Put another way, at block616the assistant client device can immediately act upon any locally interpretable semantic representation(s) of the cache request to effectuate any local smart device action(s) responsive to the current spoken utterance or other current user input.

At block618, the client device stores, in a local cache and responsive to receiving the cache request, a cache entry that includes a mapping of the text to the semantic representation(s) of the cache request.

Block618optionally includes sub-block618A, which can occur at a time near block618, or at a later time. In sub-block618A, the client device optionally assigns the text, of the cache entry, for invocation-free action. For example, the client device can assign the text as a “hot phrase” as described herein. Assigning the text as a “hot phrase” can optionally be based on determining that one or more criteria are satisfied. When recognized text, from on-device speech recognition of audio data that is occurring independent of any explicit automated assistant invocation, is determined to match a hot phrase, the assistant client device can automatically initiate one or more corresponding action(s) based on the text (e.g., action(s) based on the semantic representation(s) of a corresponding cache entry).

FIG.7is a flow chart illustrating another example method700that can be implemented by one or more processors of assistant client devices, according to various implementations disclosed herein. Method700illustrates a particular implementation of method600ofFIG.6, where it is determined that text matches text of a cache entry (i.e., a “yes” in block604ofFIG.6), and where semantic representation(s) of the cache entry include locally interpretable semantic representation(s).

At block702, an assistant client device processes audio data, that captures a spoken utterance, to generate recognized text for the spoken utterance. In other implementations, instead of processing audio data to generate recognized text, the assistant client device can identify text based on it being typed directly (e.g., using a virtual keyboard) or being provided based on it corresponding to a hardware or software button press (e.g., text of “dim the lights” being mapped to a graphical software button labeled “dim”).

At block704, the assistant client device determines that the text matches text of a cache entry. Block704optionally includes sub-block704A, where the client device optionally determines the text is assigned (e.g., at the assistant client device) to invocation-free action. For example, when the processing of block702is occurring without an explicit automated assistant invocation, block704A can optionally be performed. Moreover, in such an example, one or more subsequent blocks of method700may only be performed if it is determined the text is assigned (e.g., locally at the client device) to invocation-free action. Put another way, block704A can be performed when the processing of block702is occurring without an explicit automated assistant invocation, and when text is determined to not be assigned to invocation-free action, it can be discarded and without performing further blocks of method700based on the text—whereas when text is determined to be assigned to invocation-free action, further blocks of method700will be performed based on the text.

At block706, the client device selects a locally interpretable semantic representation, based on it being mapped, in the cache entry, to the text of the cache entry.

At block708, the client device determines whether a state change, of the locally interpretable semantic representation, is a current-state dependent state change. If not, the client device proceeds to block712and uses the state change of the locally interpretable semantic representation. For instance, if the state change indicates a fixed value, that fixed value can be utilized as the state change. If the determination at block708is “yes”, the client device proceeds to block710, and resolves the state change based on current state value(s). For example, if the state change is current state-dependent, it can be resolved by identifying the current state of one or more smart devices, then determining the state change based on the identified current state(s). For instance, if the state change is “set point=(current temperature+3 degrees)”, it can be resolved by identifying the “current temperature”.

At block712, the client device generates local control command(s) using the state change, and optionally using an adapter identified based on the cache entry.

At block714, the client device transmits the local command(s) over a local channel that is optionally selected based on the cache entry (e.g., explicitly identified in the cache entry, or selected based on a protocol suite, smart device, and/or radio/channel identified in the cache entry). The transmitted local command(s) can be addressed to smart device(s) based on those addresses being identified in the locally interpretable semantic representation, or being resolvable therefrom.

At block718, the client device determines if there are any additional unprocessed semantic representations in the cache entry. If not, the client device proceeds to block720and method700ends. If so, the client device proceeds to block722and determines whether a next unprocessed semantic representation is locally interpretable. If so, the client device proceeds back to block708. If not (i.e., it is a cloud semantic representation), the client device proceeds to block608B of method600ofFIG.6, then returns back to block718after performing block608B.

FIG.8is a flow chart illustrating an example method800that can be implemented by one or more processors of a remote assistant system, according to various implementations disclosed herein.

At block802, the system receives a request from an assistant client device. At block804, the system determines whether the request is a semantic cloud representation. If so, the system performs blocks812and814(described below) of method800, without performing any other blocks of method800.

If not, the system proceeds to block808and generates, based on a representation of a spoken utterance in a request, semantic representation(s) of the spoken utterance. In other implementations, the representation(s) can be of a typed utterance, or of text mapped to a press of a hardware element or software graphical element.

At optional block810, the system generates confirmatory content for rendering at the assistant client device in response to the request.

At optional block812, the system generates control request(s) based on a cloud semantic representation generated based on the request. Optional block812can include optional block812A, where the system generates the control request(s) based on current state(s) of smart device(s), if a state representation of the semantic representation is a relative/current state-dependent state representation.

At optional block814, the system transmit(s) the generated control request(s) to smart device remote system(s).

At block816, the system generates a cache request that includes semantic representation(s) (locally interpretable and/or cloud) and optionally includes a mapping of the text to the semantic representation(s).

At block818, the system transmits the cache request to the assistant client device. Block818optionally includes sub-block818A, where the system, if the semantic representation of the cache request is a locally interpretable semantic representation, optionally generates a separate request for implementation of the locally interpretable semantic representation, responsive to the current request.

FIG.9is a block diagram of an example computing device910that may optionally be utilized to perform one or more aspects of techniques described herein. Computing device910typically includes at least one processor914which communicates with a number of peripheral devices via bus subsystem912. These peripheral devices may include a storage subsystem924, including, for example, a memory subsystem925and a file storage subsystem926, user interface output devices920, user interface input devices922, and a network interface subsystem916. The input and output devices allow user interaction with computing device910. Network interface subsystem916provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

User interface input devices922may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computing device910or onto a communication network.

User interface output devices920may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computing device910to the user or to another machine or computing device.

Storage subsystem924stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem924may include the logic to perform selected aspects of one or more methods described herein.

These software modules are generally executed by processor914alone or in combination with other processors. Memory925used in the storage subsystem924can include a number of memories including a main random access memory (RAM)930for storage of instructions and data during program execution and a read only memory (ROM)932in which fixed instructions are stored. A file storage subsystem926can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem926in the storage subsystem924, or in other machines accessible by the processor(s)914.

Bus subsystem912provides a mechanism for letting the various components and subsystems of computing device910communicate with each other as intended. Although bus subsystem912is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

Computing device910can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device910depicted inFIG.9is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device910are possible having more or fewer components than the computing device depicted inFIG.9.

In situations in which certain implementations discussed herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how the information is collected about the user, stored and used. That is, the systems and methods discussed herein collect, store and/or use user personal information only upon receiving explicit authorization from the relevant users to do so.

In some implementations, a method is provided that includes receiving, at a remote assistant system and from an assistant client device, a representation of a spoken utterance captured at the assistant client device. The spoken utterance is a request related to one or more states of one or more smart devices that are linked with the assistant client device. As one example, the request can be a request to alter the state(s) of the smart device(s) (i.e., cause the state(s) to transition from current state value(s) to new state value(s)). The representation of the spoken utterance comprises audio data that captures the spoken utterance and/or text, of the spoken utterance, that is generated at the client device utilizing a speech-to-text model stored locally on the client device. The method further includes, at the remote system, and responsive to receiving the representation of the spoken utterance: generating, based on the representation of the spoken utterance, a semantic representation of the spoken utterance; generating, based on the semantic representation of the spoken utterance, at least one control request to transmit to at least one smart device remote system that controls the one or more smart devices, where the at least one control request differs from the semantic representation; transmitting the at least one control request to the at least one smart device remote system to cause the at least one smart device remote system to provide one or more corresponding commands to the one or more smart devices; and transmitting, to the assistant client device, a cache request that includes the semantic representation. Transmitting the cache request to the assistant client device causes the assistant client device to store, in a cache on the assistant client device: a cache entry that includes a mapping of the text to the semantic representation.

These and other implementations of the technology can optionally include one or more of the following features.

In some implementations, the method further includes, subsequent to transmitting the cache request to the assistant client device: capturing, at the assistant client device, additional audio data that captures an additional spoken utterance; processing, using the speech-to-text model stored locally on the assistant client device, the additional audio data to generate current text that corresponds to the additional spoken utterance; determining that the current text matches the text of the cache entry; and in response to determining that the current text matches the text of the cache entry, and in response to the cache entry including the mapping of the text to the semantic representation: transmitting the semantic representation to the remote assistant system. In some of those implementations, processing the additional audio data to generate the current text is performed without detection of an explicit automated assistant invocation at the assistant client device. In some versions of those implementations, transmitting the semantic representation to the remote assistant system is further in response to determining that the current text satisfies one or more criteria for enabling matching text to be acted upon without explicit automated assistant invocation at the assistant client device. In some additional or alternative versions, the method further includes: receiving, at the remote assistant system, the semantic representation transmitted from the assistant client device; generating, based on the semantic representation, at least one alternative control request to transmit to the at least one smart device remote system that controls the one or more smart devices, where the at least one alternative control request differs from the at least one control request; and transmitting the at least one alternative control request to the at least one smart device remote system to cause the at least one smart device remote system to alter the one or more states of the one or more smart devices. Optionally, in those alternative or additional implementations: generating the at least one control request is further based on at least one current state of the one or more smart devices at a time of generating the at least one control request; generating the at least one alternative control request is further based on at least one alternative current state of the one or more smart devices at a different time of generating the at least one alternative control request; the at least one alternative control request differs from the at least one control request based on the at least one current state differing from the at least one alternative current state.

In some implementations, generating the semantic representation of the spoken utterance is further based on a device topology for an account of the assistant client device, where the device topology defines the assistant client device, the one or more smart devices, additional assistant client devices, and additional smart devices. In some of those implementations, generating the semantic representation of the spoken utterance further based on the device topology includes resolving, based on the device topology, that an ambiguous term, in the text of the spoken utterance, references the one or more smart devices.

In some implementations, the semantic representation includes: a state change representation of the one or more state changes and, for each of the one or more smart devices, a corresponding unique identifier. In some versions of those implementations the state change representation is a relative state change representation that defines the one or more state changes in a relative manner, and not in an absolute manner. In some additional or alternative versions of those implementations, the method further includes: determining, at the remote assistant system or the assistant client device, an alteration to the device topology; and in response to determining the alteration to the device topology: causing the cache entry to be cleared from the cache of the assistant device. Determining the alteration to the device topology can optionally include: determining that the alteration is to one or more properties defined, in the device topology, for the assistant client device, the one or more smart devices, and/or one or more rooms assigned to the assistant client device and/or the one or more smart devices; and, optionally, causing the cache entry to be cleared from the cache can be further in response to determining the cache entry is affected by the one or more properties.

In some implementations, the method further includes receiving, at the remote assistant system and from an additional assistant client device, an additional representation of the spoken utterance. The spoken utterance is captured at the additional assistant client device, and the additional assistant client device and the assistant client device are both members of a common device topology. In some of those implementations the method further includes, at the remote system, and responsive to receiving the additional representation of the spoken utterance: generating, based on the representation of the spoken utterance, an alternative semantic representation of the spoken utterance, where the alternative semantic representation of the spoken utterance differs from the semantic representation based on the additional representation being captured at the additional assistant client device and the representation being captured at the assistant client device; and transmitting, to the additional assistant client device, an additional cache request that includes the alternative semantic representation. Transmitting the additional cache request to the additional assistant client device causes the additional assistant client device to store, in an additional cache on the additional assistant client device: an additional cache entry that includes a mapping of the text to the alternative semantic representation.

In some implementations, a method is provided that includes receiving, at a remote assistant system and from an assistant client device, a representation of a spoken utterance captured at the assistant client device. The spoken utterance is a request for altering at least a given state of a given smart device that is linked with the assistant client device. The representation of the spoken utterance includes audio data that captures the spoken utterance and/or text, of the spoken utterance, that is generated at the client device utilizing a speech-to-text model stored locally on the client device. The method further includes at the remote system, and responsive to receiving the representation of the spoken utterance: determining that the given smart device is controllable locally by the assistant client device and/or by an additional client device having a local connection to the assistant client device; generating a semantic representation, of the spoken utterance, that includes a locally interpretable semantic representation; and transmitting, to the assistant client device, a cache request that comprises the semantic representation. The locally interpretable semantic representation is locally interpretable, by the assistant client device and/or the additional client device, to generate a corresponding control command that is transmittable over a local channel to cause the altering of at least the given state of the given smart device. Generating the locally interpretable semantic representation for the at least one smart device includes including the locally interpretable semantic representation, in the semantic representation, responsive to determining that the given smart device is controllable locally. Transmitting the cache request to the assistant client device causes the assistant client device to store, in a cache on the assistant client device: a cache entry that includes a mapping of the text to the semantic representation.

These and other implementations of the technology can optionally include one or more of the following features.

In some implementations, the method further includes, subsequent to transmitting the cache request to the assistant client device: capturing, at the assistant client device, additional audio data that captures an additional spoken utterance; processing, using a voice-to-text model stored locally on the assistant client device, the additional audio data to generate current text that corresponds to the additional spoken utterance; determining that the current text matches the text of the cache entry; and in response to determining that the current text matches the text of the cache entry, and in response to the cache entry including the mapping of the text to the semantic representation: processing, at the assistant client device, the locally interpretable semantic representation to generate a given control command, and transmitting, via a local channel, the given control command to cause the altering of at least the given state of the given smart device. In some versions of those implementations, the locally interpretable semantic representation includes: an identifier of the given smart device; and a current-state dependent state change representation that defines the altering, of at least the given state of the given smart device, relative to a corresponding current value for the given state of the given smart device. In some of those versions, processing the locally interpretable semantic representation to generate the given control command includes: determining, at the assistant client device, a current value for the given state of the given smart device; generating the given control command based on applying the current value to the current-state dependent state change representation. Optionally, the locally interpretable semantic representation further includes an indication of a manufacturer of the smart device, and generating the given control command optionally includes utilizing an adapter, stored locally at the assistant client device, responsive to the adapter being assigned to the indication of the manufacturer of the smart device. Optionally, the method further includes selecting the local channel, for transmitting the given control command, based on the local channel being directly or indirectly indicated in the locally interpretable semantic representation. For example, based on the local channel being assigned to an indication of the manufacturer of the smart device that is included in the locally interpretable semantic representation and/or being assigned to a protocol suite that is included in the locally interpretable semantic representation.

In some implementations, processing the additional audio data to generate the current text is performed without detection of an explicit automated assistant invocation at the assistant client device. In some of those implementations, transmitting the given control command is further in response to determining that the current text satisfies one or more criteria for enabling matching text to be acted upon without explicit automated assistant invocation at the assistant client device.

In some implementations, generating the semantic representation of the spoken utterance is further based on a device topology for an account of the assistant client device. The device topology defines the assistant client device, the given smart device, additional assistant client devices, and additional smart devices. In some of those implementations, generating the semantic representation of the spoken utterance includes resolving that an ambiguous term, in the text of the spoken utterance, references the given smart device.

In some implementations, the method further includes: determining, at the remote assistant system or the assistant client device, an alteration to the device topology; and in response to determining the alteration to the device topology: causing the cache entry to be cleared from the cache of the assistant client device. In some of those implementations, determining the alteration to the device topology includes: determining that the alteration is to one or more properties defined, in the device topology, for the assistant client device, the given smart device, and/or one or more rooms assigned to the assistant client device and/or the given smart device; and causing the cache entry to be cleared from the cache is further in response to determining the cache entry is affected by the one or more properties.

In some implementations, a method is provided that includes storing, in a cache on the client device: a cache entry that includes a mapping of text to a semantic representation. The semantic representation includes a locally interpretable semantic representation that is locally interpretable by the client device. The method further includes capturing, at the client device, additional audio data that captures a spoken utterance. The method further includes processing, using a voice-to-text model stored locally on the client device, the additional audio data to generate current text that corresponds to the spoken utterance. The method further includes determining that the current text matches the text of the cache entry. The method further includes in response to determining that the current text matches the text of the cache entry, and in response to the cache entry including the mapping of the text to the semantic representation: processing, at the client device, the locally interpretable semantic representation to generate a given control command; and transmitting, via a local channel, the given control command to cause the altering of at least the given state of the given smart device.