Patent Description:
Some automated assistants can act as an interface between a user and an application with which a user is intending to control. For example, a user can provide a command to initialize an application that is separate from the automated assistant. The automated assistant can communicate the command to the application and thereafter receive a response from the application, which can then be provided back to the user. However, when multiple applications are available for the automated assistant to select from, the automated assistant can exhibit latency when searching for a suitable application to invoke. Furthermore, computational resources and network resources can be wasted when an automated assistant invokes an incorrect application. In order to resolve such issues, an automated assistant can be pre-loaded with responses for the user to more readily receive. Unfortunately, a device that hosts the automated assistant can operate inefficiently when loaded with an extensive amount of responses at one time. Furthermore, the automated assistant can exhibit latency when tasked with repeatedly accessing content over a network each time the automated assistant is queried by a user.

Document <CIT> describes a method for invoking an agent during a dialog between a user and an automated assistant directed to receiving, during a human-to-automated assistant dialog, natural language input of the user that indicates a desire to engage an agent, but that fails to indicate a particular agent to be engaged. Further directed to selecting a particular agent from a plurality of available agents, and transmitting an invocation request to the selected particular agent.

Document <CIT> is directed to natural language processing. An input interface of a computing device receives input (e.g., speech input) and generates a digital signal corresponding to that input. Text corresponding to the digital signal is obtained, and the text is processed using each of a context-free and a context-specific linguistic model to generate linguistic processing results for that text. The text and linguistic processing results may be processed using a natural-language understanding (NLU) model to generate an NLU recognition result corresponding to the input received at the input interface. The text and the linguistic processing results may also be annotated and used to train a NLU model. The linguistic processing results may relate to, e.g., the tokenization of portions of the text, the normalization of portions of the text, sequences of normalizations for portions of the text, and rankings and prioritization of the linguistic processing results.

The present disclosure is generally directed to methods for providing context specific schema files that allow an automated assistant to facilitate human-to-computer dialogs between a user and an application (sometimes referred to as a "third party application") that is separate from the automated assistant. A context specific schema file can be provided to one or more devices that direct the activities of the automated assistant. The context specific schema files can identify entry points through which a user can access an application through an automated assistant. An entry point can include one or more actions that are specified by user to the automated assistant. For example, a ride sharing application can be associated with a context-specific schema file that identifies an action of ordering a ride to a destination. Therefore, an entry point to the application through the automated assistant can include the user ordering a ride to a destination using a command such as, "Assistant, please order me a ride to the airport. " In some implementations, a context specific schema file can be temporarily active depending on a context of the user, the automated assistant, and/or the application being invoked. For example, the automated assistant can access one or more context specific schema files at any given time according to a context in which the automated assistant is operating.

Each of the context-specific schema files can include one or more different entry points that are provided according to the context. For instance, if a user is using the ride sharing application to order a ride and the user has already ordered the ride, a context specific schema file can be accessible to the automated assistant during the pendency of the ride. In this case, the context specific schema file can identify entry points to the application that correspond to the ride that was ordered, as well as parameters for responding to queries from the user to the automated assistant. For example, a context-specific schema file can identify an entry point or action of determining a time of arrival of the ride, and/or parameters such as the estimated time of arrival. In this way, the user can access information from the application via the automated assistant by asking the automated assistant when the ride will arrive (e.g., "Assistant, when will my ride arrive?"), and the automated assistant can provide a response using the parameters in the context specific schema file without expressly invoking the application.

In some implementations, a context specific schema can identify actions, intents, parameters, and optionally triggers for causing an automated assistant to invoke an application. A trigger can identify one or more queries or patterns of queries that can be provided by a user in order to cause an automated assistant to invoke or otherwise access on application. For example, a trigger can correspond to subject matter with which an application is associated, such as words or phrases that can describe a name of an application, a function of an application, data that an application can process, and/or any other terms or phrases that can be associated with an application. For instance, a ride sharing application can correspond to the context specific schema file that identifies a trigger such as the term "ride," "destination," "car," and/or "arrive.

In some implementations, one or more triggers can be used to invoke an automated assistant and cause the automated assistant to access a context specific schema file that includes the one or more triggers. In this way, the automated assistant may not necessarily invoke an application in response to receiving a query that identifies a trigger, but rather can access one or more context specific schema files that are associated with an application. The automated assistant can then provide responses to the user, if such information for responding to the user is available in a context specific schema file. As an example, when a user orders a ride by invoking the automated assistant to interact with a ride sharing application, the ride sharing application can provide a context specific schema file to the automated assistant in response to the ordering of a ride. When the user is waiting for the ride to arrive, the user may provide a query such as, "Assistant, when will my ride arrive?" In response, the automated assistant can access the context specific schema file provided by the ride sharing application in order to identify the appropriate response to the user. For example, the context specific schema file can include information regarding the estimated time of arrival for the ride. Therefore, the automated assistant can provide the response to the user without necessarily communicating with the ride sharing application.

In some implementations, multiple different context specific schema files can be considered active at a given time at least according to the automated assistant. In this way, an application that provides the context specific schema files will not be limited to a single schema file from which the automated assistant can operate. Rather, the automated assistant and/or the application can activate and/or deactivate each context specific schema file of the multiple different context specific schema files based on a variety of different factors. For example, a user can invoke the automated assistant in order to initialize an application, and the automated assistant can filter the context specific schema files associated with the application based on the content of an invocation phrase provided by the user. Each context specific schema file can include trigger words or phrases that can allow the context specific schema files to be searched and/or filtered in order for the automated assistant to identify one or more context specific schema files from which to operate.

For instance, when the user invokes the automated assistant to order food from a food ordering application, the food ordering application can provide the automated assistant with context specific schema files that can include trigger words. A first schema file can include a trigger word such as "order," and a second schema file can include a trigger word such as "status. " When the automated assistant is invoked for initially providing a food order to the food ordering application, the automated assistant can identify the first schema file, at least based on the trigger word "order. " Therefore, the first schema file can be considered active by the automated assistant when the user is placing the food order with the food ordering application via the automated assistant. However, once the food order has been placed, the user may invoke the automated assistant to check a status of their food order in order to find out when the food will be delivered. For instance, the user can provide a natural language input such as, "Assistant, what is the status of the food order?" In response, the automated assistant can search the available context-specific schema files for trigger words or phrases that have some correspondence to the natural language input. When the automated assistant identifies the second schema file as having correspondence to the natural language input, the automated assistant can deactivate the first schema file and activate the second schema file. Alternatively, when the second schema file has been identified as having a correspondence to the natural language input, the application can deactivate the first schema file and activate the second schema file. By allowing certain schema files to be activated and deactivated, latency from the automated assistant can be reduced as schema files can be more readily identified or filtered before responding to a user. Furthermore, computational resources such as memory and processing and bandwidth can be preserved as a reduced number of schema files will need to be managed by the automated assistant for particular responses and/or contexts.

In some implementations, an application can refer a partially installed application that includes less functionality than a complete installation of a corresponding application. For example, a user may install a partially installed application on a computing device in order to determine whether they would like to purchase the complete installation of the corresponding application. The complete installation can be associated with multiple different context specific schema files, and the partially installed application can be associated with a smaller number of context specific schema files than the complete installation. In this way, the user will be able to interact with the partially installed application via the automated assistant without having to purchase the complete installation of the application. Furthermore, because activation of the context specific schema files is dynamic, the application, or a server associated with the application, can provide the context specific schema files to the automated assistant based on a version of the application that the user has installed or is otherwise accessing. For example, when the user accesses the partially installed application, the automated assistant can be provided with a first number of context specific schema files. Furthermore, when the user accesses the complete installation of the application, the automated assistant can be provided with a second number of context specific schema files that is greater than the first number of context specific schema files. However, in some implementations the number of context specific schema files available to the automated assistant can be based on data associated with interactions between the user and the automated assistant.

The above description is provided as an overview of some implementations of the present disclosure, however, the invention is set out in the appended set of claims.

<FIG> illustrates a system <NUM> for providing context specific schema files to an automated assistant <NUM> to allow the automated assistant <NUM> to more readily facilitate interactions between a user and an application <NUM>. The automated assistant <NUM> can operate as part of an assistant application that is provided at one or more computing devices, such as a client device <NUM> (e.g., a cellular phone or a standalone speaker device), and/or a remote computing device <NUM>, such as a server device <NUM>. A user can interact with the automated assistant <NUM> via an assistant interface, which can be a microphone, a camera, a touch screen display, a user interface, and/or any other apparatus capable of providing an interface between a user and an application. For instance, a user can initialize the automated assistant <NUM> by providing a verbal, textual, or a graphical input to the assistant interface to cause the automated assistant <NUM> to perform a function (e.g., provide data, control a peripheral device, access an agent, etc.). The client device <NUM> can include a display device, which can be a display panel that includes a touch interface for receiving touch inputs and/or gestures for allowing a user to control applications of the client device <NUM> via the touch interface.

The client device <NUM> can be in communication with the remote computing device <NUM> over one or more networks <NUM>, such as the Internet. The client device <NUM> can offload computational tasks to the remote computing device <NUM> in order to conserve computational resources at the client device <NUM>. For instance, the remote computing device <NUM> can host the automated assistant <NUM>, and the client device <NUM> can transmit inputs received at one or more assistant interfaces to the remote computing device <NUM>. However, in some implementations, the automated assistant <NUM> can be hosted at the client device <NUM>. In various implementations, all or less than all aspects of the automated assistant <NUM> can be implemented on the client device <NUM>. In some of those implementations, aspects of the automated assistant <NUM> are implemented via a local assistant application of the client device <NUM> and interface with the remote computing device <NUM> that implements other aspects of the automated assistant <NUM>. The remote computing device <NUM> can optionally serve a plurality of users and their associated assistant applications via multiple threads. In some implementations where all or less than all aspects of the automated assistant <NUM> are implemented via a local assistant application of the client device <NUM>, the local assistant application can be an application that is separate from an operating system of the client device <NUM> (e.g., installed "on top" of the operating system) - or can alternatively be implemented directly by the operating system of the first client device <NUM> (e.g., considered an application of, but integral with, the operating system).

In some implementations, the remote computing device <NUM> can include a voice to text engine <NUM> that can process audio data received at an assistant interface to identify the text embodied in the audio data. The process for converting the audio data to text can include a speech recognition algorithm, which can employ neural networks, word2vec algorithms, and/or statistical models for identifying groups of audio data corresponding to words or phrases. The text converted from the audio data can be parsed by a text parser engine <NUM> and made available to the automated assistant <NUM> as textual data that can be used to generate and/or identify command phrases from the user and/or an application <NUM> hosted at a remote computing device <NUM>.

In some implementations, the automated assistant <NUM> can mediate interactions between a user and the application <NUM> that is accessible to the client device <NUM> and the automated assistant <NUM>. Mediating interactions can include, but is not limited to, receiving commands from the user at the automated assistant <NUM> for invoking the application <NUM> to fulfill or respond to the commands. During interactions between the user and the automated assistant <NUM>, user data <NUM> and/or contextual data <NUM> can be collected at the client device <NUM>, the server device <NUM>, and/or any other device that can be associated with the user. User data <NUM> and/or contextual data <NUM> can be collected by one or more applications or devices that are integral with or accessible to the client device <NUM>. For instance, contextual data <NUM> can include data corresponding to temporal data, location data, event data, social network data, media data, and/or any other data that can be relevant to interactions between the user and the automated assistant <NUM>. Additionally, user data <NUM> can include account information, message information, calendar information, user preferences, historical interaction data between the user and the automated assistant <NUM>, application data associated with applications that are accessible to the client device <NUM>, and/or any other data that can be associated with a user.

In order for the automated assistant <NUM> to mediate interactions between the user and the application <NUM>, the user can invoke the automated assistant <NUM> by providing a natural language input to an assistant interface <NUM> at the client device <NUM>. For example, the natural language input can include, "Assistant, open my game application so that I can play. " The natural language input can refer to the application <NUM>, which can be a third party application that is provided by an entity that is different than an entity that is providing the automated assistant <NUM>. Alternatively, the application <NUM> can be provided by the same entity that is providing the automated assistant <NUM>. In order to determine or identify the application with which the user is referring, the automated assistant <NUM> can employ an application engine <NUM>. The application engine <NUM> can use or access the user data <NUM> and/or the contextual data <NUM> in order to search, filter, and/or rank applications that are accessible to the client device <NUM> and/or the automated assistant <NUM>. For example, the contextual data <NUM> can identify a specific gaming application that the user most recently accessed, therefore that specific gaming application can be identified by the automated assistant <NUM> in response to receiving the natural language input.

When the application <NUM> has been identified by the automated assistant <NUM>, the automated assistant <NUM> can determine whether there are any cached or received schema files <NUM> available to the automated assistant <NUM> and associated with the identified application <NUM>. In some implementations, the application engine <NUM> can manage a storage or table of schema files <NUM>, along with a status of one or more schema files <NUM>. For instance, a schema file <NUM> can be considered active or inactive depending on a variety of factors. Such factors can include a version of the application <NUM> that is installed on or accessible to the client device <NUM>, contextual data <NUM> and/or user data <NUM> associated with the application <NUM>, data from the remote device <NUM> that hosts the application <NUM>, contents of the schema files <NUM>, and/or any other information that can be associated with the application <NUM>. In some implementations, the automated assistant <NUM> can access schema files <NUM> at the application <NUM> or remote device <NUM>. The schema files <NUM> can be provided to the automated assistant <NUM> and/or the client device <NUM> by the application <NUM>. Schema files can be provided to the automated assistant <NUM> and/or the client <NUM> when the user downloads a version of the application <NUM> at the client device <NUM>. Alternatively, or additionally, the schema files <NUM> can be provided to the client device <NUM> and/or the automated assistant <NUM> in response to one or more requests for the schema files <NUM> from the client device <NUM> and/or the automated assistant <NUM>. In some implementations, the automated assistant <NUM> can access a context specific schema file <NUM> associated with the application <NUM> and determine a response to the input from the user using the context specific schema file <NUM>. In other implementations, the automated assistant <NUM> can access multiple contact specific schema files <NUM> associated with the application, and search through the schema files <NUM> to identify a schema file <NUM> that is most relevant to the input from the user. For example, searching through the schema files <NUM> can include comparing content of the input from the user with trigger data <NUM> provided in the schema files <NUM>. Therefore, trigger data <NUM> that is most correlated to the input from the user can be used to select a corresponding schema file <NUM>. For instance, a correlation between the input from the user and the trigger data <NUM> in the schema file <NUM> can be based on a comparison between text generated from input from the user and text provided in the trigger data <NUM>.

In some implementations, data generated from an input received at the assistant interface <NUM> can be provided to the application <NUM>, which can identify one or more schema files <NUM> provided to the automated assistant <NUM> based on the received data. In this way, the application <NUM> can search or filter the schema files <NUM> based on the receive data in order to identify a reduced number of schema files <NUM> that can be considered active by the automated assistant <NUM>. For example, the data from the automated assistant <NUM> can be used to search the schema files for trigger data <NUM> and/or intent data <NUM>. Intent data <NUM> can correspond to one or more commands or actions that can be performed by the application <NUM>. Therefore, any actions or commands specified by the user in an input received at the assistant interface <NUM> can be used to identify specific schema files <NUM>. The application <NUM> and/or the remote device <NUM> can transmit the identified schema files <NUM> over a network <NUM> to the server device <NUM> to allow the automated assistant <NUM> to access the identified schema files <NUM>. Alternatively, or additionally, the automated assistant <NUM> can request the identified schema files <NUM> from the application <NUM>.

In some implementations, when the automated assistant <NUM> has received one or more schema files <NUM> from the application <NUM>, the automated assistant <NUM> can determine a response to be provided to the user. The response can be based on trigger data <NUM>, which can correspond to data to be supplied to a function for execution by the application <NUM>. For example, when the automated assistant <NUM> receives input data corresponding to the user input, "Assistant, open my game application so that I can play," the automated assistant <NUM> can identify one or more slots that should be filled by supplemental data from the user. For example, when the trigger data <NUM> indicates that a type of game needs to be selected by the user, the automated assistant can generate a response requesting the supplemental data, such as, "Ok, what type of game would you like to play?" In response, the user can provide the supplemental data (e.g., "I would like to play chess. "), which can be converted to one or more slot values. The automated assistant <NUM> can provide the slot values back to the application <NUM> in order for application <NUM> to complete performance of the action that the user was intending to invoke.

In some implementations, trigger data <NUM> can include data to be incorporated into a response from the automated assistant <NUM>. For instance, when the automated assistant <NUM> receives a context specific schema file <NUM> from the application <NUM>, the automated assistant <NUM> can search the schema file <NUM> for trigger data <NUM> associated with the user input. In response to the user input, "Assistant, open my game application so that I can play," the automated assistant can use the received schema file <NUM> and identify types of games that can be played using the game application. The types of games can then be identified in the response that is provided back to the user at the assistant interface <NUM> (e.g., Would you like to play chess, go, or checkers?).

In some implementations, the automated assistant <NUM> can access one or more schema files <NUM>, or store one or more schema files <NUM>, based on a state or status of the automated assistant <NUM> and/or the application <NUM>. For example, the application <NUM> can store or manage one or more status identifiers <NUM>, which can correspond to a number of active context specific schema files <NUM> and/or schema files <NUM> that are accessible to the automated assistant <NUM>. In other words, depending on the interactions between the user and the automated assistant <NUM>, different context specific schema files <NUM> can be active or inactive, and therefore a status associated with the context specific schema files <NUM> can change. For instance, when a user invokes the automated assistant <NUM> to access an application <NUM>, a first status can be associated with the application <NUM>, and a first set of schema files can be active or otherwise accessible to the automated assistant <NUM>. When the application <NUM> has performed an initial operation in response to being accessed by the automated assistant <NUM>, the status associated with the application <NUM> can change to a second status. The second status can be associated with a second set of schema files, which can be active or otherwise accessible to the automated assistant <NUM> subsequent to the application <NUM> performing the initial operation. For instance, the initial operation can be processing a selection of a type of game to be played by the user through the application <NUM>. Once the type of game has been selected, the second set of schema files can include one or more schema files <NUM> that include intent data <NUM> for actions such initializing game play of the selected type of game. In some implementations, the first set of schema files can include at least one schema file that is not included in the second set of schema files. Each schema file can be a JSON file and/or any other file that can include a list or a table of data that can be used by an automated assistant <NUM> when responding to a user. Furthermore, the application <NUM> can be a game application, navigation application, ride sharing application, social networking application, media application, shopping application, search application, communication application, and/or any other application that can be accessed by an automated assistant.

<FIG> provides a diagram <NUM> that illustrates how an automated assistant <NUM> can manage multiple context specific schema files for enabling the automated assistant <NUM> to mediate interactions between a user <NUM> and an application. In some implementations, the automated assistant <NUM> can correspond to the automated assistant <NUM>, the server application <NUM> can correspond to the application <NUM>, and the local application <NUM> can correspond to an application that is associated with application <NUM> and loaded onto the client device <NUM>. The context specific schema files can originate at a local application <NUM>, and/or a remote device <NUM> such as a server device that hosts a server application <NUM> that is accessible to the local application <NUM>. In some implementations, the context specific schema files can be stored at a computing device <NUM>, which can be operated by the user <NUM>, and/or the remote device <NUM>. The local application <NUM> can correspond to an at least partial installation of an application. The partial installation of the application can require management of a first schema file <NUM>. The first schema file can include triggers, intents, slot data, fulfillment data, and/or any other data that can be used by an automated assistant to interact with an application. The partial installation of the application can allow the user <NUM> to perform a reduced number of functions compared to a full installation of the application. Furthermore, in order to allow the automated assistant <NUM> to mediate interactions between the user and the partial installation of the application, the first schema file can be made accessible to the automated assistant <NUM> at the computing device <NUM>. In this way, the automated assistant <NUM> will be able to interact with the local application <NUM> without necessarily contacting the server application <NUM>. This can reduce latency that would otherwise be exhibited at the computing device <NUM> when the automated assistant <NUM> and/or the local application <NUM> communicate with non-local devices in order to respond to a request from the user <NUM>. In some implementations, the application corresponding to <FIG> can refer to a complete or full installation of an application.

When the local application <NUM> has been installed at the computing device <NUM>, the user <NUM> can provide a first input command <NUM> to the automated assistant <NUM> via an assistant interface at the computing device <NUM>. For example, the local application <NUM> can correspond to a partial installation of a ride sharing application. The first input command <NUM> can therefore be, "Assistant, order a ride for me. " In response to receiving the first input command <NUM>, the automated assistant <NUM> can access a storage device available to the computing device <NUM> to determine whether any schema files are active at the computing device <NUM>. Because the first schema file <NUM> was provided with the local application <NUM>, the first schema file <NUM> can be identified by the automated assistant <NUM> as an active schema file. In some implementations, the computing device <NUM> can store multiple different schema files associated with multiple different applications. Therefore, in order for the automated assistant <NUM> to identify a schema file that corresponds to the first input command <NUM>, the automated assistant <NUM> can compare contents of the first input command <NUM> to contents of the available schema files. The schema files can thereafter be ranked or filtered according to how similar their contents are to the content of the first input command <NUM>. A schema file that is ranked highest, or is isolated from the filtering process, can be accessed in order to identify slot values and/or the filament data that can be used to respond to the user.

When the automated assistant <NUM> has identified an active schema file, and/or isolated a schema file from multiple schema files, the automated assistant <NUM> can generate a response to the user <NUM> using the schema file. For example, if the first schema file <NUM> is identified by the automated assistant <NUM>, contents of the first schema file <NUM> can be used to generate a first schema file response <NUM>, which can be provided back to the user <NUM>. For instance, contents of the first schema file <NUM> can include intent data that can act as entry points for invoking the local application <NUM>, as well as fulfillment data that can provide information from which the automated assistant <NUM> can respond to queries from the user <NUM>. The intent data can include, for example, a list of intent identifiers (e.g., "intents: 'register'; 'order'; 'location'; 'destination';) corresponding to actions that the local application <NUM> can perform. Furthermore, the fulfillment data can include a list of words or phrases (e.g., "fulfillment: 'register': 'provide email address'; 'order': 'select a type of ride'; 'location': 'provide your location'; 'destination': 'provide your destination';) corresponding to responsive content that can be provided back to the user via the automated assistant <NUM>. In this way, the automated assistant <NUM> can correlate the first input command <NUM> to the first schema file <NUM>, to identify an action provided in the first schema file <NUM>, and/or responsive content that can be used to generate a response for the user <NUM>.

When the automated assistant <NUM> has identified responsive content that satisfies the first input command <NUM>, the automated assistant <NUM> can provide a first schema file response <NUM> back to the user <NUM>. The first schema file response <NUM> can be a response transmitted through an automated assistant interface with content that is based on the response of content from the first schema file <NUM>. For example, in response to the user <NUM> providing a natural language input command such as, "Assistant, order a ride for me," the automated assistant <NUM> can provide a first schema file response <NUM> such as, for example, "Ok, 'select a type of ride. '" In this way, the automated assistant <NUM> is able to compile responsive content from a schema file without invoking the server application <NUM> and/or any other device that is separate from the computing device <NUM>. This can reduce latency that would otherwise be exhibited by the automated assistant <NUM> when communicating over a network with one or more other network devices, such as the remote device <NUM>.

In furtherance of causing the local application <NUM> to perform one or more actions via the automated assistant <NUM>, the user <NUM> can provide a second input command <NUM> to the automated assistant <NUM>. For example, subsequent to providing the first input command <NUM>, the user <NUM> can provide the details necessary for the local application <NUM> to perform an action identified in the first schema file <NUM>. The details can include, for instance, the location for the user <NUM> to be picked up, the destination for the user <NUM> to be dropped off, and/or any other information that the local application <NUM> can require for performing the action. Therefore, at the time the user <NUM> provides the second input command <NUM>, the local application <NUM> and/or the automated assistant <NUM> can be operating in a different context. For example, the different context can correspond to a waiting period when the ride has been ordered and is on the way to the user <NUM> to pick up the user <NUM>.

In some implementation, the automated assistant <NUM> and/or the local application <NUM> can generate an identifier for the context in which the local application <NUM> and/or the automated assistant <NUM> are operating. The identifier can be transmitted to the server application <NUM>, and/or be used by the local application <NUM>, in order to identify a schema file that corresponds to the context. Alternatively, or additionally, the automated assistant <NUM> and/or the local application <NUM> can search through one or more schema files stored at the computing device <NUM> to identify a schema file that is most relevant to the context in which the automated assistant <NUM> and/or the location application <NUM> are operating.

In some implementations, the automated assistant <NUM> can compare content of the second input command <NUM> to content of the first schema file <NUM> in order to determine whether to access a different schema file. For instance, the second input command <NUM> can include a natural language input such as, "Assistant, what is the status of my ride?" In response to receiving the second input command <NUM>, the automated assistant <NUM> and/or the local application <NUM> can access the first schema file <NUM> to determine whether the first schema file includes responsive content that will satisfy the second input command <NUM>. If the first schema file <NUM> includes responsive content that satisfies the second input command <NUM>, the automated assistant <NUM> can generate a response for the user <NUM> based on the responsive content of the first schema file <NUM>. However, if the first schema file <NUM> does not include responsive content that is capable of satisfying the second input command <NUM>, the automated assistant and/or the local application <NUM> can retrieve a different schema file for providing responsive content.

In some implementations, the automated assistant <NUM> and/or the local application <NUM> can communicate command data <NUM> to the server application <NUM>. The command data <NUM> can be associated with the second and input command <NUM>, and therefore identify one or more actions that the user <NUM> is requesting the local application <NUM> to perform. Alternatively, and/or additionally, the command data <NUM> can identify a context in which the automated assistant <NUM> and/or the local application <NUM> are operating. By providing the command data <NUM> to the server application <NUM>, the server application <NUM> can identify one or more schema files to provide to the automated assistant <NUM> and/or the local application <NUM>. Any subsequent schema file provided to the automated assistant <NUM> and/or the local application <NUM> can further the interaction between the user <NUM> and the local application <NUM>, without having to immediately communicate with a remote device. For instance, based on the command data <NUM>, the server application <NUM> can identify and/or transmit a second schema file <NUM>, which can be provided to the automated assistant <NUM> and/or the local application <NUM>. The first schema file <NUM> and/or the second schema file <NUM> can include at least some amount a different content. For example, the second schema file <NUM> can identify at least one action that the first schema file <NUM> is void of or is otherwise not identified within.

In response to the second schema file <NUM> being received at the computing device in <NUM>, the local application <NUM> can manage the second schema file <NUM>. In order to provide a response to the second input command <NUM>, the automated assistant can compare content from the second input command <NUM> to content from the second schema file <NUM>. For example, the second input command <NUM> can be, for example, "Assistant, what is the status of my ride," and the second schema file <NUM> can identify an action corresponding to a status check for an incoming ride. The second schema file <NUM> can include intent data (e.g., "intents: 'status'; 'price'; 'driver';) corresponding to actions that the local application <NUM> can perform as well as fulfillment data, which can include a list of words or phrases (e.g., "fulfillment: 'status': 'your ride will arrive at <NUM>:<NUM> PM'; 'price': 'your ride will cost $<NUM>'; 'driver': 'your driver is Alfred';) that can be used to respond the second input command <NUM>.

The automated assistant <NUM> can access the second schema file <NUM> in order to identify a correspondence between contents of the second input command <NUM> (e.g., "status") with contents of the second schema file <NUM> (e.g., "fulfillment: 'status'"). The correspondence can be used to generate responsive content that can be used to provide a response to the second input command <NUM>. For example, the fulfilment data, "your ride will arrive at <NUM>:<NUM> PM" can correspond to the intent data, "action," which can both be identified in the second schema file <NUM>. Therefore, the automated assistant <NUM> can provide a second schema file response <NUM> based on the correspondence. For instance, the second schema file response <NUM> provided by the automated assistant <NUM> can be, "Ok, your ride will arrive at <NUM>:<NUM> PM. " Should the user <NUM> decide to provide subsequent input commands that correspond to intents identified in the second schema file <NUM>, the automated assistant <NUM> can provide responses to the user <NUM> without contacting the server application <NUM>. As a result, latency exhibited by the automated assistant <NUM> and/or the local application <NUM> can be mitigated or eliminated by providing one or more schema files that can be dynamically updated based on a context of an interaction between the user <NUM> and an automated assistant <NUM>.

<FIG> illustrates a method <NUM> for providing a response to a user of an automated assistant based on a schema file that is accessible to the automated assistant. The method <NUM> can be performed by one or more applications and/or devices suitable for interacting with an automated assistant. The method <NUM> can include an operation <NUM> of receiving, at an automated assistant interface, a command. The command can correspond to a request for an automated assistant to initiate performance of an action associated with an application. The automated assistant interface can be located at a computing device and include one or more of a microphone, a touch display, a camera, a sensor, and/or any other application or device capable of receiving inputs from a user. For example, the command can correspond to a spoken natural language input that is received at a microphone of the computing device. The spoken natural language input can be, for instance, "Assistant, please order more office supplies. " The command can be intended for an application that is accessible to the automated assistant or the computing device. The application can be any application capable of being controlled by an automated assistant or other application. For example, the application can be a shopping application capable of being controlled by an automated assistant that is being operated at the direction of the user.

The application can be associated with one or more context specific schema files that can be made accessible to the automated assistant. The context specific schema files can identify trigger words or phrases that can be used by the user to invoke the automated assistant to interact with the application. The context specific schema files can also include response data corresponding to content that can be used by the automated assistant to respond to the user on behalf of the application. Optionally, the context specific schema files can also include intent data, which can identify actions that can be performed by the application upon request by the automated assistant. In order to cause the automated assistant to interact with the application, the command provided by the user can include content corresponding to a context specific schema file that is associated with the application and available to the automated assistant.

The method <NUM> can further include an operation <NUM> of determining whether there is an available schema file that includes a corresponding response for the command provided by the user. The determination made at operation <NUM> can be performed by the automated assistant or by the application. For example, the automated assistant can access a storage available to the automated assistant for determining whether there are any context specific schema files associated with the application. Alternatively, the automated assistant can access multiple different file storage locations in order to determine whether any context specific schema files are associated with the application. If there is a context specific schema file associated with the application, then the method <NUM> can proceed to operation <NUM> where a response is provided to the user based on data available in the context specific schema file. However, if there is no context specific data file that is associated with the application and currently accessible to the automated assistant, the method <NUM> can proceed to operation <NUM>.

At operation <NUM> of the method <NUM>, a request for a context specific schema file is provided to the application or an entity associated with the application. For example, in response to determining that there is no context specific schema file currently active and/or accessible to the automated assistant, the automated assistant can generate a request to receive at least one context specific schema file. The request can include data that is based on the command received from the user. For instance, the request can identify a word or phrase from the command, a subject matter of the command, an action associated with the command, and/or any other data that can be used to identify an appropriate context specific schema file. In some implementations, the request can include context data, which can be based on a context in which the user has provided the command to the automated assistant interface. The context data can include user account information, temporal data, geographic data, historical command data, device descriptive data, schedule or calendar data, search history data, and/or any other contextual data that can be used by an application. In response to receiving the context data with the request, the application can identify one or more suitable context specific schema files to provide to the automated assistant.

At operation <NUM> of the method <NUM>, the automated assistant can provide a response to the command from the user based on data that is provided in the context specific schema file from the application. In some implementations, the context specific schema file can include a list of responses that are associated with the command provided to the automated assistant and/or the context with which the user is interacting with the automated assistant. In some implementations, the response can be a natural language output such as audible words and/or phrases generated by the automated assistant, text presented at a display of the computing device, and/or any other output that can be provided by a computing device. In some implementations, the response can identify an action to be performed by the application and/or slot values to be filled by content of subsequent inputs from the user to the automated assistant and/or the application.

<FIG> illustrates a method <NUM> for providing a context specific schema file according to a context in which an application is being accessed at a client device. The method <NUM> can be performed by one or more computing devices and/or one or more applications that are suitable for communicating with an automated assistant or any other application. The method <NUM> can include an operation <NUM> of determining whether an application is accessible at a client device. The determination at operation <NUM> can include acknowledging a request to download an application, acknowledging a request to register a user to use an application, providing data for at least partially installing an application, and/or receiving or accessing any other communication from a device that is associated with or includes the client device. The application can be any program or script that can operate at a computing device and can be controlled using an automated assistant that can receive natural language input from a user at an automated assistant interface of the computing device. For example, the application can be an internet of things (IOT) application that is loaded onto an IOT device or robotic device that is located within a home of a user. In some implementations, the application can correspond to a partial installation of an application or a trial version of an application, which allows the user to control the application with at least a limited amount of functionality. However, in other implementations the application can be a complete installation of an application that allows the user to control the application with complete functionality. For example, the application can be one that is loaded onto an appliance or robotic hardware within the home of the user, and the application can allow the user to control the appliance via the application. Furthermore, the automated assistant can act as a mediator between the user and the application, thereby allowing the automated assistant to initialize actions of the application at the request of the user.

The method <NUM> can further include an operation <NUM>, which can be performed when the operation <NUM> positively determines that the application is accessible to the client device. Specifically, the operation <NUM> can include providing a schema file to the client device. In some implementations, the schema file can be provided to the client device as part of the application or separate from the application. The schema file can define entry points through which the automated assistant and/or the user can access the application. For example, the schema file can identify intents or actions that the user can request the application to perform. The automated assistant can access the schema file to determine the entry points, and therefore correlate natural language input from the user to contents of the schema file in order to determine whether the user is seeking to invoke or initialize the application. For example, when the client device is an IOT device such as an appliance in a home of the user, the user can provide a spoken natural language input such as, "Assistant, please start my appliance. " The input can cause the automated assistant to initialize performance of one or more actions by the application or IOT device.

The method <NUM> can also include an operation <NUM> of determining whether an action has been initialized via the automated assistant at the client device. The operation <NUM> can be performed at a remote device that is associated with the application at the client device. For example, the remote device can be a server that hosts data corresponding to the application and/or the user. When the user initializes performance of an action by the application via the automated assistant, the automated assistant and/or the client device can transmit data to the remote device indicating that a request has been made to initialize performance of the action by the application. Furthermore, or optionally, the automated assistant can access the schema file that is provided at the client device to determine whether the automated assistant needs any additional data that is not available at the client device. In other words, the automated assistant can compare the action requested by the user to intent data that is provided in the schema file to determine whether the application loaded at the client device can perform the action. Alternatively, or optionally, the remote device can receive the data transmitted from the client device and determine whether the client device has a schema file that provides some correspondence to the action requested by the user. For example, the data transmitted by the client device can identify the action requested by the user, and the remote device can process the data to determine whether the schema file at the client device identifies the action requested by the user.

The method <NUM> can further include an operation <NUM> for determining whether the action is identified in the provided schema file. For example, when the input from the user includes, "Assistant, please start my appliance," the content of the input can be compared to the contents of the schema file provided to the client device. The schema file can include, for instance, intent data such as "intents: 'start;' 'stop;' 'pause;'" therefore the content can include some correspondence to the intent data. For instance, the user requested to "start" the appliance, and the request therefore corresponds to the schema file identifying the action "start" in the intent data.

When the schema file includes content data corresponding to the action requested by the user, the method <NUM> can proceed to operation <NUM> to determine whether any other action has been requested by the user or the automated assistant. In other words, because the schema file identifies the action requested by the user, the client device and/or the automated assistant will have the information needed to satisfy requests associated with the action. For example, the provided schema file can include fulfillment data such as, "fulfillment: 'start:' 'please advise when to stop or pause,'" which can be used by the automated assistant as a response to the initial request to initialize the action. For instance, a response to the request can include, "Ok, 'please advise when to stop or pause,'" thereby putting the user on notice of other actions (e.g., "stop" and "pause") that can be performed within the current context of the start action being initialized.

When the client device does not include or have immediate access to a schema file that identifies the action, the method <NUM> can proceed to operation <NUM>. The operation <NUM> can include identifying a separate schema file corresponding to the action. In order to identify the separate schema file corresponding to the action, one or more different schema files can be searched using data that is based on the request from the user. For instance, the data can identify an action requested to be performed by the user and each of the different schema files can be searched to determine whether one or more of the schema files include intent data that identifies the action. In some implementations, the client device and/or the automated assistant can provide contextual data that identifies a context in which the user provided the request to the automated assistant. The contextual data can be used to rank and/or filter the different schema files in order to identify one or more schema files to provide to the client device. The contextual data can be compared to the intent data, the fulfillment data, slot data, and/or any other data that can be incorporated into the schema file. For example, if the request includes content that identifies a term that is similar to or synonymous with a term that is provided in the fulfillment data of a schema file, the schema file can be ranked higher than another schema file that does not include a similar or synonymous term.

When a separate schema file has been selected, the method <NUM> can proceed to operation <NUM> where the separate schema file is provided to the client device. In some implementations, the separate schema file can be transmitted over a network from the remote device to the client device. Alternatively, or optionally, the automated assistant can request the separate schema file from the remote device, or copy the separate schema file from the remote device. The separate schema file can include intent data and/or fulfillment data that the previously provided schema file is void of. Furthermore, the separate schema file can identify multiple different actions that can be performed by the application and various responsive content that can be adopted by the automated assistant for responding to the user. In this way, the automated assistant does not necessarily need to communicate with the remote device in order to provide responses to the user when the application is performing the action or any other function associated with the schema file(s) provided to the client device. As a result, latency, computational resources, and/or power can be preserved at the computing device, as well as any other device that may be tasked with assisting in performance of the action requested by the user.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of method <NUM>, method <NUM>, and/or to implement one or more of client device <NUM>, server device <NUM>, remote computing device <NUM>, remote device <NUM>, schema file <NUM>, application <NUM>, computing device <NUM>, remote device <NUM>, automated assistant <NUM>, automated assistant <NUM>, local application <NUM>, and/or server application <NUM>.

Claim 1:
A method (<NUM>) implemented by one or more processors, the method comprising:
determining (<NUM>), by a remote computing device (<NUM>, <NUM>), that an application (<NUM>) is available at a computing device (<NUM>, <NUM>), wherein the computing device (<NUM>, <NUM>) includes an automated assistant (<NUM>) that is configured to mediate performance of an action by the application;
providing (<NUM>), by the remote computing device (<NUM>, <NUM>), in response to determining that the application (<NUM>) is available at the computing device (<NUM>, <NUM>), a first context specific schema file (<NUM>, <NUM>) to the computing device (<NUM>, <NUM>), wherein the first context specific schema file (<NUM>, <NUM>) includes responsive content provided in anticipation of the automated assistant (<NUM>) receiving a request (<NUM>) for the application (<NUM>) to perform the action;
determining (<NUM>), by the remote computing device (<NUM>, <NUM>), that the automated assistant (<NUM>) has received (<NUM>) the request (<NUM>) for the application (<NUM>) to perform the action;
receiving, by the remote computing device (<NUM>, <NUM>), command data (<NUM>) identifying a context in which the automated assistant (<NUM>) and application (<NUM>) are operating within; and
providing (<NUM>), by the remote computing device (<NUM>, <NUM>), in response to determining (<NUM>) that the automated assistant (<NUM>) has received (<NUM>) the request (<NUM>) for the application (<NUM>) to perform the action, a second context specific schema file (<NUM>) associated with the identified context to the computing device, wherein the second context specific schema file (<NUM>) includes other responsive content provided in anticipation of the automated assistant (<NUM>) receiving a separate request (<NUM>) associated with the action being performed by the application (<NUM>).