Patent Description:
Automated assistants may interact with third party software agents, or "third party agents," in order to respond to some types of user intents to which the automated assistant itself is unable to respond fully. For example, vendors such as restaurants, ride share services, and so forth, may design their own third party agents that are configured to interact with an application programming interface ("API") of an automated assistant, e.g., so that the automated assistant can act as a liaison between the user and the third party agent. However, this development paradigm can be slow to scale. Each vendor must first create a third party agent that is compatible with the automated assistant's API. This may require more resources and/or technical sophistication than many vendors may have at their disposal. Moreover, many vendors already have built interactive webpages and/or mobile applications ("apps") that users can interact with to procure various items and/or services. Requiring these vendors to design yet another interface that is compatible with automated assistants may result in what appears, at least from the vendor's perspective, to be duplication of efforts.

<CIT> discloses methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for natural language control of web browsers. The methods, systems, and apparatus include actions of obtaining a query from a user, determining a likely intent and an entity from the query, determining an action and script metadata based on the likely intent and the entity, compiling scripts based on the action and the script metadata, and providing the scripts to a web browser for execution.

<CIT> discloses an application program testing method and device. Firstly, an interface screenshot of an application program to be tested is obtained, and the interface screenshot is input into an object recognition model generated by pre-training to obtain a recognition result of the interface screenshot. The recognition result may include the number of operable objects in the interface screenshots and the location information of each operable object, and the operable objects included in the interface screenshots are sequentially tested according to the location information of each operable object.

<CIT> discloses a method, system and product for automatic performance of user interaction operations on a computing device. A method comprises: identifying elements in a layout of a GUI, displaying in visible proximity to each of the elements an assigned unique label; recognizing speech by a user vocally indicating a selected element by referring to the assigned label; and, automatically performing a user interaction operation on the selected element.

The matter for protection is set out in the appended claims.

Techniques are described herein for analyzing existing interactive graphical user interfaces ("GUI") to facilitate automatic engagement with those GUIs, e.g., by automated assistants, with little or no effort from the creators and/or hosts of those GUIs (e.g., vendors). For example, in various implementations, techniques described herein may be used to abstract, validate, maintain, generalize, extend and/or distribute individual actions and "scripts" of actions (i.e. sequences of actions) that are (re)useable to navigate through portions of various interactive GUIs. For example, in some implementations, techniques described herein may be employed to enable users to engage with an automated assistant (via a spoken or typed dialog session) to interact with the third party website while a webpage of the third party web site is open.

Suppose a user navigates a web browser to a particular retailer's website, and once the website is open on the user's device, the user could submits a spoken request, e.g., to an automated assistant, "Search for energy-efficient dishwashers. " In various implementations, a previously-generated script may be associated with that website/webpage and/or with the user's intent to search that website/webpage, e.g., in a database available to the automated assistant. The script may be executed with the parameters provided in the user's free-form natural language input (intent=search <website>, parameter="energy-efficient dishwashers") to cause a search field on the webpage to be automatically populated with the parameters. To maintain an intuitive and good user experience, the user may even see the search field opened and/or populated with automatically "typed" text. The search may be submitted, just as though the user had manually typed the search parameters into the search field and pressed submit.

The scripts mentioned above may be generated automatically, e.g., with little or no human intervention, which provides a practical way to scale an automated assistant's ability to automatically navigate across myriad GUIs. These scripts may be generated "offline" or in "real time," e.g., as a user uses an automated assistant to attempt to interact with a heretofore unknown webpage. In some implementations, the scripts may be generated by attempting to resolve a user's intent vis-à-vis a GUI that is active while the user submits a free-form natural language input, e.g., spoken or typed. Additionally or alternatively, in some implementations, the scripts may be generated in batches, e.g., with lists of GUIs (e.g., lists of uniform resource locators, or "URLs," associated with webpages) and free-form natural language inputs for which resolution should be attempted on those GUIs.

In some implementations, a free-form natural language input may be analyzed to determine a user intent to interact with a particular GUI, such as a webpage. In some implementations, this may occur while a particular GUI is rendered on a computing device, e.g., automatically or in response to a user request. In some such implementations, the active GUI may be used in conjunction with the free-form natural language input to determine the user intent. For example, the free-form natural language input may simply be "search for energy-efficient dishwashers," which by itself may either be difficult for an automated assistant to resolve without further information, or may simply be resolved by performing a default/general purpose search engine search. The fact that the particular GUI is active provides additional context, however. This additional context may be used to resolve the user intent to submit the search using a search interface provided by the active GUI.

Based on the user intent (e.g., "search"), one or more target visual cues may be identified, e.g., from a lookup table or other database, that are known to be associated with the user intent. Target visual cues may take various forms, such as common graphical icons, common arrangement of graphical elements relative to other graphical elements (which may be the case for date input fields or movie theater seat selection interfaces), For example, many webpages have a search interface that is often denoted visually using a magnifying glass symbol. If a user requests a search, then the magnifying glass may be the target visual cue. Many other visual cues are contemplated, such as calendar icons for date-entry fields, humanoid icons for login and/or person-count fields, etc. Whichever the case, in some implementations, object recognition processing may be performed on a screenshot (e.g., bitmap) of the GUI to determine a location of a detected instance of the target visual cue. In other implementations, visual cues may be searched for by looking at, for instance, source code, object code, and/or markup language underlying the GUI.

Once the location of the detected instance of the targeted visual cue is determined, it may be used to identify an interactive element of the GUI that should be auto-populated with data that is determined from the user intent. For example, in some implementations, a document object model ("DOM") underlying a webpage may be analyzed to determine, spatially, where various interactive elements are rendered on a display. Various heuristics may be employed to match a particular interactive element with the determined location of the detected instance of the targeted visual cue. Once this match is made, the interactive element may be automatically-populated with data determined from the user intent, and a submission mechanism (e.g., a "submit" button) may be triggered to transition to the next state of the GUI.

In some implementations, the outcome of this GUI transition may be analyzed to validate the success or failure of techniques described herein. For example, with many interactive webpages, data input into interactive elements is submitted as HTTP data and is ultimately presented as part of the URL of the next webpage. More generally, may webpages and/or other GUIs may include visual output that tends to validate the success or failure of the auto-population of the interactive elements, e.g., as a string above search results indicating what the search was. Whichever the case, these outcomes can serve as strong signals for validating the success of techniques described herein. In some implementations, therefore, the outcomes or data indicative thereof may be used to train, for instance, a machine learning model used to locate a target visual cue in a screenshot, or to update heuristics (or another machine learning model) used to match the target visual cue's location to an interactive element. Or, in some cases, a script that is usable to navigate through the same GUI may be generated only upon verification of a successful outcome.

Techniques described herein give rise to various technical advantages. Causing text or other data to be input into an already-rendered graphical element automatically, so that a user can see it being input, provides an intuitive and user-friendly way to help users understand how they are interacting with their computing device. Also, the GUI need not be reloaded while the data is input automatically, which can be especially beneficial when the GUI is a webpage (e.g., because there will be no blinking associated with reloading the webpage). Avoidance of GUI reload also speeds up user interaction with the GUI, especially when the GUI is a webpage.

Another advantage of techniques described herein is that a state of the GUI is preserved while the data is being input automatically. Users often perform searches in stages, with each stage incrementally narrowing/filtering the search, until a manageable number of search results are presented. With techniques described herein, the existing GUI and its state-including any previously-applied filters-may be preserved while data is automatically input and submitted. By contrast, the GUI's state might be lost if, for instance, the GUI were a webpage and the web browser used to render the GUI were redirected to a new URL with the search query incorporated, rather than inputting the search query into a search field using techniques described herein. In the former case, there is no guarantee that previously-applied filters would remain in place (unless, for instance, a website is configured to preserve previously-applied filters using some other mechanism).

Also, automatic scripts generated using techniques described herein can be periodically re-validated and updated if they no longer work. Consequently, so as GUIs evolve-especially frequent in the webpage domain-so too do scripts for automatically interacting with those webpages.

In some implementations according to the invention, a method performed by one or more processors is provided in accordance with appended claim <NUM>.

In various implementations, the GUI may take the form of an interactive webpage or a GUI of an application written in any number of programming languages, such as C, C++, Python, Java, C#, etc. In various implementations in which the GUI is an interactive webpage, the interactive element of the GUI may be identified by comparing a document object model ("DOM") of the interactive webpage with the location of the detected instance of the target visual cue.

In various implementations, the method may further include automatically submitting the data determined from the user intent; and receiving a subsequent webpage that is generated at least in part on the data determined from the user intent. In various implementations, the method may further include searching a uniform resource locator ("URL") or content of the subsequent webpage to determine an outcome of the automatic submitting. In various implementations, the object recognition may be performed using a machine learning model, and the method further includes training the machine learning model based on the outcome of the automatic submitting.

In various implementations, the free-form natural language input takes the form of a speech input captured at a microphone, and the method further includes performing speech recognition processing on the speech input to generate textual output. In various implementations, the object recognition is performed using a convolutional neural network. In various implementations, the user intent comprises submission of a search query using the GUI, and the target visual cue comprises a magnifying glass.

In various implementations, the method further includes generating, based on the identified interactive element, a script that is subsequently executable in association with the GUI and a subsequent free-form natural language input to trigger automatic population of the identified interactive element with data determined from a subsequent user intent determined from the subsequent free-form natural language input and submission of the data determined from the user intent via the GUI. In some such implementations, the subsequent automatic population and submission is performed without one or more of identifying the target visual cue, performing the object recognition, or identifying the interactive element of the GUI. In various implementations, the method further includes validating that submission of the data determined from the user intent resulted in a desired outcome, wherein the script is generated based on the validating.

In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more transitory and/or non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.

Now turning to <FIG>, an example environment in which techniques disclosed herein may be implemented is illustrated. The example environment includes one or more client computing devices <NUM>. Each client device <NUM> may execute a respective instance of an automated assistant client <NUM>, which may also be referred to herein as a "client portion" of an automated assistant. One or more cloud-based automated assistant components <NUM>, which may also be referred to herein collectively as a "server portion" of an automated assistant, may be implemented on one or more computing systems (collectively referred to as a "cloud" computing system) that are communicatively coupled to client devices <NUM> via one or more local and/or wide area networks (e.g., the Internet) indicated generally at <NUM>.

In various implementations, an instance of an automated assistant client <NUM>, by way of its interactions with one or more cloud-based automated assistant components <NUM>, may form what appears to be, from the user's perspective, a logical instance of an automated assistant <NUM> with which the user may engage in a human-to-computer dialog. One instance of such an automated assistant <NUM> is depicted in <FIG> in dashed line. It thus should be understood that each user that engages with an automated assistant client <NUM> executing on a client device <NUM> may, in effect, engage with his or her own logical instance of an automated assistant <NUM>. For the sakes of brevity and simplicity, the term "automated assistant" as used herein as "serving" a particular user will refer to the combination of an automated assistant client <NUM> executing on a client device <NUM> operated by the user and one or more cloud-based automated assistant components <NUM> (which may be shared amongst multiple automated assistant clients <NUM>). It should also be understood that in some implementations, automated assistant <NUM> may respond to a request from any user regardless of whether the user is actually "served" by that particular instance of automated assistant <NUM>.

The one or more client devices <NUM> may include, for example, one or more of: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (which in some cases may include a vision sensor), a smart appliance such as a smart television (or a standard television equipped with a networked dongle with automated assistant capabilities), and/or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided. Some client devices <NUM>, such as standalone interactive speakers (or "smart speakers"), may take the form of assistant devices that are primarily designed to facilitate dialog between users and automated assistant <NUM>. Some such assistant devices may take the form of a standalone interactive speaker with an attached display, which may or may not be a touchscreen display.

In some implementations, client device <NUM> may be equipped with one or more vision sensors <NUM> having one or more fields of view, although this is not required. Vision sensor(s) <NUM> may take various forms, such as digital cameras, passive infrared ("PIR") sensors, stereoscopic cameras, RGBd cameras, etc. The one or more vision sensors <NUM> may be used to capture image frames (still images or video) of an environment in which client device <NUM> is deployed. These image frames may then be analyzed, e.g., by a visual cue module <NUM><NUM>, to detect user-provided visual cues contained in the image frames. These visual cues may include but are not limited to hand gestures, gazes towards particular reference points, facial expressions, predefined movements by users, etc. These detected visual cues may be used for various purposes, such as invoking automated assistant <NUM> and/or causing automated assistant <NUM> to perform various actions.

As described in more detail herein, automated assistant <NUM> engages in human-to-computer dialog sessions with one or more users via user interface input and output devices of one or more client devices <NUM>. In some implementations, automated assistant <NUM> may engage in a human-to-computer dialog session with a user in response to user interface input provided by the user via one or more user interface input devices of one of the client devices <NUM>. In some of those implementations, the user interface input is explicitly directed to automated assistant <NUM>. For example, a user may verbally provide (e.g., type, speak) a predetermined invocation phrase, such as "OK, Assistant," or "Hey, Assistant. " When spoken, this verbal input may be captured by a microphone <NUM> and may cause automated assistant <NUM> to begin actively listening or monitoring typed text. Additionally or alternatively, in some implementations, automated assistant <NUM> may be invoked based on one or more detected visual cues, alone or in combination with oral invocation phrases.

In some implementations, automated assistant <NUM> may engage in a human-to-computer dialog session in response to user interface input, even when that user interface input is not explicitly directed to automated assistant <NUM>. For example, automated assistant <NUM> may examine the contents of user interface input and engage in a dialog session in response to certain terms being present in the user interface input and/or based on other cues. In many implementations, automated assistant <NUM> may utilize speech recognition to convert utterances from users into text, and respond to the text accordingly, e.g., by providing search results, general information, and/or taking one or more responsive actions (e.g., playing media, launching a game, ordering food, etc.). In some implementations, the automated assistant <NUM> can additionally or alternatively respond to utterances without converting the utterances into text. For example, the automated assistant <NUM> can convert voice input into an embedding, into entity representation(s) (that indicate entity/entities present in the voice input), and/or other "non-textual" representation and operate on such non-textual representation. Accordingly, implementations described herein as operating based on text converted from voice input may additionally and/or alternatively operate on the voice input directly and/or other non-textual representations of the voice input.

Each of client computing device <NUM> and computing device(s) operating cloud-based automated assistant components <NUM> may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by client computing device <NUM> and/or by automated assistant <NUM> may be distributed across multiple computer systems. Automated assistant <NUM> may be implemented as, for example, computer programs running on one or more computers in one or more locations that are coupled to each other through a network.

As noted above, in various implementations, client computing device <NUM> may operate an automated assistant client <NUM>, or "client portion" of automated assistant <NUM>. In various implementations, automated assistant client <NUM> may include a speech capture module <NUM> and/or a visual cue module <NUM><NUM>. In other implementations, one or more aspects of speech capture module <NUM> and/or visual cue module <NUM> may be implemented separately from automated assistant client <NUM>, e.g., by one or more cloud-based automated assistant components <NUM>. For example, in <FIG>, there is also a cloud-based visual cue module <NUM><NUM> that may detect visual cues in image data.

In various implementations, speech capture module <NUM>, which may be implemented using any combination of hardware and software, may interface with hardware such as microphone <NUM> or other pressure sensor to capture an audio recording of a user's utterance(s). Various types of processing may be performed on this audio recording for various purposes.

In various implementations, visual cue module <NUM><NUM> (and/or cloud-based visual cue module <NUM><NUM>) may be implemented using any combination of hardware or software, and may be configured to analyze one or more image frames provided by vision sensor(s) <NUM> to detect one or more visual cues captured in and/or across the one or more image frames. Visual cue module <NUM><NUM> may employ a variety of techniques to detect visual cues. For example, visual cue module <NUM><NUM> (or <NUM><NUM>)may use one or more artificial intelligence (or machine learning) models that are trained to generate output indicative of detected user-provided visual cues in image frames.

Client device <NUM> may have other applications installed as well, such as a web browser <NUM> and/or other miscellaneous application(s) <NUM>. In various implementations, other miscellaneous application(s) <NUM> may provide an interface for a user to engage in typed or spoken human-to-computer dialog with automated assistant <NUM>, as a one-on-one dialog or as a multi-participant dialog in which automated assistant <NUM> may "participate. " In some implementations, web browser <NUM> and/or other miscellaneous application(s) <NUM> may be specially designed, e.g., with a microphone button or other user interface element that is operable to invoke automated assistant <NUM>, so that the user is able to issue voice commands to aid in operation of those applications.

Speech capture module <NUM> may be configured to capture a user's speech, e.g., via a microphone <NUM>, as mentioned previously. Additionally or alternatively, in some implementations, speech capture module <NUM> may be further configured to convert that captured audio to text and/or to other representations or embeddings, e.g., using speech-to-text ("STT") processing techniques (also referred to as "automatic speech recognition," or "ASR"). Additionally or alternatively, in some implementations, speech capture module <NUM> may be configured to convert text to computer-synthesized speech, e.g., using one or more voice synthesizers. However, in some cases, because client device <NUM> may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), speech capture module <NUM> local to client device <NUM> may be configured to convert a finite number of different spoken phrases-particularly phrases that invoke automated assistant <NUM>-to text (or to other forms, such as lower dimensionality embeddings). Other speech input may be sent to cloud-based automated assistant components <NUM>, which may include a cloud-based text-to-speech ("TTS") module <NUM> and/or a cloud-based STT module <NUM>.

Cloud-based TTS module <NUM> may be configured to leverage the virtually limitless resources of the cloud to convert textual data (e.g., natural language responses formulated by automated assistant <NUM>) into computer-generated speech output. In some implementations, TTS module <NUM> may provide the computer-generated speech output to client device <NUM> to be output directly, e.g., using one or more speakers. In other implementations, textual data (e.g., natural language responses) generated by automated assistant <NUM> may be provided to speech capture module <NUM>, which may then convert the textual data into computer-generated speech that is output locally.

Cloud-based STT module <NUM> may be configured to leverage the virtually limitless resources of the cloud to convert audio data captured by speech capture module <NUM> into text, which may then be provided to intent matcher <NUM>. In some implementations, cloud-based STT module <NUM> may convert an audio recording of speech to one or more phonemes, and then convert the one or more phonemes to text. Additionally or alternatively, in some implementations, STT module <NUM> may employ a state decoding graph. In some implementations, STT module <NUM> may generate a plurality of candidate textual interpretations of the user's utterance. In some implementations, STT module <NUM> may weight or bias particular candidate textual interpretations higher than others depending on whether there are contemporaneously detected visual cues.

Automated assistant <NUM> (and in particular, cloud-based automated assistant components <NUM>) may include intent matcher <NUM>, the aforementioned TTS module <NUM>, the aforementioned STT module <NUM>, and other components that are described in more detail below. In some implementations, one or more of the modules and/or modules of automated assistant <NUM> may be omitted, combined, and/or implemented in a component that is separate from automated assistant <NUM>. In some implementations, to protect privacy, one or more of the components of automated assistant <NUM>, such as natural language processor <NUM>, TTS module <NUM>, STT module <NUM>, etc., may be implemented wholly or at least on part on client devices <NUM> (e.g., to the exclusion of the cloud).

In some implementations, automated assistant <NUM> generates responsive content in response to various inputs generated by a user of one of the client devices <NUM> during a human-to-computer dialog session with automated assistant <NUM>. Automated assistant <NUM> may provide the responsive content (e.g., over one or more networks when separate from a client device of a user) for presentation to the user as part of the dialog session. For example, automated assistant <NUM> may generate responsive content in response to free-form natural language input provided via client device <NUM>. As used herein, free-form natural language input is input that is formulated/composed by a user and that is not constrained to a group of options presented for selection by the user. Free form natural language input may be spoken (and captured by microphone <NUM>) and/or typed.

As used herein, a "dialog session" may include a logically-self-contained exchange of one or more messages between a user and automated assistant <NUM> (and in some cases, other human participants). Automated assistant <NUM> may differentiate between multiple dialog sessions with a user based on various signals, such as passage of time between sessions, change of user context (e.g., location, before/during/after a scheduled meeting, etc.) between sessions, detection of one or more intervening interactions between the user and a client device other than dialog between the user and the automated assistant (e.g., the user switches applications for a while, the user walks away from then later returns to a standalone voice-activated product), locking/sleeping of the client device between sessions, change of client devices used to interface with one or more instances of automated assistant <NUM>, and so forth.

An intent matcher <NUM> may be configured to determine a user's intent based on free-form natural language input(s) (e.g., vocal utterances, visual cues, etc.) provided by a user (or from automatically-generated or human-curated batches of free-form natural language inputs) and/or based on other signals, such as sensor signals, online signals (e.g., data obtained from web services), and so forth. In some implementations, intent matcher <NUM> may include a natural language processor <NUM> and the aforementioned cloud-based visual cue module <NUM><NUM>. In various implementations, cloud-based visual cue module <NUM><NUM> may operate similarly to visual cue module <NUM><NUM> except that cloud-based visual cue module <NUM><NUM> may have more resources at its disposal. In particular, cloud-based visual cue module <NUM><NUM> may detect visual cues that may be used by intent matcher <NUM>, alone or in combination with other signals, to determine a user's intent.

Natural language processor <NUM> may be configured to process free-form natural language input generated by user(s) via client device <NUM> and may generate annotated output (e.g., in textual form) for use by one or more other components of automated assistant <NUM>. For example, the natural language processor <NUM> may process natural language free-form input that is generated by a user via one or more user interface input devices of client device <NUM>. The generated annotated output includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the free-form natural language input.

In some implementations, the natural language processor <NUM> is configured to identify and annotate various types of grammatical information in free-form natural language input. For example, the natural language processor <NUM> may include a morphological module that may separate individual words into morphemes and/or annotate the morphemes, e.g., with their classes. Natural language processor <NUM> may also include a part of speech tagger configured to annotate terms with their grammatical roles. For example, the part of speech tagger may tag each term with its part of speech such as "noun," "verb," "adjective," "pronoun," etc. Also, for example, in some implementations the natural language processor <NUM> may additionally and/or alternatively include a dependency parser (not depicted) configured to determine syntactic relationships between terms in natural language input. For example, the dependency parser may determine which terms modify other terms, subjects and verbs of sentences, and so forth (e.g., a parse tree) - and may make annotations of such dependencies.

In some implementations, the natural language processor <NUM> may additionally and/or alternatively include an entity tagger (not depicted) configured to annotate entity references in one or more segments such as references to people (including, for instance, literary characters, celebrities, public figures, etc.), organizations, locations (real and imaginary), and so forth. In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some implementations, the knowledge graph may include nodes that represent known entities (and in some cases, entity attributes), as well as edges that connect the nodes and represent relationships between the entities. For example, a "banana" node may be connected (e.g., as a child) to a "fruit" node," which in turn may be connected (e.g., as a child) to "produce" and/or "food" nodes. As another example, a restaurant called "Hypothetical Café" may be represented by a node that also includes attributes such as its address, type of food served, hours, contact information, etc. The "Hypothetical Café" node may in some implementations be connected by an edge (e.g., representing a child-to-parent relationship) to one or more other nodes, such as a "restaurant" node, a "business" node, a node representing a city and/or state in which the restaurant is located, and so forth.

The entity tagger of the natural language processor <NUM> may annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and/or a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on content of the natural language input to resolve a particular entity and/or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity.

In some implementations, the natural language processor <NUM> may additionally and/or alternatively include a coreference resolver (not depicted) configured to group, or "cluster," references to the same entity based on one or more contextual cues. For example, the coreference resolver may be utilized to resolve the term "there" to "Hypothetical Café" in the natural language input "I liked Hypothetical Café last time we ate there.

In some implementations, one or more components of the natural language processor <NUM> may rely on annotations from one or more other components of the natural language processor <NUM>. For example, in some implementations the named entity tagger may rely on annotations from the coreference resolver and/or dependency parser in annotating all mentions to a particular entity. Also, for example, in some implementations the coreference resolver may rely on annotations from the dependency parser in clustering references to the same entity. In some implementations, in processing a particular natural language input, one or more components of the natural language processor <NUM> may use related prior input and/or other related data outside of the particular natural language input to determine one or more annotations.

Intent matcher <NUM> may use various techniques to determine an intent of the user, e.g., based on output from natural language processor <NUM> (which may include annotations and terms of the natural language input) and/or based on output from visual cue module (e.g., <NUM><NUM> and/or <NUM><NUM>). In some implementations, intent matcher <NUM> may have access to one or more databases (not depicted) that include, for instance, a plurality of mappings between grammars, visual cues, and responsive actions (or more generally, intents). In many cases, these grammars may be selected and/or learned over time, and may represent the most common intents of users. For example, one grammar, "play <artist>", may be mapped to an intent that invokes a responsive action that causes music by the <artist> to be played on the client device <NUM> operated by the user. Another grammar, "[weather | forecast] today," may be match-able to user queries such as "what's the weather today" and "what's the forecast for today?".

In addition to or instead of grammars, in some implementations, intent matcher <NUM> may employ one or more trained machine learning models, alone or in combination with one or more grammars and/or visual cues. These trained machine learning models may also be stored in one or more databases and may be trained to identify intents, e.g., by embedding data indicative of a user's utterance and/or any detected user-provided visual cues into a latent space, and then determining which other embeddings (and therefore, intents) are most proximate, e.g., using techniques such as Euclidean distance, cosine similarity, etc..

As seen in the "play <artist>" example grammar, some grammars have slots (e.g., <artist>) that can be filled with slot values (or "parameters"). Slot values may be determined in various ways. Often users will provide the slot values proactively. For example, for a grammar "Order me a <topping> pizza," a user may likely speak the phrase "order me a sausage pizza," in which case the slot <topping> is filled automatically. Additionally or alternatively, if a user invokes a grammar that includes slots to be filled with slot values, without the user proactively providing the slot values, automated assistant <NUM> may solicit those slot values from the user (e.g., "what type of crust do you want on your pizza?"). In some implementations, slots may be filled with slot values based on visual cues detected by visual cue modules <NUM><NUM>-<NUM>. For example, a user could utter something like "Order me this many cat bowls" while holding up three fingers to visual sensor <NUM> of client device <NUM>. Or, a user could utter something like "Find me more movies like this" while holding of a DVD case for a particular movie.

Fulfillment module <NUM> may be configured to receive the predicted/estimated intent that is output by intent matcher <NUM>, as well as an associated slot values (whether provided by the user proactively or solicited from the user) and carry out (or "resolve," or "fulfill") the intent. In various implementations, fulfillment (or "resolution" or "carrying out") of the user's intent may cause various fulfillment information (also referred to as "responsive" information or "resolution information") to be generated/obtained, e.g., by fulfillment module <NUM>. As will be described below, the fulfillment information may in some implementations be provided to a natural language generator ("NLG" in some Figs. ) <NUM>, which may generate natural language output based on the fulfillment information. Also, in some implementations, fulfillment module <NUM> may be configured to execute scripts for automatically interacting with GUIs, which may be generated using techniques described herein.

Fulfillment (or "resolution") information may take various forms because an intent can be fulfilled (or "resolved") in a variety of ways. Suppose a user requests pure information, such as "Where were the outdoor shots of 'The Shining' filmed?" The intent of the user may be determined, e.g., by intent matcher <NUM>, as being a search query. The intent and content of the search query may be provided to fulfillment module <NUM>, which as depicted in <FIG> may be in communication with one or more search modules <NUM> configured to search corpuses of documents and/or other data sources (e.g., knowledge graphs, etc.) for responsive information. Fulfillment module <NUM> may provide data indicative of the search query (e.g., the text of the query, a reduced dimensionality embedding, etc.) to search module <NUM>. Search module <NUM> may provide responsive information, such as GPS coordinates, or other more explicit information, such as "Timberline Lodge, Mt. Hood, Oregon. " This responsive information may form part of the fulfillment information generated by fulfillment module <NUM>.

Additionally or alternatively, fulfillment module <NUM> may be configured to receive, e.g., from intent matcher <NUM>, a user's intent and any slot values provided by the user or determined using other means (e.g., GPS coordinates of the user, user preferences, etc.) and trigger a responsive action. Responsive actions may include, for instance, ordering a good/service, starting a timer, setting a reminder, initiating a phone call, playing media, sending a message, etc. In some such implementations, fulfillment information may include slot values associated with the fulfillment, confirmation responses (which may be selected from predetermined responses in some cases), etc..

Natural language generator <NUM> may be configured to generate and/or select natural language output (e.g., words/phrases that are designed to mimic human speech) based on data obtained from various sources. In some implementations, natural language generator <NUM> may be configured to receive, as input, fulfillment information associated with fulfillment of an intent, and to generate natural language output based on the fulfillment information. Additionally or alternatively, natural language generator <NUM> may receive information from other sources, such as third party applications (e.g., required slots), which it may use to compose natural language output for the user.

Various aspects of the present disclosure may be implemented in whole or in part by GUI navigation engine <NUM>. In general, GUI navigation engine <NUM> may be configured to analyze GUIs in order to generate (and apply in some cases) scripts to automatically navigate through all or portions of these GUIs. These GUIs may take various forms. In some implementations, GUIs may be native and/or proprietary to miscellaneous application(s) <NUM>. These types of application GUIs can take numerous forms and be written in numerous different programming languages.

Additionally or alternatively, in some implementations, GUIs may take the form of webpages that are written in various markup languages, such as the hypertext markup language ("HTML"), extensible HTML ("XHTML"), the extensible markup language ("XML"), and so forth. In some cases, webpages are associated with other types of documents that, for instance, impose styles on and/or add functionality to webpages. For example, cascading style sheets ("CSS") allow for description of a visual presentation of a markup language document, such as an HTML, XHTML, and/or XML document. In addition, many webpages may include and/or are linked to client-side code such as JavaScript that is executable at web browser <NUM> of client device <NUM> to facilitate local interactivity.

GUIs may include various types of interactive elements. Interactive elements may include but are not limited to radio buttons, toggle switches, check boxes, text input fields, drop down menus, sliders, and any other user interface element that is available, for instance, using technologies such as HTML, XML, CSS, JavaScript, etc., as well as GUI libraries and/or application programming interfaces ("APIs") available in various programming languages, such as C, C++, Java, Python, C#, etc..

GUI navigation engine <NUM> may have access to multiple databases and/or indexes, such as a webpage database <NUM>. Webpage database <NUM> may include one or more databases of HTML, XHMTL, CSS, XML, PDF, or other documents that are generally available on the Internet using web browser <NUM> or other applications and are searchable, e.g., by search module <NUM>, using various search engines (not depicted).

GUI navigation engine <NUM> also may have access to a database of scripts <NUM> that are generated using techniques described herein. Each script may be associated with a GUI, and may include instructions that are performable to automatically navigate through and/or interact with at least a portion of the GUI. As an example, a script may be generated in association with a webpage. When that webpage is rendered by web browser <NUM> of an automated assistant-equipped client device <NUM>, automated assistant <NUM> may retrieve the script, e.g., in response to a user's free-form natural language input requesting interaction with the webpage. Automated assistant <NUM> and/or another component may execute the script to trigger automatic interaction with one or more interactive elements of the webpage.

As will be described in more detail shortly, in some implementations, scripts generated using techniques described herein may be "validated," e.g., by a human or automatically by GUI navigation engine <NUM> using techniques described herein. To "validate" a script means to determine that the script is capable of carrying out a particular intent or purpose of the interactive GUI corresponding to the script. The process of validating a script may occur periodically (e.g., every N seconds, minutes, hours, days, weeks, etc., wherein N is a positive integer), on demand, etc..

<FIG> depict an example of how techniques described herein may be used to automatically interact with a GUI in the form of a webpage. These figures and other figures herein demonstrate both a "learning" phase-i.e., generation of a script that is usable subsequently to interact with the GUI-and an "application" phase-i.e. subsequent execution of such a script to trigger automatic interaction with the GUI. In <FIG> and subsequent figures, disclosed techniques will be demonstrated using a client device <NUM> configured with selected aspects of the present disclosure, as it may be operated to interact with a hypothetical website, "feuxfilm. " However, this is not meant to be limiting.

In <FIG>, client device <NUM> has been navigated to the website associated with the URL, feuxfilm. This may occur by a user typing that URL into a URL field, or by the user invoking automated assistant <NUM> (e.g., by pressing a microphone element <NUM>) and requesting navigation to the website ("open feuxfilm. In response to this request, in <FIG>, client device <NUM> loads a first webpage that serves as a "home" page for feuxfilms. com and includes, for instance, a list of movies opening this week and star ratings associated with those movies.

Also included on the webpage is a search bar <NUM> that is operable by the user to search documents associated with feuxfilm. com for keywords. As is common on webpages in particular and across GUIs in general, a visual cue <NUM> in the form of a magnifying glass icon is provided to draw the user's attention to the purpose of search field <NUM> (this may be beneficial if, for instance, the user doesn't speak English). In some cases, the search field may not include the word "SEARCH," as depicted in <FIG>, therefore making visual cue <NUM> even more useful.

In <FIG>, automated assistant <NUM> has been invoked (e.g., by a user pressing the microphone element <NUM> during the application phase, as indicated by its shading, or automatically during the training phase. The free-form natural language input, "search for Dracula," has been provided, e.g., by the user during the application phase or automatically during the training phase (e.g., selected from a batch of automatically-generated and/or human- curated free form natural language inputs). From this free-form input the user intent "search for" and the slot value "Dracula" are determined.

During the training phase, when there is no (validated) script available for automated assistant <NUM> to execute, techniques described herein may be employed to identify a target visual cue to be located in the GUI. In this example, the user intent is "search for," and the target visual cue is the magnifying glass icon (<NUM>) that is so frequently presented with search bars. As will be described in more detail shortly, in some implementations, a location of a detected instance of the target visual cue may be determined, e.g., using object recognition processing on a screenshot provided by client device <NUM>. Based on the location of the detected instance of the target visual cue, an interactive element of the webpage may be identified. In <FIG>, the location of target visual cue <NUM> (magnifying glass icon) can be associated spatially with search bar <NUM>, e.g., using heuristics or other techniques.

Once the interactive element (search bar <NUM>) is identified, as is the case in <FIG>, the text "Dracula" from the free-form natural language input may be input automatically into search bar <NUM>, e.g., without it being typed manually. When this occurs during the application phase, the user may see the search text being typed at a rate of speed that is perceptible to the user, e.g., so that the user can intuit what is happening. The search is then submitted automatically to yield the search results webpage depicted in <FIG>.

In <FIG>, search results for the movie "Dracula" are shown. These search results effectively comprise an outcome of the actions taken previously by automated assistant <NUM>. During the training phase, the actions taken previously may be validated by analyzing various aspects of the outcome of <FIG>, e.g., in comparison with a desired outcome. For example, the URL <NUM> in <FIG> has been updated to reflect a current webpage (search) and to include the search term "Dracula. " This may be used as a relatively strong signal that the intent of "search" and the slot value of "Dracula" were successfully resolved by automated assistant <NUM>. Additionally or alternatively, in some implementations, other content of the webpage of <FIG>, such as the search results themselves, may be used as an outcome that can validate that the actions taken previously by automated assistant <NUM> were successful. During the application phase, by contrast, the user may be able to select various links depicted in <FIG> in order to advance towards their goal of procuring a ticket to the movie "Dracula.

<FIG> depict another example of how techniques described herein may be applied during the training or application phases for a webpage that allows a user to select seats. This example follows the previous example, after the user has selected a particular showing for the movie "Dracula" depicted in <FIG>. In <FIG>, the URL <NUM> is updated to include "SP?<NUM>," which can be assumed to represent a "seat picker" ("SP") interface for the particular showing of "Dracula" the user selected in <FIG> (which is numbered <NUM> in this example). The interface of <FIG> prompts a user to select an open seat. The white seats are open, and the shaded seats are already taken.

In <FIG>, microphone element <NUM> has been actuated to invoke automated assistant <NUM>, and the query "Select third row, third and fourth from left" has been submitted. As was the case with the previous example and with other examples described herein, this action may be performed automatically as part of training (in which case the query may be randomly-generated or human-curated) or during application with an actual user interacting with the website.

In <FIG>, the seats specified in the query submitted in <FIG> are selected, as shown in black. In some implementations, particularly during training, the visual cue that is targeted based on the user's query is a pattern associated with the movie theater seats. Although seating arrangements vary amongst different movie theaters, patterns may still emerge, and these patterns may be used as visual cues. For example, assuming the screen is at top, the visual cue associated with the current query may seats that are three rows back from the screen, and then three and four seats from the left.

In some implementations, the training and application phases may not necessarily be entirely distinct. In fact, in many implementations, training may occur even when a user interacts with a webpage in real time. If a script was executed to aid the user's interaction with the webpage, the script can be verified (or flagged as dysfunctional) based on various signals generated by subsequent webpages and/or by the user. In <FIG>, for instance, the user may be able to see that the correct seats were selected. In some cases, the user's actuation of the "SELECT" button after seeing those seats selected automatically may serve as a strong signal that the script operated properly.

If no script was used (e.g., because it hasn't been generated yet for this particular URL, or it was previously generated but the movie theater rearranged its seats, rendering the previous script obsolete), then techniques described herein relating to performing image processing on a screenshot to identify a location of a target visual cue (particular seats) in a screenshot may be performed in real time. If the user sees that the correct seats were selected and presses the "SELECT" button, that may be used as a positive example to train, for instance, an object recognition machine learning model that was applied to the screenshot to identify the requested seats. If the user selects different seats than those that were selected automatically, that may be a negative training example.

<FIG> depicts a next screen that may be presented to the user. The URL <NUM> has been updated so that the film ("F") <NUM> remains active and the seats ("S") 3E and 3F are selected. As before, URL <NUM> can be used as a signal of the success or failure of a script that resulted in the interface of <FIG> (either during training or application). In <FIG>, the user is presented with a summary of their selection, some links that allow them to make changes, and a "SUBMIT" button. If the user changes seats, times, or theaters, that may be taken as a signal that a script (if available) didn't operate properly, or a signal that techniques described herein to identify a target visual cue's location didn't work properly (in which case no script should be generated). On the other hand, if the user presses "SUBMIT," the script and/or techniques may be validated.

A sought-after interactive element such as search bar <NUM> may not always be readily available on all webpages. Some websites may require users to click on an intermediate interactive element first, in order to expose a search bar. Additionally or alternatively, in some cases, a website may include a search interface on an intermediate menu that is rendered using client-side logic such as JavaScript code or other similar technology. Techniques described herein may nonetheless be employed to facilitate automatic interaction with these websites.

Turning now to <FIG>, in <FIG>, an alternative version of the website associated with the URL "feuxfilm. com" is shown on a client device <NUM>. There is no readily available search interface in <FIG>. Instead, there is an intermediate element that is commonly referred to as a "hamburger" graphical <NUM> that can be pressed to cause another menu to appear (other intermediate graphical elements are contemplated). For example, in <FIG>, hamburger icon <NUM> has been pressed to cause it to expand a menu and expose additional controls, including a "login" element and a search element that is identifiable visually based on visual cue <NUM>. In some implementations, when the search element of <FIG> is actuated, the result may be the interface depicted in <FIG>, which adds a fully functional search bar <NUM>. In other implementations, the menu that is exposed upon actuation of the hamburger graphical element <NUM> includes a fully functional search bar <NUM>, as is depicted in <FIG>.

In various implementations, techniques described herein may be used to generate scripts for automatic navigation of a GUI such as those depicted in <FIG>. For example, multiple visual cues may be identified and interacted with in succession in order to eventually expose a search bar that can be populated with data from a user intent. In some implementations, a single script may be generated that, for instance, locates and/or actuates a hamburger graphical element <NUM> to expose a next state of a GUI (e.g., using image processing of a screenshot as described herein). Then, another script may be generated to locate and interact with a search bar on the next state of the GUI, again using image processing of a screenshot as described herein.

<FIG> schematically depicts an example of how data may flow between and/or be processed by various components to implement selected aspects of the present disclosure, e.g., during the training phase to generate scripts for automatically interacting with GUIs. A plurality of GUIs <NUM> (e.g., individual webpages, states of application GUIs, etc.) are provided to and/or retrieved by website navigation engine <NUM>, e.g., from website database <NUM>.

In some implementations, website navigation engine <NUM> may include and/or operate one or more sand boxes <NUM>, which may include areas or memory that are cordoned off from others for security and/or safety purposes. For example, a sand box <NUM> could be implemented as a virtual machine that includes its own operating system. In some implementations, sand box <NUM> includes one or more instances of a web browser <NUM> that can be used to during the training phase to attempt to automatically interact with webpages in accordance with free-form natural language inputs (which may be human-curated and/or automatically generated). Although not shown in <FIG>, in some implementations, other non-web-browser applications may be implemented in sand box <NUM> in order to generate scripts for automatic interaction with other types of GUIs besides webpages.

In <FIG>, web browser <NUM> is operated to interact with GUIs <NUM>, which allows a screenshot module <NUM> to generate one or more screenshots <NUM>. These screenshots <NUM> may then be processed by a screenshot processing module <NUM> in various ways. In some implementations screenshot processing module <NUM> may have access to a database <NUM> of machine learning model(s) that are trained to identify particular visual cues in screenshots <NUM>, and to provide output indicative of, for instance, the location of detected instances of those visual cues within screenshot <NUM>. This output may be provided in some implementations to a script module <NUM>. The machine learning models employed by screenshot processing module <NUM> may take various forms, including but not limited to various forms of neural networks, such as convolutional neural networks (which are particularly well-suited for image processing).

The location of the detected instance of the target visual cue provided by screenshot processing module <NUM> may take various forms. In some implementations the location may take the form of a bounding box that captures or envelopes a detected instance of a target visual cue. Additionally or alternatively, in some implementations, screenshot processing module <NUM> may annotate a screenshot, e.g., with pixel-wise probabilities that each pixel renders a portion of a target visual cue. In other implementations, screenshot processing module <NUM> may provide a location (e.g., x,y coordinates) of a particular part of a target visual cue, such as its center, top-left corner, top-right corner, etc..

Script module <NUM> may be configured to process the location provided by screenshot processing module <NUM>, along with other data about the GUI under consideration, and generate a script that is usable subsequently to automatically interact with the GUI. In <FIG>, a DOM module <NUM> may be configured to obtain/extract, from GUIs <NUM>, underlying DOM information. In some implementations, the operations performed by DOM module <NUM> may be performed in whole or in part by web browser <NUM>.

The DOM information provided by DOM module <NUM> to script module <NUM> may be used by script module <NUM>, in conjunction with the location of the detected instance of the target visual cue, to identify a particular interactive element of the GUI that is associated with the detected instance of the target visual cue. In some implementations, CSS selectors in the DOM data may be searched based on a user intent being used to generate the script. For example, if the user intent is to "search," then a CSS selector for "search," "find," or some other semantically-similar token may be identified and matched to the location provided by screenshot processing module.

As indicated by the dashed arrow in <FIG>, in other implementations, DOM module <NUM> may provide the DOM data directly to screenshot processing module <NUM>, rather than to script module <NUM>. This may be the case, for instance, where screenshot processing module <NUM> is configured to provide output other than a spatial location of an interactive element that corresponds to a target visual cue. For example, in some implementations, screenshot processing module <NUM> may apply features of the DOM data as input across one or more machine learning models that are used in conjunction with the image processing machine learning model. The output generated based on these model(s) may include, for instance, one or more candidate DOM elements that may correspond to a detected instance of the target visual cue.

For example, suppose the user intent is to search, and the target visual cue is a magnifying glass. Screenshot processing module <NUM> may apply feature(s) of the screenshot <NUM> and the DOM data as inputs across one or more machine learning models. The output may include a list of DOM elements that are ranked based on the probability that they correspond to the user intent of search, as well as to a location within the screenshot <NUM> at which a detected instance of the magnifying glass icon was identified. In many cases one DOM element will be ranked far higher than others (e.g., where there is a single search bar on the GUI).

Script module <NUM> may be configured to generate scripts based on data it receives from screenshot processing module <NUM>, and from other sources as applicable. Scripts it generates may be stored in database of scripts <NUM>. In various implementations, a validation module <NUM> may periodically and/or on demand cause scripts in database <NUM> to be executed, e.g., in order to validate those scripts as still functional. This may occur daily, weekly, monthly, etc. In some implementations, GUIs that are more popular and/or that are known to change more frequently may be validated more often than other, less popular and/or less dynamic GUIs.

For example, validation module <NUM> may retrieve a script from database <NUM>, and cause web browser <NUM> to retrieve the corresponding GUI <NUM> (e.g., a webpage associated with the same URL as the script) and execute the script on the retrieved GUI <NUM> using a human-curated or automatically generated free-form natural language input. Validation module <NUM> may validate an outcome of the script's execution by, for instance, searching a resulting URL for terms extracted from the free-form natural language input (e.g., as might be found in the user intent that results from the free-form natural language input). If the outcome is successful, or if some threshold percentage of outcomes obtained by executing the script numerous times are successful, the script may be retained in database <NUM> for future use.

<FIG> is a flowchart illustrating an example method <NUM> according to implementations disclosed herein. For convenience, the operations of the flow chart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of automated assistant <NUM> and/or GUI navigation engine <NUM>. Moreover, while operations of method <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added. The operations of <FIG> will be described in the context of generating scripts during the training phase. However, it should be understood that some of these operations may also be performed during the application phase, e.g., to validate a just-executed script.

At block <NUM>, the system may determine a user intent to interact with a particular GUI based at least in part on a free-form natural language input. As should be clear, the term "user intent" as used herein is not limited to an actual intent of an actual user in real time. Rather, the term "user intent" refers to a collection of data, such as a data structure, that is generated, e.g., by intent matcher <NUM> in <FIG>, from processing a free-form natural language input. The free-form natural language input may be human-curated and/or automatically generated. For example, a list of test search queries could be generated, with the first portion being "search for" and then follow-up terms being selected randomly from a dictionary and used as slot values.

Based on the user intent determined at block <NUM>, at block <NUM>, the system may identify a target visual cue to be located in the GUI. For example, if the user intent is to "search," then a magnifying glass visual cue may be identified. If the user intent is to select a seat, then a visual cue may take the form of an arrangement of seats (or icons representing seats) in a GUI that allows for picking of seats. If the user intent is to select a destination (e.g., from a travel website), the visual cue may take the form of a bed icon. If the user intent is to select dates, the visual cue may take the form of a calendar icon or an arrangement of fields often provided to enable input of dates. If the user intent is to select a number of people (e.g., number of guests for hotel room), or if the user intent is to log in to an account, the visual cue may take the form of an icon that depicts a bust (head, shoulders, part of chest) of a faceless person. In some implementations, optical character recognition ("OCR") processing may be performed on the screenshot, and the user intent may be matched to text output of the OCR processing.

At block <NUM>, the system, e.g., by way of screenshot processing module <NUM>, may perform object recognition processing on a screenshot of the GUI to determine a location of a detected instance of the target visual cue in the screenshot. For example, screenshot processing module <NUM> may apply data indicative of the screenshot as input across one or more machine learning models, alone or in combination with other data (e.g., features of a DOM), to generate output. This output may be indicative of, for instance, x,y coordinates of the detected instance of the target visual cue, and may take various forms, such as a bounding box, annotated pixel-wise probabilities, etc..

Based on the location of the detected instance of the target visual cue determined at block <NUM>, at block <NUM>, the system may identify an interactive element of the GUI. For example, script module <NUM> may analyze the DOM data to determine where on a display each DOM element would likely be displayed. Those locations may then be compared to the location of the detected instance of the target visual cue. Various heuristics may be applied to determine which interactive element (e.g., corresponding to a DOM element) is closest to, and therefore most likely associated with, the detected instance of the target visual cue.

At block <NUM>, the system may automatically populate the identified interactive element with data determined from the user intent. For example, if the user intent is to "search," and the parameters of the search are "energy-efficient dishwashers," then "energy-efficient dishwashers" may be input into a search bar. If the user intent is to "select a date," and the slot-values are {month=December, day=<NUM>, year=<NUM>), then an interactive calendar date picking element may be automatically interacted with to select the appropriate date.

In some implementations, at block <NUM>, the system may automatically (e.g., without a user instruction) submit the data determined from the user intent (as it was used to populate the interactive element), e.g., by automatically actuating a "SUBMIT" button or other similar interface element. If the GUI under question is a webpage, in some cases, the submission may cause HTTP data to be submit to a web server, which can then generate the next webpage based at least in part on the data from the user intent.

At block <NUM>, the system may receive (e.g., retrieve, obtain) a subsequent GUI generated based at least in part on the data determined from the user intent. As noted above, in some implementations, this next GUI may be the next webpage generated based at least in part on the HTTP data that included the data from the user intent. At block <NUM>, the system, e.g., by way of validation module <NUM>, may validate that submission of the data at block <NUM> resulted in a desired outcome. For example, the system may search for terms from the user intent in a URL of the next webpage. As another example, the system may analyze content of the next webpage to determine whether the terms from the user intent are present or otherwise represented.

In response to the validation of block <NUM> and based on the identified interactive element, at block <NUM>, the system may generate a script that is subsequently executable in association with the same GUI. In some implementations, the script may be stored in database <NUM> in association with the URL. The script may take various forms and be written in various languages, including but not limited to JavaScript.

In some implementations, one or more of a client computing device, GUI navigation engine <NUM>, automated assistant <NUM>, and/or other component(s) may comprise one or more components of the example computing device <NUM>.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of the method of <FIG>, as well as to implement various components depicted in <FIG> and <FIG>.

Claim 1:
A method implemented using one or more processors, comprising:
determining (<NUM>) a user intent to interact with a particular graphical user interface, GUI, based at least in part on a free-form natural language input;
based on the user intent, identifying (<NUM>) a target visual cue (<NUM>) to be located in the GUI;
performing (<NUM>) object recognition processing on a screenshot of the GUI to determine a location of a detected instance of the target visual cue in the screenshot;
based on the location of the detected instance of the target visual cue, identifying (<NUM>) an interactive element (<NUM>) of the GUI associated with the detected instance of the target visual clue; and
automatically populating (<NUM>) the identified interactive element with data determined from the user intent.