Patent ID: 12189697

DETAILED DESCRIPTION

Various technologies pertaining to providing content retrieved by a search engine and/or content obtained by a web browser as a part of a prompt to a generative model are now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident, however, that such aspect(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by multiple components. Similarly, for instance, a component may be configured to perform functionality that is described as being carried out by multiple components.

Moreover, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from the context, the phrase “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, the phrase “X employs A or B” is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from the context to be directed to a singular form.

Further, as used herein, the terms “component”, “system”, “engine”, and “module” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor. The computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices.

The technologies described herein are related to “grounding” a generative model with information that is usable by the generative model to generate output. Grounding the generative model refers to providing the generative model with context that is usable by the generative model to generate output, where the context is in addition to user-generated input. With more specificity, a generative model generates output based upon a prompt; conventionally, the prompt includes input generated by a user who is interacting with the generative model during a conversation (such as a query set forth by a user), previous inputs set forth by the user during the conversation, previous outputs generated by the generative model during the conversation, and previously defined instructions that describe how the generative model is to generate output. The technologies described herein relate to inclusion of additional information in the prompt, where such additional information can be obtained from a search engine (based upon a query generated by a user and/or based upon a query generated by the search engine). In another example, the additional information can be obtained from a web browser (or other application) that has loaded a webpage being viewed by a user. The generative model generates output based upon this additional information in the prompt, which leads to an increase in likelihood that the output generated by the generative model is factually accurate and/or based upon content being viewed by the user.

Referring now toFIG.1, a functional block diagram of a computing system100is illustrated. While illustrated as a single system, it is to be understood that the computing system100can include several server computing devices, can be distributed across data centers, etc. The computing system100is configured to obtain information based upon a query set forth by a user and is further configured to provide the obtained information as a portion of a prompt to a generative model.

A client computing device102operated by a user (not shown) is in communication with the computing system100by way of a network104. The client computing device102can be any suitable type of client computing device, such as a desktop computer, a laptop computer, a tablet (slate) computing device, a video game system, a virtual reality or augmented reality computing system, a mobile telephone, a smart speaker, or other suitable computing device.

The computing system100includes a processor106and memory108, where the memory108includes instructions that are executed by the processor106. More specifically, the memory108includes a search engine110and a generative model112, where operations of the search engine110and the generative model112are described in greater detail below. In an example, the generative model112is a generative language model (GLM), although it is to be understood that the generative model112can output images, video, etc. The computing system106also includes data stores114-122, where the data stores114-122store data that is accessed by the search engine110and/or the generative model112. With more particularity, the data stores114-122include a web index data store114, an instant answers data store116, a knowledge graph data store118, a supplemental content data store120, and a dialog history data store122. The web index data store114includes a web index that indexes webpages by keywords included in or associated with the webpages. The instant answers data store116includes an index of instant answers that are indexed by queries, query terms, and/or terms that are semantically similar or equivalent to the queries and/or query terms. For example, the instant answer “2.16 meters” can be indexed by the query “height of Shaquille O'Neal” (and queries that are semantically similar or equivalent, such as “how tall is Shaquille O'Neal”).

The knowledge graph data store118includes a knowledge graph, where a knowledge graph includes data structures about entities (people, places, things, etc.) and their relationships to one another, thereby representing relationships between the entities. The search engine110can use the knowledge graph in connection with presenting entity cards on a search engine results page (SERP). The supplemental content data store120includes supplemental content, such as electronic advertisements, that can be returned by the search engine110based upon a query.

The dialog history data store122includes dialog history, where the dialog history includes dialog information with respect to users and the generative model112. For instance, the dialog history can include, for a user, identities of conversations undertaken between the user and the generative model112, input provided to the generative model112by the user for multiple dialog turns during the conversation, dialog turns in the conversation generated by the generative model112in response to the inputs from the user, queries generated by the generative model112during the conversation that are used by the generative model112to generate responses, and so forth. In addition, the dialog history can include context obtained by the search engine110during conversations; for instance, with respect to a conversation, the dialog history122can include content from search results identified based upon queries set forth by the user and/or the generative model112during the conversation, content from webpages identified by the search engine110based upon queries set forth by the user and/or the generative model112during the conversation, and so forth. The data stores114-122are presented to show a representative sample of types of data that are accessible to the search engine110and/or the generative model112; it is to be understood that there are many other sources of data that are accessible to the search engine110and/or the generative model112, such as data stores that include real-time finance information, data stores that include real-time weather information, data stores that include real-time sports information, data stores that include images, data stores that include videos, data stores that include maps, etc. Such sources of information are available to the search engine110and/or the generative model112.

The search engine110includes a web search module124, an instant answer search module125, a knowledge module128, a supplemental content search module130, and a SERP constructor module132. The web search module124is configured to search the web index data store114based upon queries received by users, queries generated by the search engine110based upon queries received by users, and/or queries generated by the generative model112based upon interactions of users with the generative model112. Similarly, the instant answer search module126is configured to search the instant answers data store116based upon queries received by users, queries generated by the search engine110based upon queries received by users, and/or queries generated by the generative model112based upon interactions of users with the generative model112. The knowledge module128is configured to search the knowledge graph data store118based upon queries received by users, queries generated by the search engine110based upon queries received by users, and/or queries generated by the generative model112based upon interactions of users with the generative model112. Likewise, the supplemental content search module130is configured to search the supplemental content data store120based upon queries received by users, queries generated by the search engine110based upon queries received by users, and/or queries generated by the generative model112based upon interactions of users with the generative model112.

The SERP constructor module132is configured to construct SERPs based upon information identified by searches performed by the modules124-130; for instance, a SERP can include links to webpages identified by the web search module124, an instant answer identified by the instant answer search module126, an entity card (that includes information about an entity) identified by the knowledge module128, and supplemental content identified by the supplemental content search module130. Further, a SERP may include a widget, a card that depicts current weather, and the like. The SERP constructor module132can also generate structured, semi-structured, and/or unstructured data that is representative of content of the SERP or a portion of the content of the SERP. For instance, the SERP constructor module132generates a JSON document that includes information obtained by the search engine110based upon one or more searches performed over the data stores114-120(or other data stores). In an example, the SERP constructor module132generates data that is in a structure/format to be used as a portion of a prompt by the generative model112.

As discussed above, operation of the search engine110is improved through use of the generative model112, and operation of the generative model112is improved through use of the search engine110. For instance, the search engine110is able to provide outputs that the search engine110was not previously able to provide (e.g., based upon outputs generated by the generative model112), and the generative model112is improved by using information obtained by the search engine110to generate outputs (e.g., information identified by the search engine110can be included as a portion of a prompt used by the generative model112to generate outputs). Specifically, the generative model112generates outputs based upon information obtained by the search engine110, and accordingly the outputs have a higher likelihood of being accurate when compared to outputs generated by generative models that are not based upon such information, as the search engine110is associated with years of development in connection with curating information sources to ensure accuracy thereof.

Examples of operation of the computing system100are now set forth. It is to be understood that these examples are non-limiting, and that permutations of such examples are contemplated. Referring toFIG.2, a communications diagram200depicting communications between the client computing device102, the search engine110, and the generative model112in accordance with a first example is presented. The client computing device102is executing an application such as a web browser, and the application receives a query (user input) that is to be transmitted to the search engine110executing at the computing system100. For instance, the web browser can load a homepage of the search engine and the web browser receives a query from a user in a text entry field. In another example, the web browser can receive a query in an address field of the web browser. In yet another example, the webpage can be associated with the generative model112(e.g., a user can access the generative model112directly rather than through a search engine interface). At202, the client computing device102transmits the query to the search engine110. Alternatively, the client computing device102transmits the query to the generative model112, which forwards the query to the search engine110.

The search engine110performs at least one search over at least one of the data stores114-122based upon the query and identifies search results. As indicated previously, the search results can include identities of webpages, an instant answer, a widget, supplemental content, an entity card, and the like. In an example, the search engine110can construct several queries based upon the query, the location of the client computing device102, a profile of the user of the client computing device102, etc., and can search over the data stores using queries identified by the search engine110as being well-suited for use when searching over such data stores114-120.

At204, the search engine110provides the search results (or a portion thereof) to the generative model112. In an example, the search engine110provides the generative model112with search information that the search engine110has identified as being relatively highly likely to be relevant to the user (e.g., information from an entity card, information pertaining to a threshold number of most highly ranked webpages, etc.). In a more specific example, the search engine provides the generative model112with information according to a predefined priority level: 1) when the search engine identifies an instant answer, entity card, or the like, the search engine provides information in the instant answer, the entity card, or the like to the generative model; 2) for a top threshold number of most highly ranked webpages (e.g., the three most highly ranked webpages), the search engine110provides the URL for such webpages, titles of such webpages, and content extracted from such webpages based upon the query to the generative model112. The content extracted from the webpages can be or include a snippet that is extracted from a webpage by the search engine (where the snippet is displayed with a link to the webpage in the search results page). In another example, the content extracted from the webpages can be portions of such webpages that are identified by the search engine110as being most relevant portions with respect to the input query.

Optionally, at206, the search engine110returns the search results (or a portion thereof) to the client computing device102, whereupon the search results are presented to the user. While the communications diagram200depicts the search information being provided to the generative model112before the search results are provided to the client computing device102, it is to be understood that the search engine110can provide the search results to the client computing device102prior to providing the search information to the generative model112. In addition, the search engine110may refrain from providing the search information to the generative model112until further user input is received (e.g., indicating that the user intends to interact with the GLM112).

The client computing device102receives an indication that the user thereof intends to interact with the generative model112. For example, the SERP can include a selectable interactive element that initiates a conversation with the generative model112. In another example, an interface by way of which the user of the client computing device102can interact with the generative model112can be initiated on the SERP by way of a gesture (e.g., an upward or downward scroll). The client computing device102receives (conversational) input from the user that is to be provided to the generative model112as part of a prompt, and at208the input is provided to the generative model112. WhileFIG.2indicates that the input is provided directly from the client computing device102to the generative model112, in an example the client computing device102(the web browser executing on the client computing device102) transmits the input to the search engine110, which then forwards the input to the generative model112. Further, if not provided previously, the search engine110can transmit the search information (shown as being transmitted at204) to the generative model112after receiving an indication that a conversation with the generative model112has been initiated.

Optionally, the generative model112generates a second query that is well-suited for use by the search engine110in connection with identifying further search results that are relevant to the input, where the generative model112generates the second query based upon the search information received at204and the input received at208. Thus, in contrast to conventional approaches, the generative model112generates the second query based not only upon the input, but generates the second query based further upon search results identified by the search engine110. In another example, the generative model112generates a query based solely upon the input. The generative model112can query the search engine upon predicting that additional information is desired in connection with providing a useful response to the user; for instance, the generative model112can predict that information that the generative model112has available to generate output is stale, and can query the search engine after predicting that the information is stale. At210, the second query generated by the generative model112is provided to the search engine110, and the search engine110searches over at least one of the data stores114-120based upon the second query. The search engine110identifies second search results based upon the second query and provides information extracted from the second search results as second search information to the generative model112at212, where the second search information is included in a prompt for the generative model112.

The generative model112generates output based upon the search information obtained at204and the input obtained at208. In addition, optionally, the generative model112generates the output based upon the second search information obtained at212. Again, this is contrary to conventional approaches, where generative models generate output based solely upon input set forth by the user, conversation history (when available), and predefined instructions. The generative model112transmits the output to the client computing device102at214(optionally by way of the search engine110).

The generative model112generates output based upon the search information provided thereto at204and the input transmitted to the generative model112at208. Thus, the prompt employed by the generative model112to generate the output includes the search information provided to the generative model112by the search engine110at204and the input received from the client computing device102at208. The generative model112outputs the output, and the output is provided to the client computing device102for presentment to the user thereof. Moreover, the generative model can include citations to sources used by the generative model to generate the output. Thus, when the generative model is112provided with content from a webpage by the search engine110and generates output based upon such content, the generative model112can include a citation to the webpage in the output, thereby informing the user of source of the output generated by the generative model112.

This process can repeat as the generative model112receives additional inputs from the user of the client computing device102. For instance, the generative model112may receive second input from the client computing device102; upon receipt of such input, the generative model112can construct a third query, where the generative model112constructs the third query based upon: 1) the query initially obtained by the search engine110at202; 2) at least some search results identified by the search engine110based upon the query; 3) the input obtained at208; 4) the second query generated by the generative model112; 4) the second search results generated by the search engine110; 5) the output generated by the generative model112and provided to the client computing device102at214; and 6) the second input received from the user. The generative model112provides the third query to the search engine110, which performs another search based upon the third query and identifies third search results. Information extracted from the third search results are provided to the generative model112, which uses such information to generate second output.

While the examples referenced above have referred to the generative model112generating text, it is understood that the generative model112can generate images or other multimedia; further, the generative model112can request that the search engine110perform visual searches (e.g., provide the search engine110with an image and request that the search engine110identify similar images).

Moreover, in another example, the generative model112can initially receive input (rather than the search engine110initially receiving the query at202). In such an embodiment, the generative model112provides the input to the search engine110and/or generates a query based upon the input and provides the query to the search engine110. The search engine110identifies search results based upon the input and/or the query and provides at least a portion of the search results to the generative model112as search information, where the search information is included in a prompt for the generative model112. The generative model112can then generate the output; therefore, the search results need not be directly provided to the user and the user can initially set forth the query to the generative model112(rather than initially setting forth the query to a search engine home page or other entry page associated with a search engine).

Turning now toFIG.3, a graphical user interface (GUI)300of a SERP that depicts interaction between the search engine110and the generative model112in accordance with the communications diagram200ofFIG.2is presented. As illustrated inFIG.3, the client computing device102has received the query “famous people born in Chicago” from the user and has transmitted the query to the search engine110. The search engine110searches at least one of the data stores114-120based upon the query, and the SERP constructor module132constructs the SERP depicted inFIG.3, where the SERP includes a list of links302that point to webpages identified by the search engine110as being relevant to the query, an image304identified by the search engine110as being relevant to the query, supplemental content306and308identified as being relevant to the query, and so forth.

The client computing device102receives an indication from the user that a conversation is to be initiated with the generative model112(e.g., the user can select a button or the like), resulting in an interface310being presented that is configured to receive (conversational) inputs and display (conversational) outputs. The client computing device102then receives the input “What about Al Capone?”, and the query is provided to the generative model112(directly from the client computing device102or by way of the search engine110). The generative model112is further provided with at least some information from the search results identified by the search engine110as being relevant to the aforementioned query. In addition, and optionally, the generative model112generates a second query based upon: 1) the input; 2) the search results identified by the search engine110; and/or 3) the original query received by the search engine110. For instance, the GLM112generates the query “birthplace of Al Capone” and provides such query to the search engine110. The search engine110conducts a search based upon such query and generates search results (where the search engine110may or may not cause such search results to be displayed to the user). The generative model112receives information extracted from the search results, and such information is included in the prompt used by the generative model112to generate output (e.g., conversational output). The prompt can include: 1) the initial query “famous people born in Chicago”; 2) first search information extracted from the initial set of search results identified by the search engine110based upon the initial query; 3) the input set forth by the user “what about Al Capone”; 4) optionally, second search information extracted from second search results identified by the search engine based upon the query “what about Al Capone”; 5) the query generated by the generative model112(“birthplace of Al Capone”); and third search information extracted from search results identified by the search engine110based upon the query “birthplace of Al Capone”.

The generative model112can generate the conversational output “Al Capone was born in New York City in 1899” based upon the prompt. This approach increases the likelihood that the output generated by the generative model112is factually accurate, as the generative model112generates the output based upon search results identified and ranked by the search engine110(rather than based generally upon training data that is not curated based upon information retrieval intent of users).

As illustrated inFIG.3, after being provided with the output “Al Capone was born in New York City in 1899”, the client computing device102receives input “Michael Jordan?” from the user, and the generative model112is provided with such input. The generative model112constructs a query based upon the conversation context (the information in the prompt referenced above, the output generated based upon the prompt, the input “Michael Jordan?”, and optionally information extracted from search results identified by the search engine110based upon such conversational input). In an example, the generative model112constructs the query “birthplace of Michael Jordan” and provides such query to the search engine110. The search engine110identifies search results based upon the query received from the generative model112, extracts information from these search results, and provides the information to the generative model112. The generative model112can then generate output (e.g., “Michael Jordan was also born in New York City (1963)”) based upon at least some of the aforementioned context. As size of the prompt that can be provided to the generative model112is limited, techniques can be employed to identify appropriate information to include in the prompt. In an example, the prompt can be managed on a first in-first out basis, such that older information is removed while newer information is retained in the prompt. In another example, information can be assigned relevance scores, and information with the highest relevance scores can be retained in the prompt while information with lowest relevance scores can be removed. A combination of such approaches can also be employed. For example, a relevance score can be used as a filter, such that information with a relevance score beneath a threshold is removed; afterwards, a first in first out approach can be employed.

It is again emphasized thatFIG.3represents an improvement over both conventional search engines as well as conventional generative models. Utilizing the technologies described herein, the GUI300of the search engine110presents information to the user that was previously unable to be presented—a conversational interface that provides the user with information that is of interest to the user. In addition, the generative model112provides information to the user that is relatively up-to-date (as the search engine110returned such information as a relevant search result) and is factually accurate.

Now referring toFIG.4, another communications diagram400depicting communications between the client computing device102, the search engine110, and the generative model112is illustrated. The client computing device102receives a query from the user (e.g., by way of a GUI of a search engine, an address field of a web browser, etc.). The client computing device102transmits the query to the search engine110at402. The search engine110executes a search based upon the query (searches at least one of the data stores114-120based upon the query). In an example, the search engine110identifies an instant answer based upon the query.

At404, the search engine110transmits the instant answer to the client computing device102. For instance, the instant answer is a graphical element that includes an answer to the query set forth by the user, and further includes an input field by way of which text (or other suitable input) directed to the generative model112can be received. The client computing device102receives input by way of such text entry field, and at406the client computing device402transmits the input to the generative model112(e.g., optionally by way of the search engine110). At408, the generative model112obtains the instant answer identified by the search engine110based upon the query received at402. Further, optionally, the generative model112requests information extracted from webpages identified by the search engine110as being relevant to the query. For instance, the information can be extracted from the N most highly ranked webpages, where N can be 3, 5, or 10. In another example, when the input is received by the generative model112by way of the search engine110, the search engine110provides the generative model112with the instant answer upon receiving an indication that conversational input is being set forth to the generative model112.

The generative model112generates a second query based upon, for example, the first query received at the search engine110, the instant answer to the search query provided by the search engine110, (optionally) the information extracted from the N most highly ranked webpages, and the conversational input received by the generative model112. Notably, since the interaction was received by way of an interface associated with the instant answer, the search results provided to the generative model112can be limited to the instant answer (and may not include other search results identified by the search engine110). The generative model112provides the second query to the search engine110at410, and the search engine110generates updated search results based upon the second query. The search engine110provides the updated search result to the generative model112at412. The generative model112generates (conversational) output based upon the conversational input obtained by the generative model112, the second query generated by the generative model112, and the search results obtained from the search engine110by the generative model112. The generative model112provides the conversational output to the client computing device102at414(e.g., by way of the search engine110).

Now referring toFIG.5, a GUI500is set forth as an example that accords to the communications diagram400illustrated inFIG.4. The GUI500can be a search engine GUI, where the user of the client computing device102has set forth the query “company A stock price.” The search engine110receives the query and generates a SERP based upon the query. The SERP includes a list of search results502identified based upon the query, and further includes an instant answer504identified by the search engine110based upon a search of the instant answers data store116. The instant answer504can include an identifier for the company referenced in the received query, a graphical indicator that identifies a trend of the stock price, and a price of the stock for Company A. The instant answer504can further include a text entry field506by way of which the user of the client computing device102can set forth conversational input that is to be provided to the generative model112. In the example illustrated inFIG.5, the text entry field506receives the conversational input “how about company B?”

In this example, the generative model112obtains the query submitted to the search engine110by the user and further obtains content of the instant answer identified by the search engine110based upon the query. The generative model112can generate a second query based upon the query submitted by the user and content of the instant answer504, and the generative model112can submit this second query to the search engine110. For instance, the second query is “stock price of company B”. The search engine110searches at least one of the data stores114-120based upon the second query provided to the search engine110by the generative model112and identifies updated search results based upon the second query. The search engine110provides these updated search results to the generative model112, and the generative model112generates output based upon: 1) the initial query submitted to the search engine110by the user; 2) content of the instant answer504identified by the search engine110based upon the query; 3) the input set forth to the generative model112by the user of the client computing device102; 4) the second query generated by the generative model112(e.g., “stock price of company B”); and 5) the updated search results identified by the search engine110based upon the second query. As illustrated inFIG.5, the generative model112can generate the output “the stock price of Company B is $50.75”.

Conventional generative models are unable to provide this output, as conventional generative models are trained based upon stale data and are unable to obtain recent information. Utilizing the technologies described herein, however, the generative model112can generate output that includes recent (and accurate) data, as such data was provided to the generative model112as part of the prompt used by the generative model112to create output.

In the example illustrated inFIG.5, it is emphasized that the instant answer includes an interface by way of which a conversation can be initiated with the generative model112. It is further emphasized that, since the conversation with the generative model112was initiated by way of the instant answer504, the context used by the generative model112to generate conversational output can be limited to what is included in the instant answer504(rather than the context including information shown in other search results identified by the search engine110). Further, similar features are contemplated with respect to entity cards, widgets, etc. returned by the search engine.

With reference now toFIG.6, a functional block diagram of the computing system100is presented. The computing system100is in communication with the client computing device102, and the client computing device102includes a processor602and memory604, where the memory604has a web browser606loaded therein. The web browser606has a generative model interface608incorporated therein, where the web browser606provides information to the generative model112by way of the generative model interface608. Similarly, the generative model112provides information to the web browser606by way of the generative model interface608.

A web server610is in communication with the client computing device102by way of the network104. The web server610hosts a website that includes a webpage612. The web browser606is configured to retrieve webpages of the website hosted by the web server610. Accordingly, the web browser606can retrieve the webpage612from the web server610(e.g., based upon user input, based upon programmatic input, etc.).

Because the generative model112is in communication with the browser606, the generative model112can be provided with any suitable information that can be obtained by the browser606as part of a prompt (in addition to conversational input set forth by a user of the client computing device102). For example, the webpage612retrieved by the web browser606can include an e-mail sent to the user. Presuming that the user has provided authorization for the generative model112to obtain part of such e-mail, in response to the web browser606receiving an indication that a conversation between the user and the generative model112is to be initiated, the web browser606can provide information from the webpage612(e.g., the email) to the generative model112. In addition, the web browser606receives input from the user and provides the generative model112with such input. Accordingly, the prompt provided to the generative model112includes not only the input set forth by the user, but additionally includes content of the webpage612being viewed by the user. Therefore, the generative112can generate output based upon content of the webpage612. In an example, the content of the webpage612provided to the generative model112is an entirety of text of the webpage612. In another example, the content of the webpage612provided to the generative model112is all content on the webpage612, including an image, a video, etc. of the webpage. In yet another example, the content of the webpage612provided to the generative model112is a portion of content of the webpage selected by the user. In still yet another example, the content of the webpage612corresponds to a threshold number of tokens that can be included in a prompt for the generative model112(e.g., 7000 tokens, where each token represents one or more characters).

Further, the generative model112can generate a query based upon the content of the webpage612provided by the browser606and/or the input set forth by the user. The generative model112provides the query to the search engine110, and the search engine110searches at least one of the data stores114-120based upon the query. The search engine110provides the generative model112with search results identified by the search engine110based upon the query, thereby providing additional context for the generative model112to employ when generating output to the user. More specifically, the prompt employed by the generative model112to generate output can additionally include the search results identified by the search engine110.

Referring now toFIG.7, a communications diagram700depicting an example flow of communications between the web browser606, the web server610, the generative model112, and the search engine110is presented. In an example, a user of the client computing device102sets forth a request to the web browser606to obtain the webpage612from the web server610. The web browser606transmits the request for the webpage612to the web server610at702. At704, the web server610transmits the webpage612to the web browser606, whereupon the webpage612is presented on a display of the client computing device102by the web browser.

While the webpage612is loaded in the web browser606, the user of the client computing device102can set forth an indication to the web browser606that input is to be provided to the generative model112from the user. In an example, the web browser606may include a button, wherein upon the button being selected by the user, a text entry field for setting forth conversational input to the generative model112is presented in the web browser606. The web browser606can then receive input from the user by way of the text entry field (or a voice command). At706, the web browser606transmits the input received from the user to the generative model112. Additionally, at708, the web browser606transmits content from the webpage612(or information about the webpage612known by the web browser606) to the generative model112. Hence, the generative model112is provided with both the input and content of the webpage612as at least a part of a prompt to be used by the generative model112to generate output.

Optionally, the generative model112generates a query based upon the received input and/or the content from the webpage612. In an example, the generative model112can generate several queries, with each query being well-suited for use by the search engine110to search over a respective data store in the data stores114-120. At710, the generative model112transmits a query generated by the generative model112to the search engine110. The search engine110generates search results based upon the query and provides the search results (or information about the search results) to the generative model112at712. The generative model112can generate output that is responsive to the input received at706, where the generative model112generates the output based upon the input, the content of the webpage612received from the web browser606, and optionally the search results received from the search engine110. At714, the generative model112transmits the output to the web browser606and the web browser606presents the output to the user of the client computing device102. While the communications flow diagram700depicts one dialogue turn, it is to be understood that the process can repeat and the generative model112can be provided with further context.

Various alternatives are also contemplated. For example, while the client computing device102has been described as executing the web browser606, it is understood that any suitable application that presents content to users can have the generative model interface608incorporated therein. Therefore, for instance, an e-mail application can include the generative model interface608, a word processing application may include the generative model interface608, and so forth. Further, while the generative model112has been described herein as being executed remotely from the client computing device102, it is contemplated that at least a portion of the generative model112may execute on the client computing device102(e.g., at least a portion of the generative model112may be included in the web browser606or other suitable application that is executing on the client computing device102).

With reference now toFIG.8, a GUI800of a webpage that depicts an email inbox of a user is presented. The GUI800can include several selectable buttons802-810that correspond to functionality associated with the email inbox, such as a button that is associated with sending an email, a button that is associated with refreshing the inbox, and so forth. The GUI800may include a pane812that depicts a list of emails in the inbox of a user. The GUI800can further include a window814that can include content of an email being read by the user. As noted above, the web browser606can include the generative model interface608, such that the web browser606can cause a conversation pane816to be presented by way of which the user can set forth conversational input to the generative model112(e.g., as an overlay on the webpage represented inFIG.8).

In the example shown inFIG.8, the email in the window814includes information about holidays for the user over the upcoming calendar year. The user can set forth input by way of the conversation pane816, and the web browser606transmits the input to the generative model112by way of the generative model interface608. In addition, the web browser606can provide information from the email shown in the window814to the generative model112by way of the generative model interface608. Therefore, the prompt used by the generative model112to generate conversational output includes the content of the email. As illustrated, the user has set forth the input “what days off do I have in May?” As the generative model112is provided with both the input and the content of the email shown in the window814, the generative model112can generate output that accurately addresses the input (identifying the holidays of the user in the month of May). Conventionally, a generative model is unable to appropriately respond to such input, as the conventional generative model does not have access to the information requested by the user.

Referring now toFIG.9, another example GUI900is presented. The GUI900corresponds to a webpage that includes information about the baseball player Babe Ruth. The GUI900includes a window902that comprises information about Babe Ruth, which may include season by season statistics (e.g., batting average, home runs, strikeouts, etc.). The GUI900additionally includes the conversation pane816. When viewing the webpage, the user may set forth an indication that a conversation is desired with the generative model112. For instance, by way of the conversation pane816, the user sets forth the input “how many home runs did he hit before age 30?”. The web browser606provides the input and at least some content of the webpage to the generative model112. In an example, the generative model112generates queries based upon the conversational input and the content of the webpage. The generated queries may include “how many home runs did Babe Ruth hit”, “when was Babe Ruth born”, “home runs hit by Babe Ruth in 1923”, etc. The generative model112provides the queries to the search engine110, and the search engine110searches at least one of the data stores114-120based upon these queries. The search engine110provides at least some search results obtained based upon the queries to the generative model112, and the generative model112uses the search results (in combination with the content from the webpage, the conversational input, queries generated by the generative model112, and optionally other information) to generate output.

Further, for fact-based queries, the generative model112can optionally be configured to provide some indication as to confidence in factual accuracy of the output. In an example, the more information relied upon by the generative model112from the webpage and the search results to generate the output (and the less that the generative model112relies upon the underlying training data to generate the output), the higher the confidence in factual accuracy of the conversational output.

Referring now toFIG.10, a schematic illustrating a computing system1000in communication with a client computing device1002is presented. The computing system1000includes a processor1004and memory1006. The memory1006includes instructions that are executed by the processor1004. As illustrated, the memory1006includes a classifier1008and several generative models1010-1012. In an example, the first generative model1010utilizes a first number of graphical processing units (GPU) when generating an output based upon an input, while the nth generative model1012utilizes a second number of GPUs when generating an output based upon an input. For instance, the first number of GPUs is larger than the second number of GPUs.

In another example, the first generative model1010is trained for a first topic, where the first generative model1010is trained based upon content from webpages labeled as corresponding to the first topic; the nth generative model1012is trained for a second topic, where the nth generative model1012is trained based upon content from webpages labeled as corresponding to the nth topic. Hence, the first generative model1010is well-suited to generate outputs that correspond to the first topic (but not the nth topic) while the nth generative model1012is well-suited to generate outputs that correspond to the nth topic (but not the first topic).

In yet another example, the first generative model1010is configured to receive a prompt of a first size as input, while the nth generative model1012is configured to receive prompt of a second size as input (where the second size is greater than the first size).

The classifier1008is configured to assign a class label from amongst several possible class labels to input received from the client computing device1002; a generative model from amongst the generative models1010-1012is selected based upon the class label, and the selected generative model is provided with the input upon being selected. For instance, the classifier1008is employed to select appropriate generative models such that outputs are of high quality yet computing resources needed to generate the outputs are reduced.

In operation, the computing system1000receives, from the client computing device1002, input set forth by a user of the client computing device1002. Optionally, as described above, the input can be provided to a search engine, where the search engine identifies search results based upon the input. The classifier1008obtains the input set forth by the user of the client computing device1002and optionally the search results identified by the search engine, and the classifier1008assigns a class label to such input. In an example, the classifier1008can assign a class label to input, where the class label indicates that the input is relatively complex (such as the input “can you provide me with instructions for removing, cleaning, and reinstalling a carburetor from a 1974 truck”). Based upon the assigned class label, the computing system1000selects a generative model from amongst the generative models1010-1012, where the selected generative model is configured to answer complex questions and thus consumes a relatively large amount of computing resources. In a different example, the classifier1008receives the input “what is the weather in Seattle today” and assigns a class label to such input that indicates that the input is relatively simple. The computing system1000selects a generative model from amongst the generative models1010-1012and provides the input to the selected generative model, where the selected generative model uses a relatively small amount of computing resources to generate output based upon the input. Thus, computing resources are preserved, as the more complex model is reserved for relatively complex inputs, and the less complex model is used for less complex inputs.

In another example, the classifier is configured to assign a class label to an input received from the client computing device1002that identifies a topic related to the input. The computing system1000selects a generative model from amongst the generative models1010-1012that is trained with data pertaining to such topic based upon the class label assigned to the input. For instance, the computing system1000receives the input “how many home runs did Babe Ruth hit before he turned 30” and the classifier1008assigns a class label of “sports” to such input. The computing system1000selects a generative model from amongst the generative models1010-1012that has been trained on content that is associated with the topic of “sports”, and the input is provided to such generative model. A generative model trained with respect to a single topic may require fewer computing resources to generate an output when provided with an input that pertains to the topic than a non-topic-specific generative model when provided with the same input. Further, training a topic-specific generative model can be completed more quickly when compared to training a more robust generative model, as less training data is used. It is appreciated that the training data can include webpages indexed by a search engine, as the search engine typically assigns topics to such webpages.

In yet another example, the classifier1008assigns a label to input received from the client computing device1002based upon size of such input. As indicated previously, the first generative model1010may be configured to receive a prompt of a first size, while the nth generative model1012may be configured to receive a prompt of a second size (that is larger than the first size). Further, the prompt can include content of a webpage being viewed by a user of the client computing device1002, and therefore the input referenced above can include content of the webpage. The classifier1008can identify a size of the content (e.g., a number of tokens in the content). When the classifier1008determines that the number of tokens in the content of the webpage is less than a threshold, the first generative model1010can be selected and can be provided with the prompt. When, however, the number of tokens in the content of the webpage is greater than or equal to the threshold, the nth generative model1012can be selected and provided with the prompt. As the nth generative model1012receives the larger prompt and generates output based upon the larger prompt, the nth generative model102may consume more computing resources than the first generative model1010when generating output. Hence, through use of the classifier1008, computing resources can be managed. Moreover, combinations of the approaches described herein can be combined—for instance, a generative model can be selected based upon a complexity label assigned to input and a topic label assigned to the input. In another example, a generative model can be selected based upon a topic label assigned to input and a size of webpage content. Other combinations are contemplated.

In yet another example, the classifier1008is configured to assign class labels to outputs of the generative models1010-1012, and an output generated by a generative model can be returned to the client computing device1002based upon the class label assigned to such output. For instance, the first generative model1010consumes a first amount of computing resources when generating an output while the nth generative model consumes a second amount of computing resources when generating the output, where the first amount of computing resources is less than the second amount of computing resources. Accordingly, when an input is received by the computing system1000, by default the computing system1000can provide the input to the first generative model1010. The first generative model1010generates an output based upon the input, and the classifier1008receives the output. The classifier assigns a class label to the output (e.g., based upon the output and optionally the input), where the class label indicates whether the output is acceptable or not acceptable. When the class label indicates that the output generated by the first generative model1010is acceptable, the computing system1000transmits the output the client computing device1002for presentment to the user thereof. When, however, the class label output by the classifier1008indicates that the output is not acceptable, the computing system1000provides the input to the nth generative model1012, and the nth generative model generates a second output based upon the input. The computing system1000then provides the second output to the client computing device1002.

In some embodiments, the input can be simultaneously provided to several generative models that utilize relatively small amounts of computing resources when generating outputs based upon the inputs, and when the classifier1008indicates that one of such outputs is acceptable, the computing system1000provides the output to the client computing device1002. When none of the outputs is acceptable, the computing system1000provides the input to the generative model that utilizes a relatively large amount of computing resources to generate output; hence, the generative model that utilizes a relatively large amount of computing resources to generate outputs is provided with input only when one or more generative models that utilize fewer resources to generate outputs are unable to generate an acceptable output for the input.

FIGS.11and12illustrate methodologies relating to generation of output by a generative model based upon contextual information. While the methodologies are shown and described as being a series of acts that are performed in a sequence, it is to be understood and appreciated that the methodologies are not limited by the order of the sequence. For example, some acts can occur in a different order than what is described herein. In addition, an act can occur concurrently with another act. Further, in some instances, not all acts may be required to implement a methodology described herein.

Moreover, the acts described herein may be computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media. The computer-executable instructions can include a routine, a sub-routine, programs, a thread of execution, and/or the like. Still further, results of acts of the methodologies can be stored in a computer-readable medium, displayed on a display device, and/or the like.

Referring solely toFIG.11, a methodology1100for generating, by a generative model, output based content received from a search engine is illustrated. The methodology1100starts at1102, and at1104the generative model receives input set forth by a user to a client computing device that is in network communication with the generative model, where the input is received from the client computing device. At1106, the generative model generates a query based upon the input. For instance, the input may be a relatively long text, and the generative model can identify entities in such text and generate the query based upon the identified entities, where the query is shorter than the original input.

At1108, the query is provided to the search engine. The search engine performs a search based upon the query and identifies content that is relevant to the query. The content can be a portion of a webpage or webpages identified by the search engine as being relevant to the query, an instant answer, an entity card, and so forth. At1110, the generative model receives the content identified by the search engine as being relevant to the query, and at1112the generative model generates an output based upon such content. In addition, the output can include a citation or citations to the content identified by the search engine. The output can be textual (conversational), can be or include an image, can be or include a video, can be audio output, etc. At1114, the output is transmitted to the client computing device for presentment to the user. The methodology1100completes at1116.

Turning now toFIG.12, a methodology1200performed by a generative model is depicted. The methodology1200starts at1202, and at1204data is received from an application executing on a client computing device (such as a web browser, an email application, or the like). The data includes input set forth to the client computing device by the user (where the input is intended for provision to the generative model). The data also includes content being presented on the client computing device by way of the application. For instance, a web browser can be configured to capture content displayed on a webpage and transmit such content to the generative model together with the user input upon the user setting forth the user input and indicating that the content is to be transmitted to the generative model.

At1206, the output generated by the generative model is transmitted to the client computing device for presentment by way of the application. Optionally, while not shown, the generative model generates a query based upon the received input and the content, and transmits the query to a search engine, whereupon the search engine identifies second content that is relevant to the query. The generative model can generate the output based upon the input, the query, the content, and/or the second content. The methodology1200completes at1208.

Referring now toFIG.13, a high-level illustration of an exemplary computing device1300that can be used in accordance with the systems and methodologies disclosed herein is illustrated. For instance, the computing device1300may be used in a system that is configured to provide search results identified by a search engine as a part of a prompt to a GLM. By way of another example, the computing device1300can be used in a system that is configured to provide content of a webpage as part of a prompt to a GLM. The computing device1300includes at least one processor1302that executes instructions that are stored in a memory1304. The instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more components discussed above or instructions for implementing one or more of the methods described above. The processor1302may access the memory1304by way of a system bus1306. In addition to storing executable instructions, the memory1304may also store prompts, images, etc.

The computing device1300additionally includes a data store1308that is accessible by the processor1302by way of the system bus1306. The data store1308may include executable instructions, instant answers, a web index, etc. The computing device1300also includes an input interface1310that allows external devices to communicate with the computing device1300. For instance, the input interface1310may be used to receive instructions from an external computer device, from a user, etc. The computing device1300also includes an output interface1312that interfaces the computing device1300with one or more external devices. For example, the computing device1300may display text, images, etc. by way of the output interface1312.

It is contemplated that the external devices that communicate with the computing device1300via the input interface1310and the output interface1312can be included in an environment that provides substantially any type of user interface with which a user can interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and so forth. For instance, a graphical user interface may accept input from a user employing input device(s) such as a keyboard, mouse, remote control, or the like and provide output on an output device such as a display. Further, a natural user interface may enable a user to interact with the computing device1300in a manner free from constraints imposed by input device such as keyboards, mice, remote controls, and the like. Rather, a natural user interface can rely on speech recognition, touch and stylus recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, and so forth.

Additionally, while illustrated as a single system, it is to be understood that the computing device1300may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device1300.

Various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer-readable storage media. A computer-readable storage media can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproduce data magnetically and discs usually reproduce data optically with lasers. Further, a propagated signal is not included within the scope of computer-readable storage media. Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of communication medium. Combinations of the above should also be included within the scope of computer-readable media.

Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

Described herein are various features pertaining to integration of a computer-implemented search engine and a generative model in accordance with at least the following examples.(A1) In an aspect, a method described herein includes receiving, by a generative model, input set forth by a user of a client computing device that is in network communication with the computing system. The method also includes generating, by the generative model, a query based upon the input set forth by the user. The method additionally includes providing the query to a search engine and receiving, by the generative model and from the search engine, content identified by the search engine based upon the query. The method also includes generating, by the generative model, an output based upon a prompt, where the prompt includes the content identified by the search engine based upon the query. The method further includes transmitting the output to the client computing device for presentment to the user.(A2) In some embodiments of the method of (A1), the prompt further includes the query generated by the generative model.(A3) In some embodiments of the method of at least one of (A1)-(A2), the prompt further includes the input set forth by the user of the client computing device.(A4) In some embodiments of the method of at least one of (A1)-(A3), the output comprises a textual response to the input set forth by the user of the client computing device.(A5) In some embodiments of the method of at least one of (A1)-(A4), the output comprises an image generated by the generative model based upon the prompt.(A6) In some embodiments of the method of at least one of (A1)-(A5), the content identified by the search engine based upon the query is an instant answer identified by the search engine based upon the query.(A7) In some embodiments of the method of at least one of (A1)-(A6), the content identified by the search engine based upon the query comprises a portion of a webpage identified by the search engine as being relevant to the query.(A8) In some embodiments of the method of at least one of (A1)-(A7), the method also includes receiving, from the client computing device, second content extracted from an electronic page being presented by way of a computer-implemented application on the client computing device at a time that the input is received, and further where the prompt includes the second content.(A9) In some embodiments of the method of (A8), the computer-implemented application is a web browser, and further where the second content comprises text of a webpage loaded by the web browser.(A10) In some embodiments of the method of (A8), the computer-implemented application is an email application, and further where the second content comprises text of an email displayed by way of the email application.(A11) In some embodiments of the method of at least one of (A1)-(A10), the method also includes subsequent to transmitting the output to the client computing device, receiving, by the generative model, second input set forth by the user of the client computing device. The method also includes generating, by the generative model, a second query based upon the second input set forth by the user. The method further includes providing the second query to a search engine and receiving, by the generative model and from the search engine, second content identified by the search engine based upon the second query. The method also includes generating, by the generative model, a second output based upon a second prompt. The second prompt includes: 1) the content identified by the search engine based upon the query; and 2) the second content identified by the search engine based upon the second query. The method also includes transmitting the second output to the client computing device for presentment to the user.(A12) In some embodiments of the method of at least one of (A10)-(A11), the prompt further includes the input set forth by the user and the query generated by the generative model.(A13) In some embodiments of the method of at least one of (A10)-(A12), the generative model is a transformer-based model.(B1) In another aspect, a method performed by a processor of a computing system that executes a generative model includes receiving, from a client computing device that is in network communication with the computing system, textual input set forth to the client computing device by a user of the client computing device. The method also includes generating a query based upon the input. The method further includes providing the query to a search engine that is in communication with the generative model. The method additionally includes providing the query to a search engine that is in communication with the generative model. The method further includes receiving a portion of a webpage identified by the search engine as being relevant to the query generated by the generative model. The method also includes generating output based upon a prompt, where the prompt includes the portion of the webpage and the textual input received from the client computing device. The method additionally includes transmitting the output to the client computing device for presentment to the user.(B2) In some embodiments of the method of (B1), the portion of the webpage is received due to the webpage being amongst the three highest ranked webpages by the search engine based upon the query.(B3) In some embodiments of the method of at least one of (B1)-(B2), the method also includes receiving an instant answer identified by the search engine as being relevant to the query, where the prompt includes the instant answer.(B4) In some embodiments of the method of at least one of (B1)-(B3), the method also includes receiving information from an entity card identified by the search engine as being relevant to the query, where the prompt includes the information extracted from the entity card.(B5) In some embodiments of the method of at least one of (B1)-(B4), the output includes a citation to the webpage.(B6) In some embodiments of the method of at least one of (B1)-(B5), the output includes an image generated by the generative model.(C1) In another aspect, a computing system includes a processor and memory, where the memory stores instructions that, when executed by the processor, cause the processor to perform at least one of the methods disclosed herein (e.g., any of the methods of (A1)-(A13) or (B1)-(B6)).(D1) In yet another aspect, a computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to perform at least one of the methods disclosed herein (e.g., any of the methods of (A1)-(A13) or (B1)-(B6)).

What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable modification and alteration of the above devices or methodologies for purposes of describing the aforementioned aspects, but one of ordinary skill in the art can recognize that many further modifications and permutations of various aspects are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.