INTEGRATION OF A GENERATIVE MODEL INTO COMPUTER-EXECUTABLE APPLICATIONS

A computing system described herein includes a processor and memory storing instructions that, when executed by the processor, cause the processor to perform several acts. The acts include generating a prompt that is to be input to a generative language model, where the prompt includes content of a webpage being presented to the user. The acts also include providing the prompt as input to the generative language model. The acts further include receiving output from the generative language model, where the generative language model generated the output based upon the prompt. The acts additionally include causing the output to be presented to the user by way of a client computing device concurrently with the webpage being presented to the user.

BACKGROUND

Relatively recently, generative models, such as generative language models (GLMs) (also referred to as large language models (LLMs)) have been developed. An example of a GLM is the Generative Pre-trained Transformer 3 (GPT-3). Another example of a GLM is the BigScience Language Open-science Open-access Multilingual (BLOOM) model, which is also a transformer-based model. Briefly, a generative model is configured to generate an output (such as text in human language, source code, music, video, and the like) based upon a prompt set forth by a user and in near real-time (e.g., within a few seconds of receiving the prompt). The generative model generates content based upon training data over which the generative model has been trained. Accordingly, in response to receiving the prompt “how many home runs did Babe Ruth bit before he turned 30”, a GLM can output “Before he turned 30, Babe Ruth hit 94 home runs.” In another example, in response to receiving the prompt “provide me with a list of famous people born in Seattle and Chicago”, the GLM can output two separate lists of people (one for Seattle and one for Chicago), where the list of people born in Chicago includes Barrack Obama. In both these examples, however, the GLM outputs information that is incorrect-for instance, Babe Ruth hit more than 94 home runs before he turned 30, and Barrack Obama was born in Hawaii (and not Chicago). Accordingly, both conventional search engines and generative models are deficient with respect to identifying and/or generating appropriate information in response to certain types of user input.

In addition, GLMs are not well-suited to provide accurate answers to user input that pertains to recent information. For instance, a GLM, upon receipt of input “what is the current weather in Chicago,” is unable to provide accurate output, as the GLM generates textual output based upon training data which tends to be at least somewhat stale (i.e., it is impractical to retrain a GLM every minute with updated information about weather, sporting events, stock markets, news events, and so forth).

SUMMARY

Various technologies are described herein that relate to integrating a generative model (such as a GLM) with applications, such that the generative model is provided with input from the applications and generates output based upon such input. This integration addresses several deficiencies of generative models referenced above; in an example, a web browser can be configured to have a side panel, where the side panel includes features that can be employed to interact with the generative model. For public pages, the generative model is provided with content of such pages and/or metadata pertaining to such pages, and this information can be used as a portion of a prompt and/or can be used by the generative model to provide suggestions. For example, when a webpage loaded by the web browser is a relatively long news article, the generative model is provided with such news article by way of the web browser and outputs a suggestion, such as “would you like me to summarize this article”, which is presented on the web browser by way of the side panel. Upon receiving an indication that the user desires to have the article summarized, the generative model generates a summary of the article and presents such summary by way of the side panel.

In another example, the web browser may have a webpage loaded therein that includes private information, such as an email page, a social media page, an enterprise page, or the like. In such an example, the web browser can receive an indication that the user wishes to interact with the generative model by way of the side panel. The web browser can ascertain that the page is a “private” page (e.g., based upon the page not being indexed by a search engine, for example), and can present a request for consent to the user; thus, the content of the private page is not provided to the generative model unless explicit user consent is received. Upon receiving such consent, the generative model is provided with access to content of the webpage and can generate output based upon content of the webpage (e.g., where the output is conversational in nature).

In still yet another example, a web browser can have several open tabs, with each tab having a webpage loaded therein. The generative model can be provided with URLs for the webpages loaded in the tabs, times that the webpages were last viewed by the user, amongst other information. The generative can receive input such as “make active the browser tab that I was viewing about thirty minutes ago” and can identify which tab was active in the web browser approximately thirty minutes prior to the input being received. This feature may be particularly beneficial for voice input, allowing for the user to quickly switch between tabs based upon conversational input.

In still yet another example, an operating system is configured to have a pinnable side panel, thereby providing permanent access to a generative model regardless of the application that is being used by a user. Thus, a user can be employing a word processing document that includes content and can relatively quickly copy and paste content from the word processing document to an input field in the side panel; the generative model is provided with such content, and can summarize the content, rewrite the content, generate an image based upon the content, generate video based upon the content, and so forth. Additionally, the operating system can provide the generative model with information about the application that is open (e.g., identity of the application, type of the application, etc.), and the generative model can use such information to generate output (e.g., conversational output, an image, and so forth). Put differently, the information about the application is included in the prompt provided to the generative model, and the generative model generates output based upon the prompt.

DETAILED DESCRIPTION

When reference is made to a webpage herein, such term is intended to cover conventional webpages as well as web applications.

The technologies described herein relate to integrating a generative model with an application (such as a web browser) and/or an operating system (OS). In an example, a client computing device executes a web browser, where the web browser receives user input to retrieve a webpage. The web browser is configured to communicate with a generative model (such as a GLM). For example, upon receipt of an indication that the user desires to interact with the generative model, a side panel is presented by the web browser. The side panel can overlay a portion of the webpage being displayed by the web browser. In another example, content of the webpage is resized to accommodate the screen real estate consumed by the side panel. When the webpage is a public page (e.g., indexed by a search engine), the web browser can cause content of the webpage to be provided to the generative model. In addition, the web browser can cause other information pertaining to the web browser to be provided to the generative model, such as uniform resource locators (URLs) of webpages loaded in tabs of the browser, titles of such webpages, times when the webpages were accessed, and so forth.

In an example, based upon such information, the generative model generates suggestions and causes such suggestions to be presented as selectable “chips”. An example suggestion is “can I summarize this for you?” In another example, the user sets forth input to an input field of the side panel, and such input is provided to the generative model. The generative model generates output based upon the conversational input and the content of the webpage provided to the generative model. Accordingly, the prompt provided to the generative model includes not only the user generated input but also includes content from the webpage, information from webpages loaded in other tabs of the web browser, amongst other information. Accordingly, the output of the generative model is “grounded” in the content of the web page.

In another example, the web browser has loaded a private page, such as a webpage that includes emails of the user, a social media page, and so forth. When the web browser has such webpage loaded therein, and further when the web browser receives an indication that the user wishes to interact with the generative model, a determination is made as to whether the user has provided consent for content of the webpage to be provided to the generative model. When the user has not previously provided consent, a request is provided to the user that the user provide consent for the content of the webpage to be provided to the generative model. When the user fails to provide consent, the content of the webpage is not provided to the generative model. Contrarily, when consent of the user is received, content of the webpage is provided to the generative model such that output generated by the generative model is based upon content of the webpage. The generative model may be requested to assist with generation of text in fields of the webpage, summarizing the text, and so forth. Further, the generative model can reason over content of the webpage (public or private) and provide output. Therefore, if a webpage includes content about statistics of Babe Ruth, and with respect to the conversational input “how many home runs did Babe Ruth hit before turning 30”, the generative model can reason over content of the webpage and generate output based upon content of such web page—e.g., “Babe Ruth hit 284 home runs before turning 30”.

In still yet another example, an operating system of a computing device is configured to present a pinnable side panel on a GUI of the operating system, where a user of the computing device can interact with the generative model by way of the side panel. Therefore, an application can be executed by the computing device, such as a word processing application, a web browser, a slideshow application, etc. The application can be presenting content, and the user can readily copy and paste content displayed by way of the application into an input field in the side panel. Such content can be provided to the generative model, which can summarize the content, rewrite the content, etc., at the request of the user.

Referring now toFIG.1, a functional block diagram of a computing environment100is illustrated. The computing environment100includes a computing system101. While illustrated as a single system, it is to be understood that the computing system101can include several different server computing devices, can be distributed across data centers, etc. The computing system101is configured to facilitate interaction between a user and a generative model by way of a web browser (or other suitable application).

A client computing device102operated by a user (not shown) is in communication with the computing system101by way of a network104. The client computing device102can be any suitable type of client computing device, such as a desktop computer, a laptop computer, a tablet (slate) computing device, a video game system, a virtual reality or augmented reality computing system, a mobile telephone, a smart speaker, or other suitable computing device.

The computing system101includes a processor106and memory108, where the memory108includes instructions that are executed by the processor106. More specifically, the memory108includes a search engine110, a generative model112, and an interface module113that, as will be described in greater detail below, acts as an interface between an application executing on the client computing device102(and/or an operating system of the client computing device102), the search engine110, and the generative model112. Operations of the search engine110, the generative model112, and the interface module113are described in greater detail below. The computing system106also includes data stores114-122, where the data stores114-122store data that is accessed by the search engine110and/or the generative model112. With more particularity, the data stores114-122include a web index data store114, an instant answers data store116, a knowledge graph data store118, a supplemental content data store120, and a dialog history data store122. The web index data store114includes a web index that indexes webpages by keywords included in or associated with the webpages. The instant answers data store116includes an index of instant answers that are indexed by queries, query terms, and/or terms that are semantically similar or equivalent to the queries and/or query terms. For example, the instant answer “2.16 meters” can be indexed by the query “height of Shaquille O'Neal” (and queries that are semantically similar or equivalent, such as “how tall is Shaquille O'Neal”).

The knowledge graph data store118includes a knowledge graph, where a knowledge graph includes data structures about entities (people, places, things, etc.) and their relationships to one another, thereby representing relationships between the entities. The search engine110can use the knowledge graph in connection with presenting entity cards on a search engine results page (SERP). The supplemental content data store120includes supplemental content, such as electronic advertisements, that can be returned by the search engine110based upon a query.

The dialog history data store122includes dialog history, where the dialog history includes dialog information with respect to users and the generative model112. For instance, the dialog history can include, for a user, identities of conversations undertaken between the user and the generative model112, input provided to the generative model112by the user for multiple dialog turns during the conversation, responses in the conversation generated by the generative model112in response to the inputs from the user, queries generated by the generative model112during the conversation that are used by the GLM112to generate responses, and so forth. In addition, the dialog history can include context obtained by the search engine110during conversations; for instance, with respect to a conversation, the dialog history122can include content from SERPs generated based upon queries set forth by the user and/or the generative model112during the conversation, content from webpages identified by the search engine110based upon queries set forth by the user and/or the generative model112during the conversation, and so forth. The data stores114-122are presented to show a representative sample of types of data that are accessible to the search engine110and/or the generative model112; it is to be understood that there are many other sources of data that are accessible to the search engine110and/or the generative model112, such as data stores that include real-time finance information, data stores that include real-time weather information, data stores that include real-time sports information, data stores that include images, data stores that include videos, data stores that include maps, etc. Such sources of information are available to the search engine110and/or the generative model112.

The search engine110includes a web search module124, an instant answer search module125, a knowledge module128, and a supplemental content search module130. The web search module124is configured to search the web index data store114based upon queries received by users, queries generated by the search engine110based upon queries received by users, and/or queries generated by the generative model112based upon interactions of users with the generative model112. Similarly, the instant answer search module126is configured to search the instant answers data store116based upon queries received by users, queries generated by the search engine110based upon queries received by users, and/or queries generated by the generative model112based upon interactions of users with the generative model112. The knowledge module128is configured to search the knowledge graph data store118based upon queries received by users, queries generated by the search engine110based upon queries received by users, and/or queries generated by the generative model112based upon interactions of users with the generative model112. Likewise, the supplemental content search module130is configured to search the supplemental content data store120based upon queries received by users, queries generated by the search engine110based upon queries received by users, and/or queries generated by the generative model112based upon interactions of users with the generative model112.

The search engine110can generate structured, semi-structured, and/or unstructured data that is representative of content identified by at least one of the modules124-130. For instance, the search engine110generates a JSON document that includes information obtained by the search engine110based upon one or more searches performed over the data stores114-120(or other data stores). In an example, the search engine110generates data that is in a structure/format that is suitable for inclusion in a prompt that is provided to the generative model112.

The client computing device102includes a processor132and memory134, where the memory134has a web browser136loaded therein. The web browser136has a client interface module138incorporated therein, where the web browser136is in communication with the interface module113of the computing system101, such that the web browser136can exchange information with the search engine110and the generative model112by way of the client interface module138and the interface module113. Optionally, the web browser136includes a model140, which can be a relatively small generative model that is trained to perform fairly specific functionality that is often repeated at the web browser136, such as text summarization and/or text completion (e.g., summarizing text displayed in a webpage and/or completing a sentence or paragraph based upon a set of words or phrases).

The environment100also includes a web server142that is in communication with the client computing device102by way of the network104. The web server142hosts a website that includes a webpage144. The web browser136is configured to retrieve webpages of the website hosted by the web server142. Accordingly, the web browser136can retrieve the webpage144from the web server142(e.g., based upon user input, based upon programmatic input, etc.).

Because the generative model112is in communication with the web browser136by way of the modules138and113, the generative model112can be provided with any suitable information that can be obtained by the web browser136as part of a prompt (in addition to input set forth by a user of the client computing device102to the generative model112). For example, the webpage144retrieved by the web browser136can include an e-mail sent to the user. Presuming that the user has provided authorization for the generative model112to obtain such e-mail, in response to the web browser136receiving an indication that a conversation between the user and the generative model112is to be initiated, the web browser136can provide information from the webpage144(e.g., the email) to the generative model112by way of the modules138and113. Specifically, at least one of the client interface module138or the interface module113is configured to structure content of the webpage in a manner that can be consumed by the generative model112(e.g., as at least a portion of a prompt). In addition, the web browser136receives input from the user and provides the interface module113with such input, which in turn provides such input to the generative model112. Accordingly, the prompt provided to the generative model112includes not only the input set forth by the user, but additionally includes content of the webpage144being viewed by the user. Therefore, the generative model112can generate output based upon content of the web page144.

Further, the generative model112can generate a query based upon the content of the webpage144and the input set forth by the user. The generative model112provides the query to the interface module113, which in turn provides the query to the search engine110(in a format suitable for use by the search engine110). The search engine110searches at least one of the data stores114-120based upon the query. The search engine110provides the generative112with at least a portion of the search results identified by the search engine110based upon the query (by way of the interface module113), thereby providing additional context for the generative model112to employ when generating output to the user (where the output can be conversational output, an image, a video, etc.). More specifically, the prompt employed by the generative112to generate conversational output can additionally include at least a portion of the search results identified by the search engine110.

Operation of the generative model112is improved due to the generative model112generating output based upon a prompt that includes content from the webpage144(or information about the webpage144determined by the search engine110). In contrast, conventionally, generative models generate output based solely upon user input and dialog history. There are numerous use cases where the generative model112provides functionality that conventional generative models are unable to provide, and several of such use cases are set forth below.

Examples of operation of the computing environment100are now set forth. It is to be understood that these examples are non-limiting, and permutations of such examples are contemplated. Referring toFIG.2, a communications diagram200depicting communications between the web browser136, the web server142, the interface module113, the GLMM112, and the search engine110is illustrated. The communications diagrams presented herein illustrate exemplary methodologies relating to integration of computer-executable applications (such as web browsers, email applications, word processing applications, etc.) and/or operating systems with generative models. While the methodologies are shown and described as being a series of acts that are performed in a sequence, it is to be understood and appreciated that the methodologies are not limited by the order of the sequence. For example, some acts can occur in a different order than what is described herein. In addition, an act can occur concurrently with another act. Further, in some instances, not all acts may be required to implement a methodology described herein.

The search engine110can obtain information about “public” pages, where public pages are those represented in a search engine index. When indexing a webpage, for example, the search engine110can identify topics discussed in the webpage, entities referenced in the web page, sentiment of text in the web page (e.g., positive, negative, neutral), and so forth. At202, public page information is provided by the search engine110to the interface module113. While the search engine110is depicted inFIG.2as providing the public page information prior to the web browser136loading a webpage, it is understood that the interface module113can obtain information for a webpage in response to receiving an indication that the webpage has been loaded by the web browser136.

At204, the web browser136requests the webpage144from the web server142, and at206the web browser136obtains the webpage144from the web server142. At208, the web browser136transmits an information request to the interface module113, where the information request is for the public page information for the webpage144(provided to the interface module113by the search engine110). At210, the interface module113provides such page information to the web browser136. In an example, the page information can be presented to a user by the web browser136upon receipt of a selection by the user of a graphical icon that corresponds to such page information.

The web browser136receives an indication that a user of the web browser136desires to interact with the generative model112. At212, the web browser136transmits an indication to the interface module113that the user has requested to interact with the generative model112. In response to receiving such indication, at214the interface module113provides, for example, content of the webpage, the page information generated by the search engine110, and/or input set forth by the user (if any). The generative model112generates output based upon the input provided to the generative112by the interface module113(where the input is in the in the form of a prompt). In an example, the output includes a query that is constructed to cause the search engine110to obtain additional information. At216, the generative model112provides the output to the interface module113, and at218the interface module113transmits the query to the search engine110. The search engine110performs a search over at least one of the data stores114-120based upon the query and identifies search results, where the search results can include webpages identified by the search engine110as being relevant to the query, an instant answer, a knowledge card, and so forth. At220, the search engine110provides the interface module113with at least some of the search results identified by the search engine110. At222, the interface module113provides the generative model112with additional context, where the additional context includes at least some of the search results identified by the search engine110(formatted in a manner that can be consumed by the generative model112).

The generative model112generates model output based upon the information provided to the generative model112by the interface module113. The model output can be conversational output, output to be included in a suggestion “chip”, an image, and so forth. At224, the generative model112provides the interface module113with the model output, and at226the interface module113transmits the model output to the browser136. The model output can be presented by the browser136concurrently with the webpage144.

The communications diagram200illustrates that functionality of the generative model112is integrated into the web browser136, and output of the generative model112is based upon content of at least one webpage loaded by the web browser136and further optionally based upon information provided by the search engine110. In an example, the web browser136is configured to have a side panel where the model output is displayed, and further where the user can interact with the generative model112by way of an input field in the side panel.

Referring now toFIG.3, a schematic that depicts a GUI300of the web browser136is shown. In the example depicted inFIG.3, the web browser136has retrieved a webpage144that includes textual content302about the acid tryptophan. The webpage144is public, and accordingly the search engine110can provide the interface module113with information about such webpage144, including topics discussed in the webpage144, entities recognized in the webpage144, and so forth. The user of the web browser136can set forth an indication that the user desires to interact with the generative model112by way of the web browser136. In response to receipt of such indication, the web browser136can cause a side panel304to be presented concurrently with the webpage144. In an example, the side panel304is presented after receiving input from the user. In another example, the side panel304is presented by default.

The generative model112is provided with at least some of the textual content302, the information about the web page144provided by the search engine110, and/or user input. In this example, the user input is “tell me about tryptophan”. The generative model112generates model output based upon such input as well as the content of the webpage144and/or the information about the webpage144provided by the search engine110.

FIG.4depicts another GUI400corresponding to the webpage144displayed by the web browser136. In this example GUI400, without receiving an explicit user prompt, the generative model112generates a summary of the textual content302included in the webpage144. Accordingly, the user can fairly quickly read the summary presented by the generative model112in the side panel304before deciding whether to read an entirety of the textual content302. The generative model112may be configured to automatically provide other information, such as a description of an image included in the webpage144, a recommendation about filling out a form on the webpage144, and so forth.

Turning toFIG.5, another communications diagram500that depicts flow of communications between the web browser136, the web server142, the interface module113, and the generative model112is shown. At502, the web browser136issues a request for the webpage144from the web server142. In this example, the webpage144is a private page, such as a page corresponding to an e-mail account of a user, a page corresponding to a social media account of a user, or other page that is not indexed by the search engine110. At504, the web browser136obtains the webpage144from the web server142. The browser136receives an indication that the user desires to interact with the generative model112. At506, the web browser136transmits an indication that the user desires to interact with the generative model112to the interface module113. The interface module113ascertains that the webpage144is a private webpage and determines whether the user has consented to providing the generative model112with content of the private webpage144. When it is determined that the user has not provided such consent, the interface module113transmits a consent request to the web browser136, where, for example, graphical data that indicates that consent of the user is requested is presented at the browser136.

The user can consent to content of the webpage144being shared with the generative model112, and at510the browser136transmits such consent to the interface module113. The interface module113can store a record of the consent so that the user need not be requested to consent each time that the browser136retrieves the webpage144. In response to receiving the consent, the interface module113can request information from the web browser136. Such information can include content of the webpage144. At514, the web browser136transmits information to the interface module113, where such information can include content of the webpage144. At516, optionally, when the user sets forth user input to be provided to the generative model112(such as textual or voice input), the browser136can provide such user input to the interface module113.

The interface module113constructs a prompt at518based upon the information received by the interface module113at514and516. The generative model112generates model output based upon the prompt received at518. The model output can be conversational output, a summary of content shown on the webpage144, and so forth. The generative model112transmits the model output to the interface module113at520, and at522the interface module113transmits the model output to the web browser136, whereupon the web browser136presents the model output together with content of the webpage144.

It is noted that the search engine110is not represented in the communications diagram500. In an example, however, the generative model112can generate a query based upon the prompt received at518and can cause such query to be provided to the search engine110(e.g., by way of the interface module113). Accordingly, the prompt used by the generative model112to generate the model output can include information identified by the search engine110. Moreover, as indicated previously, at least a portion of the generative model112may be included in the web browser136.

With reference now toFIG.6, a GUI600of a webpage that depicts an email inbox of a user is presented. The GUI600can include several selectable buttons602-610that correspond to functionality associated with the e-mail inbox, such as a button that is associated with sending an email, a button that is associated with refreshing the inbox, and so forth. The GUI600may include a pane612that depicts a list of emails in the inbox of a user. The GUI600can further include a window614that can include content of an email being read by the user. A side panel816is presented by way of which the user can set forth input to the generative model112(e.g., as an overlay on the webpage represented inFIG.6or beside the webpage).

In the example shown inFIG.6, the email in the window614includes information about holidays for the user over the upcoming calendar year. The user can set forth input by way of the side panel616, and the web browser136transmits the input to the generative model112by way of the interface module113. In addition, when user consent has been received, the web browser136can provide information from the email shown in the window614to the generative model112by way of the interface module113. Therefore, the prompt used by the generative model112to generate conversational output includes the content of the email. As illustrated, the user has set forth the input “what days off do I have in May?” As the generative model112is provided with both the input and the content of the email shown in the window614, the generative model112can generate output that accurately addresses the input (identifying the holidays of the user in the month of May). Conventionally, a generative model is unable to appropriately respond to such input, as the generative model does not have access to the information requested by the user.

With reference now toFIG.7, another GUI700of the webpage144that depicts the e-mail inbox of the user is presented. The webpage includes a text entry field702by way of which the user is setting forth textual content. The generative model112can be provided with such textual content by way of the interface module113and can also be provided with an indication that the user is setting forth such content by way of a text entry field. The generative model112can output a suggestion to the user based upon such content, such as “can I help you complete the e-mail”. Upon the user indicating that the assistance of the generative model112is desired in connection with completing the e-mail, the generative model112can provide a suggested completion that is editable by the user. For instance, the user can copy and paste the suggested completion into the text entry field702and edit such text to the liking of the user.

While various examples pertaining to the generative model112generating model output based upon content of webpages being presented to users have been set forth, it is noted that the generative model112can generate model output based upon other information accessible to the web browser136. For example, the generative model112can be provided with queries that pertain to a search history of the user (e.g., to find a particular webpage in the browsing history of the user). In another example, the generative model112is provided with information pertaining to some most recent threshold number of tabs or pages in a current tab as context to use when generating model output. When the generative model112is provided with input and cannot effectively generate a suitable answer, the generative model112can generate queries to provide to the search engine110in connection with obtaining additional context to be used when generating model outputs. Further, the generative model112can be configured to interact with the settings of the web browser136, such that the generative model112can receive input pertaining to settings of the web browser, favorites stored in the web browser, amongst other information, and can update the browser settings based upon such input. Further, as indicated previously, the generative model112can receive user input with respect to images or videos presented in webpages, such as “who painted that picture?” or “find the best price for the yellow one”, etc. In still yet another example, the generative model112, with respect to a webpage, receives the conversational input “search this website for more like this”, and the generative model112generates a query for provision to the search engine110in connection with identifying similar webpages that belong to the website. According to still yet another example, the generative model112receives the conversational input “search for information similar to what is in this paragraph”, which can result in provision of an appropriate query to the search engine110.

In still more specific examples, the generative model112, with respect to a webpage, is provided with the HTML of the webpage and/or a rendered image of the webpage. The generative model112can be provided with the main page body as clean text. In another example, the generative model112is provided with information selected by the user (e.g., when the user highlights a portion of a webpage). In another example, entity extraction is undertaken on the page and the generative model112is provided with named entities. In connection with performing such tasks, the HTML (or image) can be converted to text and/or other models can be applied, such as object character recognition, object classifiers, image embedding models, and the like. The generative model112can be provided with user dialogue history and page content to generate model output. For large documents, techniques to summarize parts of the document to form a document index or techniques to send one or more chunks/snippets of the document into a limited size buffer of the generative model112can be utilized. The generative model112can output links to other webpages, and such links can cause the web browser136to load new pages when selected. User dialogue may be used to navigate within the page or to other pages or may be used to open pages in new tabs or in a current tab. Answers to questions or passages found for a search/find query can be highlighted in the main document pane, while the side panel may also have access to the rendered image of the main pages to allow conversation, search/find, question and answer over non text elements such as images or figures. Recent history of all main page content across multiple browser tabs can be accessible to the generative model112to allow very richly grounded conversations over time. Reduced representations may be stored to save memory/bandwidth and control prompt size. History can be compressed further by models that embed the data into vector representations. More contextual weight can be given to main page content that is currently scrolled into view and or where the user has spent more time viewing.

Referring now toFIG.8, a functional block diagram of a computing environment800is depicted. The computing environment800includes the computing system101and the client computing device102. The memory134of the client computing device102includes an operating system802. The memory134further includes several applications804that are executed in conjunction with the operating system802. Applications include the web browser136, a word processing application, or any other suitable application. The operating system802further includes the client interface module138. Optionally, the operating system802can also include at least a portion of a generative model.

The computing environment800can operate in a manner similar to the computing environment100depicted inFIG.1. Instead of the client interface module138being included in the web browser136, the client interface module138is included in the operating system802. Accordingly, a pinnable side panel can be displayed in an operating system GUI, such that the user is provided with access to the generative model112regardless of the application being employed by the user. Further, the client interface module138can obtain contextual information pertaining to use of the client computing device102as tracked by the operating system802, such as an identity of a current application being employed by the user, a length of time that the application has been active, and so forth. The interface module113of the computing system101obtains input set forth by the user by way of the side panel with respect to content presented in an application being employed by the user, and the interface module113constructs a prompt for provision to the generative model112based upon such input. The generative model112generates model output based upon the prompt and provides the model output to the operating system802of the client computing device102by way of the interface module113. Additionally, as described above, the generative model112can generate queries based upon input received from the interface module113and can cause such queries to be presented to the search engine110, which conducts searches and provides at least a portion of search results identified based upon the searches to the generative model112. The generative model112can generate output based upon these identified search results.

Now referring toFIG.9, a communications diagram illustrating a flow of communications between the operating system802, the interface module113, the generative model112, and the search engine110is presented. The user of the client computing device102sets forth an indication that the user intends to interact with the generative model112, and at902the operating system802transmits an indication to the interface module113that the user intends to interact with the generative model112. At904, the operating system802transmits context that is to be provided to the generative model112to the interface module113. The context can include an identity of an application that is active, content of such application (presuming that the operating system is able to obtain such content), etc. The context can also include user provided information, such as conversational input, content from the application that has been selected by the user for presentment to the generative model112(e.g., copied and pasted by the user), and so forth. At906, the interface module113provides the context to the generative model112(formatted in a manner suitable as a prompt that can be provided as input to the generative model112). The generative model112generates output based upon the context received at906. Optionally, the output includes a query generated by the generative model112for provision to the search engine110. At908, the generative model112provides the output to the interface module113, and when the output includes a query, the interface module113provides the query to the search engine110at910. The search engine110conducts a search based upon the query and identifies search results. At912, the search engine110provides at least some of the search results to the interface module113, and at914the interface module113provides additional context to the generative model112. The additional context can include at least some of the search results identified by the search engine110formatted in a manner suitable for consumption by the generative model112. The generative model112generates model output and at916the generative model112provides the model output to the interface module113. At918, the interface module113transmits the model output to the operating system802for presentment to the user.

With reference now toFIG.10, a GUI1000of the operating system802is presented. The GUI1000has a task bar1002, where the taskbar1002includes an input field1004by way of which a user can set forth input. The GUI1000further includes a display area1006that can display GUIs of applications being executed by the client computing device102, selectable icons corresponding to applications, and so forth. The GUI1000also includes a side panel1008that can be pinned in the GUI1000(can be opened and closed by the user). The user can interact with the generative model112by way of an input field1010in the side panel1008. In an example, the client computing device102can be executing a word processing application and content of the word processing application can be displayed (text). The user can copy and paste text from the application into the input field1010and can cause such text to be provided to the generative model112. This text can be used as at least a portion of a prompt; for instance, the user requests that the text be summarized, that the text be analyzed for presentment to a specific audience, and so forth.

Referring now toFIG.11, a high-level illustration of a computing device1100that can be used in accordance with the systems and methodologies disclosed herein is illustrated. For instance, the computing device1100may be used in a system that is configured to provide content displayed in a web browser as at least a portion of a prompt to a generative model. By way of another example, the computing device1100can be used in a system that is configured to facilitate user interaction with a generative model by way of an operating system. The computing device1100includes at least one processor1102that executes instructions that are stored in a memory1104. The instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more components discussed above or instructions for implementing one or more of the methods described above. The processor1102may access the memory1104by way of a system bus1106. In addition to storing executable instructions, the memory1104may1104may also store prompts, images, etc.

The computing device1100additionally includes a data store1108that is accessible by the processor1102by way of the system bus1106. The data store1108may include executable instructions, instant answers, a web index, etc. The computing device1100also includes an input interface1110that allows external devices to communicate with the computing device1100. For instance, the input interface1110may be used to receive instructions from an external computer device, from a user, etc. The computing device1100also includes an output interface1112that interfaces the computing device1100with one or more external devices. For example, the computing device1100may display text, images, etc. by way of the output interface1112.

Various features pertaining to integration of a generative language model into a computer-executable application are described herein in accordance with at least the following examples.

(A1) In an aspect, a method performed by a computing system includes generating a prompt that is to be input to a generative model, where the prompt includes content of a webpage being presented to the user by way of a web browser. The method also includes providing the prompt as input to the generative model. The method additionally includes receiving output from the generative model, where the generative model generated the output based upon the prompt. The method further includes causing the output to be presented to the user by way of a client computing device concurrently with the webpage being presented to the user.

(A2) In some embodiments of the method of (A1), the output is displayed in a side panel of the web browser concurrently with the web browser displaying the content of the webpage.

(A3) In some embodiments of the method of at least one of (A1)-(A2), the output is a summary of the content of the webpage.

(A4) In some embodiments of the method of (A3), the summary of the webpage is automatically generated by the generative model in response to the web browser loading the webpage.

(A5) In some embodiments of the method of at least one of (A1)-(A4), the method also includes, prior to generating the prompt, receiving input from the user, where the input from the user is set forth by way of an input field in a sidebar displayed in the web browser, wherein generating the prompt comprises including the input from the user in the prompt.

(A6) In some embodiments of the method of (A5), the method also includes providing the input from the user to the generative model, where the generative model generates a query based upon the input from the user. The method further includes providing the query generated by the generative model to a search engine, where the search engine identifies a search result based upon the query, where the prompt includes at least a portion of the search result identified by the search engine.

(A7) In some embodiments of the method of at least one of (A1)-(A6), the method also includes, prior to generating the prompt: 1) identifying that the webpage is not represented in a search engine index; and 2) upon identifying that the webpage is not represented in the search engine index, causing a request for consent to be presented concurrently on the webpage with the content, where the request for consent is configured to receive user consent to provide the content on the webpage to the generative model, where the prompt is generated subsequent to receiving the user consent.

(B1) In another aspect, a computing system includes a processor and memory, where the memory stores instructions that, when executed by the processor, cause the processor to perform at least one of the methods disclosed herein (e.g., any of the methods of (A1)-(A7)).

(C1) In yet another aspect, a computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to perform at least one of the methods disclosed herein.