DATA SOURCE CURATION FOR LARGE LANGUAGE MODEL (LLM) PROMPTS

Data source curation for large language model (LLM, generically multimodal model) prompts enables compliance with privacy requirements, while allowing for personal preferences, when Generated passages relating to a person import information from external data sources. A set of data sources is presented to a user, permitting the user to select which to use or avoid using. In some examples, when a user adds a data source, based on the nature of the data source, the user is provided a prompt to provide consent for the use of the content of that data source, enabling user-directed curation of the data sources used in the LLM generation of a summary about the person. In some examples, the set of data sources presented to a user had been previously curated according to governing policies, to restrict the user from accepting use of a data source that is disfavored for legal or policy reasons.

BACKGROUND

Large language models (LLMs) are used to generate summary passages of various data sets and combinations of data sets. However, when the summary passage pertains to a human data subject, some issues arise, including privacy rules and personal preferences of the data subject regarding which audience sees what information. For example, in a large system that spans multiple legal jurisdictions, the inclusion of certain data sources in the LLM prompt may vary from being legally permissible to impermissible, even while the personal preferences of the data subjects may vary widely and sometimes independently of the legal permissions. Additionally, legal requirements and personal preferences vary over time. These factors render governance of the use of LLM to generate people summaries highly challenging.

SUMMARY

The disclosed examples are described in detail below with reference to the accompanying drawing figures listed below. The following summary is provided to illustrate some examples disclosed herein.

Example solutions for data source curation for multimodal model (e.g., large language model (LLM)) prompts include: presenting, in a user interface (UI), a set of data source identifications and a set of data source selections, each data source selection of the set of data source selections corresponding to a respective data source identification of the set of data source identifications, the set of data source selections indicating whether to use a data source identified by the corresponding data source identification; detecting, through the UI, changes to the set of data source selections; detecting an approval, through the UI, to generate a generated passage using the set of data source selections; persisting the set of data source selections and a record of the approval; based on at least detecting the approval, generating the generated passage using each data source identified as to be used in the set of data source selections; and displaying, in the UI, the generated passage.

DETAILED DESCRIPTION

Data source curation for multimodal model (MM, e.g., large language model (LLM)) prompts enables compliance with privacy requirements, while allowing for personal preferences, when generated passages (e.g., LLM-generated or generically, MM-generated) relating to a person import information from external data sources. A set of data sources is presented to a user, permitting the user to select which to use or avoid using. In some examples, when a user adds a data source, based on the nature of the data source, the user is provided a prompt to provide consent for the use of the content of that data source, enabling user-directed curation of the data sources used in the MM generation of a summary about the person. In some examples, the set of data sources presented to a user had been previously curated according to governing policies, to restrict the user from accepting use of a data source that is disfavored for legal or policy reasons. In some examples, records of user consent are stored during the data source curation process, in order to comply with privacy requirements in some legal jurisdictions.

Aspects of the disclosure enhance data security and privacy, as well as reducing bandwidth for queries-even while permitting MMs to generate data using up-to-date data sources, thereby providing a practical, useful result to solve technical problems in the domain of computing. This is accomplished, at least in part, by permitting user curation of data sources, some of which may be updated after the curation process, so that when the MM generates the passage for a viewer, in some examples, current information is used, but only based upon the permission that had been provided by of a human data subject.

Aspects of the disclosure provide for less memory usage, reduced processing load, and reduced bandwidth and computing resource usage at least by reducing repeated manual changes of profiles, saving or freeing up available network bandwidth and computing resources and thereby improving the functioning of the underlying device. Aspects of the disclosure further enhance compliance with privacy and security requirements for personal user data by automatically generating compliant profiles (people summaries), further reducing bandwidth usage for compliance checks on generated profiles. Updates may be implemented dynamically in response to changes, governance and tenant policies providing restrictions in the options for creating profiles, preventing generation of non-compliant profiles, improving user interaction between a user and the computing device. Aspects of the disclosure provide for tailoring dynamically generated profiles based on audiences (viewer classes) as well as compliance parameters, and displaying the dynamically generated profiles to a user in a user interface for transparency and explainability of the MM-generated data, thereby improving user interaction between the user and the device.

Aspects of the disclosure enable MM generation of personal summaries from disparate data sources to be compliant with privacy and security requirements, even when tenants span multiple legal jurisdictions. Curating data sources, which selects the best or most appropriate for use, enables users, such as the data subjects of personal summaries to control how an artificial intelligence (AI) model, such as an LLM, generates passages that are visible to other viewers. A UI permits a user to see, in real time, how adding or removing a data source changes generated passages. Some examples highlight differences for with/without a particular data source. Examples are described herein for an LLM, but may be generalized to MMs (which include LLMs) and other machine learning (ML) models.

Users are able to select and experiment with different data sources, such as meeting and document activity, communications such as recent emails, profile data, tone and style, and other personal data, such as formality and pronouns, to dynamically update and customize their personal summaries. Some examples permit different selections for different viewing audiences, static or dynamic passages, and contextual summaries versus non-contextual summaries.

Contextual people summaries are generated for and are private to a particular viewer or defined set of viewers. The data sources used may include input data that is shared among the data subject and a defined set of viewers, such as non-public documents, communications and other tenant-proprietary data. Context may be based on shared activities such as collaborations or projects, or on meetings. Some contextual people summaries are generated in real-time, on demand (dynamically), rather than persisted (with the exception of caching for a relatively short time period). In some scenarios, if a data subject is involved with multiple different projects or groups of people, there may be a wide variety of different contexts and thus a large number of differing contextual summaries. In such scenarios, it may be preferable for the data subject to approve data sources by type (e.g., emails, documents), rather than needing to approve every specific contextual summary. The selections (curations) of the data sources by type is persisted, so that the passages may be generated on an as-needed basis.

Non-contextual people summaries are those that are shown to all viewers for whom a contextual summary is not warranted, such as an unknown or generic viewer. Due to practical considerations, such as the general need for contextual summaries to have up-to-date information, non-contextual summaries are more likely to be static than are contextual summaries, although some non-contextual summaries may be dynamic, and some contextual summaries may be static (rather than dynamic). Static summaries may be edited, after LLM generation, prior to persistence.

The various examples will be described in detail with reference to the accompanying drawings. Wherever preferable, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made throughout this disclosure relating to specific examples and implementations are provided solely for illustrative purposes but, unless indicated to the contrary, are not meant to limit all examples.

FIG.1illustrates an example architecture100that advantageously curates data sources for Generated passages, such as people summaries. A set of data sources102includes data sources102a-102d. Data source102aand data source102bare external data sources104that are external to a tenant110, such as possibly pulled in from publicly-available social media websites. Data source102cand data source102dare internal data sources106that are internal to tenant110, such as possibly pulled in from email accounts, document management systems, or other proprietary sources. As used herein, a tenant is a boundary around configuration and data, representing rights to access the internal tenant data.

A data importer retrieves and, if necessary, coverts data from the format of the individual ones of data sources102to what is needed for use within architecture100(e.g., for use by a prompt generator120, which is described below). An administrative configuration tool600generates an LLM prompt template122and configures an editor114that is used by a user (e.g., a data subject230ofFIG.2A) during data source curation. Administrative configuration tool600is shown and described in further detail in relation toFIG.6.

Editor114provides a UI200that is shown and described in further detail in relation toFIGS.2A and2B. UI200displays, among other things, a generated passage116aand other generated passages. Some examples include a highlighter252that highlights changes that occur when a new data source is added or removed. Highlighter252is shown and described in further detail in relation toFIG.2B. Editor114edits and persists data curation options300, which are shown and described in further detail in relation toFIG.3.

A prompt generator120intakes data from selected ones of data sources102(identified within data curation options300) and possibly converted by importer112, along with LLM prompt template122, to generate an LLM prompt, such as LLM prompt400a. LLM prompt400ais provided to LLM124to generate generated passage116a. Another LLM prompt400b(described below) is provided to LLM124to generate generated passage116b. LLM prompt400ais shown and described in further detail in relation toFIG.4; other LLM prompts are similar.

In some examples, generated passage116a, generated passage116b, and other generated passages are generated using another technique, such as using another model relevant to language processing or natural language processing (NLP). Generated passages, such as generated passages116aand116bare passed through a verifier126, which may compare the Generated passages with data from the selected ones of data sources102(possibly converted by importer112) to identify and correct potential AI hallucination.

In this illustrated example, generated passage116ais static, and so is persisted in a storage130, along with a set of data source selections118a(from data curation options300) that was used to generate LLM prompt400a. Generated passage116ais persisted so that it may be retrieved by a viewer window140for a static view142. Set of data source selections118ais persisted so that the user (data subject230) may edit set of data source selections118astarting from the prior saved version.

In this illustrated example, generated passage116bis dynamic, and so is not persisted in a storage130, but instead set of data source selections118b(from data curation options300) that was used to generate LLM prompt400b, is persisted. In some examples, some or all of LLM prompt400bis also persisted in storage130to save time. Storage130represents any storage available to architecture100, and may be distributed, rather than a single storage location. When a viewer attempts to view dynamic generated passage116b, a more current version, generated passage116c, is dynamically generated, using the up-to-date (current) data within the selected ones of data sources102, and generated passage116cis displayed in viewer window140for a dynamic view144. In some examples, generated passage116cis also passed through verifier126.

FIG.2Aillustrates further detail for UI200. Data subject230uses UI200to curate data sources102and save sets of data source selections (e.g., set of data source selections118aand also set of data source selections118b) in data curation options300. Subject overview202includes a name204of data subject230(shown as “Subject_Name”), a photograph206or avatar, and a job title208. A set of window tabs216may include overview, contact, organization, and feed tabs that enable further customization.

InFIG.2A, a preview pane220is currently displaying generated passage116a. A viewer class selection window222permits data subject230to separately select options, including curating data sources102differently for different intended audiences, such as members of the public, subordinates, supervisors, and colleagues within tenant110, but in a different work relationship than supervisors or subordinates. Two viewer classes are shown, a viewer class224a, for generated passage116a, and a viewer class224b, for generated passage116b(which will actually be generated passage116c, at the time of display). Viewer class selection window222is illustrated as a drop-down menu, but other configurations are also possible. In some examples, data subject230is permitted to define a viewer class, such as using UI200, whereas in some examples, viewer classes are defined using administrative configuration tool600. In some examples, preview pane220doubles as an editor for static Generated passages, such as generated passage116a.

An options selection pane214displays a set of data source identifications210that has a data source identifications210a-210d. Data source identification210aidentifies data source102a; data source identification210bidentifies data source102b; data source identification210cidentifies data source102c; and data source identification210didentifies data source102d. Set of data source selections118acorresponds to individual ones of set of data source identifications210, for example, data source selection218aacorresponds to data source identification210aand data source selection218abcorresponds to data source identification210b. Identifying the other correspondences fromFIG.2Ais straightforward.

Set of data source selections118ais illustrated as a set of “Yes/No” radio buttons, but other configurations are also possible. In the illustrated example, based on the settings of data source selections118a, data source102aand data source102cwill be used for generated passage116a, but data source102cand data source102dwill not be used for generated passage116a. Style options212e-212hrepresent various style and tone options, such as formality (e.g., use first and last name or just first name), using specified pronouns, performing automatic language translation, and other style options as provided for by administrative configuration tool600. Set of data source selections118b(not shown) may be set differently, and is used for generated passage116b. Each viewer classes may have its own set of data source selections, to produce Generated passages specific to that viewer class.

FIG.2Bis similar toFIG.2A, although there is a change250to data source selection218ab, from “No” to “Yes”. This adds data source102b. As a result generated passage116a(which is the prior generated passage, prior to change250), becomes new generated passage116d. Based on latency times and workload of architecture100, new generated passage116dmay be displayed automatically upon change250, without requiring further action by data subject230, or data subject230may need to click a “Regenerate” button or other similar trigger button.

Highlighter252notes a difference, between prior generated passage116aand new generated passage116d, with a highlighting254(which may be a background color, bold font or other visual or audio emphasis). The illustrated difference is the addition of a sentence stating. “In her free time, she enjoys go-cart racing and spending time with her two feline companions.” Data source102bmay be a social media website that includes information about the pets cared for by data subject230.

The word “two” is in bold inFIG.2B(although it may not be in UI200), to illustrate the dynamic nature of some Generated passages. If data source102bis used in a dynamic Generated passage, such as generated passage116b, then at the time that generated passage116cis displayed, if data source102bhas information about a different number of cats, this different number will be reflected in generated passage116c. This is illustrated inFIG.5, where generated passage116crefers to “three feline companions.”

In some examples, adding a new data source results in the immediate appearance of a user approval prompt260that is specific to the new data source use. When data subject230approves the use of data source102b, such as by clicking the “Accept” button, an approval261is noted. Approvals may be for specific data sources, sets of data source selections, and/or generated passages. For example, approval261may be specific to data source102bor set of data source selections118a, whereas an approval262may be specific to generated passage116a(possibly after editing). An approval263may be specific to a data source, whereas an approval264may be specific to set of data source selections118b(since generated passage116bis dynamic and may not be shown with its exact same content when viewed by a viewer).

FIG.3shows various exemplary data sets stored within storage130, which represents storage generally, and should not be construed to necessarily be a single monolithic storage solution. Data curation options300includes set of data source identifications210, which may be defined by administrative configuration tool600, as shown and described below in relation toFIG.6. Data curation options300also includes generated passage116a, set of data source selections118a, set of data source selections118b, and LLM prompt400b. LLM prompt400bmay be used in the event that a data source, identified in set of data source selections118bas to be used, is unavailable, preventing the generation of a new LLM prompt for generated passage116c.

Other data310is also used, in some examples. For example, administrative configuration tool600permits specification of governance policies312, such as local rules and laws (e.g., based on the general data protection regulations (GDPR) or others), and tenant policies314that may further restrict use of data sources even beyond governance policies312, or have other impacts on what is required or allowed in UI200.

Personal data316may include pronouns and/or a writing style sample for LLM124to mimic (if specified in a style option). A record361is an approval record of approval261; a record362is an approval record of approval262; a record363is an approval record of approval263; and a record364is an approval record of approval264. Records361-364may be used in the event of a privacy compliance audit of tenant110, to prove that the required consent had been obtained from data subject230for Generated passages produced by architecture100.

FIG.4illustrates an exemplary workflow410involved with generating and an LLM prompt400, which represents either LLM prompt400aor LLM prompt400bin various scenarios. As described in further detail in relation toFIG.6, administrative configuration tool600generates LLM prompt template122. Importer112imports (retrieves and coverts, if necessary) data from selected ones of data sources102. Prompt generator120uses LLM prompt template122, data from importer112, data curation options300, and personal data316to produce LLM prompt400. LLM prompt400has LLM instructions402(e.g., style and tone options, LLM parameters) and data404retrieved from each selected one of data sources102.

LLM prompt400is provided to LLM124, which generates passages at various times. LLM124produces generated passage116afor UI200using set of data source selections118aand produces generated passage116bfor UI200using set of data source selections118b. LLM124produces generated passage116c(and other dynamic generated passages) for viewer window140using set of data source selections118b, but at a later time than generated passage116b, when some of data sources102may have updated information. LLM124does not produce generated passage116a(or other static generated passages) for viewer window140), but instead, generated passage116a(and other static generated passages) are retrieved from storage130.

FIG.5illustrates an exemplary UI500as seen by a viewer530, who views generated passage116aor generated passage116c. Generated passage116cis shown in a display pane520with “three felines” rather than “two felines” because, in this illustrated scenario, data subject230updated data source102bto indicate the arrival of a new cat. UI500may appear similar to UI200, for example, with subject overview202including name204, photograph206(or avatar), job title208, and set of window tabs216.

UI500may appear within an internet browser502of viewer530, which sends a viewing request504to viewer window140. In some examples, viewing request504includes a language506for automatic translation functionality, and a viewer class indication524that permits viewer window140to select from generated passages116aand116c. As described previously, viewer window140either retrieves generated passage116afrom storage130, or retrieves set of data source selections118b(or LLM prompt400b) to produce generated passage116cin real-time (i.e., on demand).

In some examples, generated passage116cis cached in a cache510for some time period, in the event that generated passage116cis to be displayed for another viewer in the same viewer class as viewer530, or viewer530sends another viewing request504fairly quickly. In such a scenario, viewer window140may pull generated passage116cfrom cache510, to save compute resources.

In some examples, prediction logic508makes a prediction that a viewing request504will be incoming in the near future, and so triggers viewer window140to produce generated passage116cin anticipation of receiving viewing request504. In some examples, various trigger conditions may be used to force a refresh of generated passage116cwithin cache510, such as a timer event, an update to one of data sources102(e.g., a new email, meeting, or document edit event), or LLM124may send a trigger.

FIG.6shows an example UI602of administrative configuration tool600that enables an administrative user630to configure parameters for data curation options300, select the particular LLM to use, and prepare LLM prompt template122. A user list selection618permits administrative user630to add users to architecture100, such as data subject230. A viewer class editor604defines viewer classes for viewer class selection window222, such as viewer classes224aand224b. An LLM prompt template editor606enables construction of LLM prompts400aand400b, and a policies selection tool608enables administrative user630to define, define, and/or edit tenant policies314and governance policies312. Different legal jurisdictions may have different privacy and security rules, so governance policies312may have variations based on the locality of data subject230(possibly as indicated by the IP address of a computer being used by data subject230to access UI200). Additionally, different cultural norms may also drive locality variations in tenant policies314.

A data source selection window612enables selection (down-selection) of set of data source identifications210from among a wider set of data source identifications610that are visible to administrative user630. That is, administrative user630is able to select only those data sources in set of data source identifications610that fit within the policies of tenant110to include within set of data source identifications210, from which data subject230may select. In this way, administrative user630sets boundaries by limiting sources and data subject230is able to tailor data curation options300(including selecting data sources and style/tone) within those boundaries. In some examples, UI200shows only data sources within set of data source identifications210. In some examples, UI200also shows at least some data sources within set of data source identifications610that are also not within set of data source identifications210, but greyed out or with some other display manner that prevents their selection by data subject230in UI200.

An LLM selection widow614permits selection of LLM124from among a set of candidate LLMs, and an LLM parameter editor616sets LLM parameters for LLM124. Examples of LLM parameters that may be set using LLM parameter editor616include temperature, frequency penalty, presence penalty, top p, and others.

FIG.7shows a flowchart700illustrating exemplary operations that may be performed by architecture100. In some examples, operations described for flowchart700are performed by computing device1000ofFIG.10. Flowchart700commences with presenting UI602to administrative user630in operation702. Administrative user630uses UI602to select LLM124, set LLM parameters, generate LLM prompt template122, define set of data source identifications210, and define viewer classes, in operation704. Also, in operation704, administrative user630uses UI602to restrict profile creation options (data curation options300) by selecting governance policies312and tenant policies314. That is, UI602receives restrictions on set of data source identifications210and/or set of data source selections118a.

Flowchart700then diverts to flowchart800ofFIG.8, which is described below. Upon returning from completing flowchart800, in some examples, operation706predicts the likelihood of receiving a viewing request from viewer class224b(who will be shown a dynamically-generated LLM passage) and, based on at least meeting a threshold likelihood, operation706uses LLM124to generate (produce) generated passage116c, using each data source identified as to be used in set of data source selections118b. Operation708then caches generated passage116cfor a time period.

Viewing request504is received from viewer530(e.g., from internet browser502of viewer530) in operation710. In some examples, this includes receiving viewer class indication524in operation712. Decision operation714determines whether viewer window140will display a stat or dynamic Generated passage, such as based on viewer class indication524. If static, such as viewer class indication524indicating viewer class224a, operation716displays generated passage116ain UI500, and flowchart700returns to operation706.

If a dynamic view is to be provides, such as viewer class indication524indicating viewer class224b, operation718displays generated passage116cin UI500. In some scenarios, operation718is performed by retrieving generated passage116cfrom cache510in operation720. In some scenarios, operation718is performed by LLM124generating generated passage116cusing each data source identified as to be used in set of data source selections118b, in operation722. In some examples, generated passage116cis then stored in cache510(using operation708). After operation718, flowchart700returns to operation706.

FIG.8shows a flowchart800illustrating exemplary operations that may be performed by architecture100. In some examples, operations described for flowchart800are performed by computing device1000ofFIG.10. Flowchart800commences with preparing UI200for data subject230by retrieving set of data source identifications210and set of data source selections118a(which is associated with viewer class224a), in operation802.

In operation804, UI200presents set of data source identifications210and set of data source selections118ato data subject230(the user). Each data source selection of set of data source selections118acorresponds to a respective data source identification of set of data source identifications210, and set of data source selections118aindicates whether to use a data source identified by the corresponding data source identification. In some examples, UI200also presents viewer class selection window222.

UI200receives changes, including change250, to set of data source selections118ain operation806. In operation808, UI200receives approval261, from data subject230, to generate generated passage116ausing set of data source selections118a. In some examples, operation808includes operation810, which generates user approval prompt260(possibly specific to the new use of a data source) upon receiving change250. Operation812persists set of data source selections118aand record361of approval261.

Operation814uses LLM124to generate generated passage116ausing each data source identified as to be used in set of data source selections118a, based on at least detecting (receiving) approval261in some examples. Operation814is performed using operations816-820. Operation816retrieves data from each data source identified as to be used in set of data source selections118a, which may involve converting the retrieved data. Operation818generates LLM prompt400a, to have LLM instructions402and data404retrieved from each data source identified as to be used in set of data source selections118a. Operation820provides LLM prompt400ato LLM124.

UI200displays generated passage116ain operation822, using operations824and826in some examples. Upon generating new generated passage116d(seeFIG.2B), operation824determines a difference between new generated passage116dand prior generated passage116a, and operation826highlights the difference between new generated passage116dand prior generated passage116ain UI200. In some examples, operation822comprises dynamically updating generated passage116ain response to changes, such as change250(i.e., without requiring data subject230to click an update button).

Data subject230then proceeds to prepare a personal summary for another audience. UI200receives selection of viewer class224bin operation832and retrieves set of data source selections118b(associated with viewer class224b) in operation834. Some examples may change set of data source identifications210permitted for different viewer classes. UI200receives changes to set of data source selections118bin operation836and approval262from data subject230to generate generated passage116busing set of data source selections118b, in operation838. This may be similar to operation808.

Operation840persists record362of approval262, and operation842uses LLM124to generate generated passage116busing each data source identified as to be used in set of data source selections118b(e.g., based on at least detecting approval262) This may be performed similarly to operation814. Operation844displays generated passage116bin UI200, similarly to operation822. In operation846, UI200receives approval264by data subject230for generated passage116b, and operation848persists record364of approval264.

FIG.9shows a flowchart900illustrating exemplary operations that may be performed by architecture100. In some examples, operations described for flowchart900are performed by computing device1000ofFIG.10. Flowchart900commences with operation902, which includes presenting, in a first UI, a first set of data source identifications and a first set of data source selections, each data source selection of the first set of data source selections corresponding to a respective data source identification of the first set of data source identifications, the first set of data source selections indicating whether to use a data source identified by the corresponding data source identification.

Operation904includes detecting, through the first UI, changes to the first set of data source selections. Operation906includes detecting a first approval, through the first UI, to generate a first generated passage using the first set of data source selections. Operation908includes persisting the first set of data source selections and a record of the first approval. Operation910includes, based on at least detecting the first approval, generating the first generated passage using each data source identified as to be used in the first set of data source selections. Operation912includes displaying, in the first UI, the first generated passage.

Additional Examples

An example system comprises: a processor; and a computer-readable medium storing instructions that are operative upon execution by the processor to: present, in a first UI, a first set of data source identifications and a first set of data source selections, each data source selection of the first set of data source selections corresponding to a respective data source identification of the first set of data source identifications, the first set of data source selections indicating whether to use a data source identified by the corresponding data source identification; detect, through the first UI, changes to the first set of data source selections; detect a first approval, through the first UI, to generate a first generated passage using the first set of data source selections; persist the first set of data source selections and a record of the first approval; based on at least detecting the first approval, generate the first generated passage using each data source identified as to be used in the first set of data source selections; and display, in the first UI, the first generated passage.

An example computer-implemented method comprises: presenting, in a first UI, a first set of data source identifications and a first set of data source selections, each data source selection of the first set of data source selections corresponding to a respective data source identification of the first set of data source identifications, the first set of data source selections indicating whether to use a data source identified by the corresponding data source identification; detecting, through the first UI, changes to the first set of data source selections; detecting a first approval, through the first UI, to generate a first generated passage using the first set of data source selections; persisting the first set of data source selections and a record of the first approval; based on at least detecting the first approval, generating the first generated passage using each data source identified as to be used in the first set of data source selections; and displaying, in the first UI, the first generated passage.

One or more example computer storage devices have computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising: presenting, in a first UI, a first set of data source identifications and a first set of data source selections, each data source selection of the first set of data source selections corresponding to a respective data source identification of the first set of data source identifications, the first set of data source selections indicating whether to use a data source identified by the corresponding data source identification; detecting, through the first UI, changes to the first set of data source selections; detecting a first approval, through the first UI, to generate a first generated passage using the first set of data source selections; persisting the first set of data source selections and a record of the first approval; based on at least detecting the first approval, generating the first generated passage using each data source identified as to be used in the first set of data source selections; displaying, in the first UI, the first generated passage; persisting the first generated passage; presenting, in the first UI, a viewer class selection window, wherein the first set of data source selections is associated with a first viewer class and a second set of data source selections is associated with a second viewer class, each data source selection of the second set of data source selections corresponding to a respective data source identification of the first set of data source identifications, the second set of data source selections indicating whether to use a data source identified by the corresponding data source identification; detecting, through the first UI, changes to the second set of data source selections; detecting a second approval, through the first UI, to generate a second generated passage using the second set of data source selections; persisting a record of the second approval; based on at least detecting the second approval, generating, using the MM, the second generated passage using each data source identified as to be used in the second set of data source selections; and displaying, in the first UI, the second generated passage.

Alternatively, or in addition to the other examples described herein, examples include any combination of the following:the MM comprises an LLM;generating the first generated passage comprises retrieving data from each data source identified as to be used in the first set of data source selections;generating the first generated passage comprises generating an MM prompt, the MM prompt including MM instructions and data retrieved from each data source identified as to be used in the first set of data source selections;generating the first generated passage comprises providing the MM prompt to the MM;detecting the first approval to generate the first generated passage using the first set of data source selections comprises, upon detecting a change indicating a new use of a data source, generating a user approval prompt specific to the new use;presenting, in a second UI, at least one input option selected from the list consisting of: an MM prompt template editor, an MM selection widow, an MM parameter editor, and a data source selection widow;the data source selection widow enabling selection of the first set of data source identifications from among a second set of data source identifications;presenting, in the first UI, a viewer class selection window;the first set of data source selections is associated with a first viewer class;a second set of data source selections is associated with a second viewer class;each data source selection of the second set of data source selections corresponds to a respective data source identification of the first set of data source identifications;the second set of data source selections indicates whether to use a data source identified by the corresponding data source identification;detecting, through the first UI, changes to the second set of data source selections;detecting a second approval, through the first UI, to generate a second generated passage using the second set of data source selections;persisting a record of the second approval;based on at least detecting the second approval, generating, using the MM, the second generated passage using each data source identified as to be used in the second set of data source selections;displaying, in the first UI, the second generated passage;persisting the first generated passage;receiving a viewing request from a viewer;receiving, from the viewer, a viewer class indication;based on at least the viewer class indication indicating the first viewer class, displaying, in a third UI, the first generated passage;based on at least the viewer class indication indicating the second viewer class, generating, using the MM, a third generated passage using each data source identified as to be used in the second set of data source selections;based on at least the viewer class indication indicating the second viewer class, displaying, in the third UI, the third generated passage;based on at least predicting a threshold likelihood of receiving a viewing request from a viewer of the second viewer class, generating, using the MM, a third generated passage using each data source identified as to be used in the second set of data source selections;based on at least predicting a threshold likelihood of receiving a viewing request from a viewer of the second viewer class, caching the third generated passage for a time period;based on at least receiving the viewing request from the viewer of the second viewer class, displaying, in the third UI, the third generated passage;upon generating a new generated passage in the first UI, determining a difference between the new generated passage and a prior generated passage;highlighting, in the first UI, the difference between the new generated passage and the prior generated passage;defining a viewer class;retrieving the first set of data source identifications for the user;retrieving the first set of data source selections;detecting a third approval by the user for the first generated passage;persisting a record of the third approval;detecting, through the first UI, selection of the second viewer class;retrieving the second set of data source selections;detecting a fourth approval by the user for the second generated passage;persisting a record of the fourth approval;retrieving data from a data source comprises converting retrieved data;based on at least predicting a threshold likelihood of receiving a viewing request from a viewer of the second viewer class, generating, using the MM, a third generated passage using each data source identified as to be used in the second set of data source selections; andcaching the third generated passage for a time period.

Example Operating Environment

FIG.10is a block diagram of an example computing device1000(e.g., a computer storage device) for implementing aspects disclosed herein, and is designated generally as computing device1000. In some examples, one or more computing devices1000are provided for an on-premises computing solution. In some examples, one or more computing devices1000are provided as a cloud computing solution. In some examples, a combination of on-premises and cloud computing solutions are used. Computing device1000is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the examples disclosed herein, whether used singly or as part of a larger set.

Neither should computing device1000be interpreted as having any dependency or requirement relating to any one or combination of components/modules illustrated. The examples disclosed herein may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program components, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program components including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks, or implement particular abstract data types. The disclosed examples may be practiced in a variety of system configurations, including personal computers, laptops, smart phones, mobile tablets, hand-held devices, consumer electronics, specialty computing devices, etc. The disclosed examples may also be practiced in distributed computing environments when tasks are performed by remote-processing devices that are linked through a communications network.

Computing device1000includes a bus1010that directly or indirectly couples the following devices: computer storage memory1012, one or more processors1014, one or more presentation components1016, input/output (I/O) ports1018, I/O components1020, a power supply1022, and a network component1024. While computing device1000is depicted as a seemingly single device, multiple computing devices1000may work together and share the depicted device resources. For example, memory1012may be distributed across multiple devices, and processor(s)1014may be housed with different devices.

Bus1010represents what may be one or more buses (such as an address bus, data bus, or a combination thereof). Although the various blocks ofFIG.10are shown with lines for the sake of clarity, delineating various components may be accomplished with alternative representations. For example, a presentation component such as a display device is an I/O component in some examples, and some examples of processors have their own memory. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are contemplated within the scope ofFIG.10and the references herein to a “computing device.” Memory1012may take the form of the computer storage media referenced below and operatively provide storage of computer-readable instructions, data structures, program modules and other data for the computing device1000. In some examples, memory1012stores one or more of an operating system, a universal application platform, or other program modules and program data. Memory1012is thus able to store and access data1012aand instructions1012bthat are executable by processor1014and configured to carry out the various operations disclosed herein. Thus, computing device1000comprises a computer storage device having computer-executable instructions1012bstored thereon.

In some examples, memory1012includes computer storage media. Memory1012may include any quantity of memory associated with or accessible by the computing device1000. Memory1012may be internal to the computing device1000(as shown inFIG.10), external to the computing device1000(not shown), or both (not shown). Additionally, or alternatively, the memory1012may be distributed across multiple computing devices1000, for example, in a virtualized environment in which instruction processing is carried out on multiple computing devices1000. For the purposes of this disclosure, “computer storage media,” “computer storage memory,” “memory,” and “memory devices” are synonymous terms for the memory1012, and none of these terms include carrier waves or propagating signaling.

Processor(s)1014may include any quantity of processing units that read data from various entities, such as memory1012or I/O components1020. Specifically, processor(s)1014are programmed to execute computer-executable instructions for implementing aspects of the disclosure. The instructions may be performed by the processor, by multiple processors within the computing device1000, or by a processor external to the client computing device1000. In some examples, the processor(s)1014are programmed to execute instructions such as those illustrated in the flow charts discussed below and depicted in the accompanying drawings. Moreover, in some examples, the processor(s)1014represents an implementation of analog techniques to perform the operations described herein. For example, the operations may be performed by an analog client computing device1000and/or a digital client computing device1000. Presentation component(s)1016present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc. One skilled in the art will understand and appreciate that computer data may be presented in a number of ways, such as visually in a graphical user interface (GUI), audibly through speakers, wirelessly between computing devices1000, across a wired connection, or in other ways. I/O ports1018allow computing device1000to be logically coupled to other devices including I/O components1020, some of which may be built in. Example I/O components1020include, for example but without limitation, a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc.

Computing device1000may operate in a networked environment via the network component1024using logical connections to one or more remote computers. In some examples, the network component1024includes a network interface card and/or computer-executable instructions (e.g., a driver) for operating the network interface card. Communication between the computing device1000and other devices may occur using any protocol or mechanism over any wired or wireless connection. In some examples, network component1024is operable to communicate data over public, private, or hybrid (public and private) using a transfer protocol, between devices wirelessly using short range communication technologies (e.g., near-field communication (NFC), Bluetooth™ branded communications, or the like), or a combination thereof. Network component1024communicates over wireless communication link1026and/or a wired communication link1026ato a remote resource1028(e.g., a cloud resource) across network1030. Various different examples of communication links1026and1026ainclude a wireless connection, a wired connection, and/or a dedicated link, and in some examples, at least a portion is routed through the internet.

The order of execution or performance of the operations in examples of the disclosure illustrated and described herein is not essential, and may be performed in different sequential manners in various examples. For example, it is contemplated that executing or performing a particular operation before, contemporaneously with, or after another operation is within the scope of aspects of the disclosure. When introducing elements of aspects of the disclosure or the examples thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. The term “exemplary” is intended to mean “an example of.” The phrase “one or more of the following: A, B, and C” means “at least one of A and/or at least one of B and/or at least one of C.”