Generative graphical explanations using large language models in AI-based services

Methods, systems, and computer-readable storage media for receiving a query from a digital assistant of an enterprise system, retrieving data that is responsive to the query from a data management system, inputting a first few-shot prompt to a LLM, and determining, in response to the first few-shot prompt, that a graphical representation of the data is to be generated, and in response: inputting a second few-shot prompt and a third few-shot prompt to the LLM, receiving code from the LLM responsive to the third few-shot prompt, and executing the code to render the graphical representation with the digital assistant.

BACKGROUND

Enterprises continuously seek to improve and gain efficiencies in their operations. To this end, enterprises employ software systems to support execution of operations. Recently, enterprises have embarked on the journey of so-called intelligent enterprise, which includes automating tasks executed in support of enterprise operations using machine learning (ML) systems. For example, one or more ML models are each trained to perform some task based on training data. Trained ML models are deployed, each receiving input (e.g., a computer-readable document) and providing output (e.g., classification of the computer-readable document) in execution of a task (e.g., document classification task). ML systems can be used in a variety of problem spaces. For example, ML systems can be used for data analytics to process data representative of enterprise operations and present insights to users.

SUMMARY

Implementations of the present disclosure are directed to a digital assistant system that leverages large language models (LLMs) for the presentation and delivery of explanations in enterprise systems including graphical representations of data. More particularly, implementations of the present disclosure are directed to a digital assistant system that enables LLM-generated text to be generated based on enterprise data and presented to users within graphical user interface (GUIs) within digital assistants. In this manner, user-friendly and intuitive explanations of the data can be provided in hand with graphical representations of the data.

In some implementations, actions include receiving a query from a digital assistant of an enterprise system, retrieving data that is responsive to the query from a data management system, inputting a first few-shot prompt to a LLM, and determining, in response to the first few-shot prompt, that a graphical representation of the data is to be generated, and in response: inputting a second few-shot prompt and a third few-shot prompt to the LLM, receiving code from the LLM responsive to the third few-shot prompt, and executing the code to render the graphical representation with the digital assistant. Other implementations of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other implementations can each optionally include one or more of the following features: the graphical representation is rendered within a popover container; the digital assistant communicates with the popover container using remote procedure calls (RPCs) to execute the code and render the graphical representation; actions further include displaying explanatory text within the digital assistant, the explanatory text being provided from the LLM in response to the second few-shot prompt and providing an explanation for the data and is responsive to the query; determining, in response to the first few-shot prompt, that a graphical representation of the data is to be generated includes generating a response by the LLM responsive to the first few-shot prompt, the response indicating that a graphical representation of the data is to be generated; the digital assistant is provided in an application; and the data management service is agnostic to multiple applications executing digital assistants.

DETAILED DESCRIPTION

Implementations of the present disclosure are directed to a digital assistant system that leverages large language models (LLMs) for the presentation and delivery of explanations in enterprise systems including graphical representations of data. More particularly, implementations of the present disclosure are directed to a digital assistant system that enables LLM-generated text to be generated based on enterprise data and presented to users within graphical user interface (GUIs) within digital assistants. In this manner, user-friendly and intuitive explanations of the data can be provided in hand with graphical representations of the data.

Implementations can include actions of receiving a query from a digital assistant of an enterprise system, retrieving data that is responsive to the query from a data management system, inputting a first few-shot prompt to a LLM, and determining, in response to the first few-shot prompt, that a graphical representation of the data is to be generated, and in response: inputting a second few-shot prompt and a third few-shot prompt to the LLM, receiving code from the LLM responsive to the third few-shot prompt, and executing the code to render the graphical representation with the digital assistant.

To provide context for implementations of the present disclosure, and as introduced above, enterprises continuously seek to improve and gain efficiencies in their operations. To this end, enterprises employ software systems to support execution of operations. Recently, enterprises have embarked on the journey of so-called intelligent enterprise, which includes automating tasks executed in support of enterprise operations using ML systems. For example, one or more ML models are each trained to perform some task based on training data. Trained ML models are deployed, each receiving input (e.g., a computer-readable document) and providing output (e.g., classification of the computer-readable document) in execution of a task (e.g., document classification task). For example, ML systems can be used for data analytics to process data representative of enterprise operations and present insights to users.

In enterprise systems, data representative of enterprise operations and/or actions, and the like needs to be communicated to users in a manner that is not only easy to understand and user-friendly, but time- and resource-efficient. To this end, GUIs can be used to graphically represent data. For data science and ML solutions, the importance of accurate and efficient conveyance of data is compounded, because the data and information generated from ML systems are mathematical and technical. As such, the data is already difficult to understand and is not readily presentable in a user-friendly manner.

In the field of artificial intelligence (AI), LLMs have gained increasing popularity. For example, LLMs can be used to generate user-friendly, textual explanations across a variety of topics, which can be presented in digital assistants (e.g., chatbots). However, functionality of LLMs is limited. For example, LLMs are not able to generate graphical representations of data (e.g., graphs, plots, charts) in the context of digital assistants.

In view of the above context, implementations of the present disclosure provide a digital assistant system that leverages LLMs for the presentation and delivery of explanations in enterprise systems including graphical representations of data. As described in further detail herein, the digital assistant system of the present disclosure enables LLM-generated text to be integrated and presented in GUIs within a digital assistant to provide intuitive and user-friendly explanations for users.

FIG.1depicts an example architecture100in accordance with implementations of the present disclosure. In the depicted example, the example architecture100includes a client device102, a network106, and a server system104. The server system104includes one or more server devices and databases108(e.g., processors, memory). In the depicted example, a user112interacts with the client device102.

In some examples, the client device102can communicate with the server system104over the network106. In some examples, the client device102includes any appropriate type of computing device such as a desktop computer, a laptop computer, a handheld computer, a tablet computer, a personal digital assistant (PDA), a cellular telephone, a network appliance, a camera, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or an appropriate combination of any two or more of these devices or other data processing devices. In some implementations, the network106can include a large computer network, such as a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, a telephone network (e.g., PSTN) or an appropriate combination thereof connecting any number of communication devices, mobile computing devices, fixed computing devices and server systems.

In some implementations, the server system104includes at least one server and at least one data store. In the example ofFIG.1, the server system104is intended to represent various forms of servers including, but not limited to a web server, an application server, a proxy server, a network server, and/or a server pool. In general, server systems accept requests for application services and provides such services to any number of client devices (e.g., the client device102over the network106).

In accordance with implementations of the present disclosure, and as noted above, the server system104can host a digital assistant system that leverages one or more LLMs to generate text that is integrated and presented in GUIs with a digital assistant to provide intuitive and user-friendly explanations for users.

As described in further detail herein, the digital assistant system of the present disclosure provides an integrated UI rendering solution within a digital assistant that leverages a LLM to generate explanatory text in view of and that is consistent with data that is displayed as a graphical representation (e.g., chart, graph). More particularly, the digital assistant system of the present disclosure leverages the ability of the LLM to generate code and an understanding of the data to automatically create a UI solution based on a user question and context.

With non-limiting reference to an example use case, a ML system can train multiple ML models based on historical data (training data). For example, and without limitation, a ML model can be tasked with matching entities between multiple computer-readable documents (e.g., line-item matching). Each of the multiple ML models has its own metrics indicating respective performance of the ML model (e.g., accuracy, precision, recall). A user may want to understand which ML model is better in performance and why. To evaluate this, the user can query a metrics comparison of ML models, the metrics comparison being represented in data. The user query can be input to a digital assistant and can include a query requesting a comparison of ML models. In response to the query, the digital assistant system of the present disclosure receives text and UI code that are generated by a LLM. The UI code is executed to render a graphical representation of the data and the text is displayed, which provides a textual explanation for the data.

FIG.2depicts an example conceptual architecture200in accordance with implementations of the present disclosure. In the depicted example, the conceptual architecture200includes an application202, a data management service204, a digital assistant service206, a prompt service208, an LLM system210, and a datastore212. In some examples, one or more of the application202, the data management service204, the digital assistant service206, the prompt service208, the LLM system210, and the datastore212are hosted in a cloud-based environment (e.g., the server system104ofFIG.1). In some examples, the application202executes functionality to support interactions with a user218, which can include, but are not limited to conveying data to the user218.

In accordance with implementations of the present disclosure, the application202includes a digital assistant220that enables interactions with the user218. For example, and as described in further detail herein, the user218can query the digital assistant220to receive query responses from the digital assistant. In some examples, one or more query responses can include data that is presented as a graphical representation in a GUI. In accordance with implementations of the present disclosure, the digital assistant220can present data as a graphical representation in a popover container222within a window224therein. In some examples, the popover container222is provided as an iframe-based container and the digital assistant220communicates with the popover container222using remote procedure calls (RPCs).

As described in further detail herein, the user218can input a query230to the digital assistant220and the digital assistant220can receive a response232to the query230. In accordance with implementations of the present disclosure, the response232can include a textual explanation and code234. In some examples, the textual explanation includes text that is generated by a LLM in view of the code234and is displayed in a UI of the digital assistant. In some examples, a graphical representation is generated by execution of the code234and is depicted in the window224of the popover container222. In some examples, the graphical representation can be provided as a web-based rendering using a web rendering runtime that is built into the popover container222(e.g., iframe). In some examples, the graphical representation is compatible with a UI framework of the popover container222. An example UI framework includes, without limitation, SAPUI5 provided by SAP SE of Walldorf, Germany.

In further detail, in response to the query230, the digital assistant sends a request to the digital assistant service206. In the example ofFIG.2, the digital assistant service206includes an application plug-in240that enables communication between the digital assistant220and other components of the conceptual architecture200, as described herein.

In some examples, the application plug-in240functions as an endpoint of the digital assistant service206for the (specific) application202. For example, from the digital assistant220(e.g., provided as a client) in the application202, the query230is sent to the application plug-in240(specific to an application202). In the application plug-in240, the connections for the application202to the data management service204is integrated. In some examples, in response to the query230, the digital assistant service206requests data that is relevant to the query from the data management service204. The data management service204retrieves the data from the datastore212and provides the data to the digital assistant service206. In some examples, the data management service204is generic. That is, the data management service204is not specific to the application202. This ensures that any data sources that the application202might query are centrally managed.

For example, and with continued reference to the non-limiting example use case, the datastore212can store metrics associated with ML models, example metrics including a set of accuracy values and, for each accuracy value a number of matches not proposed, a number of matches incorrectly proposed, and a number of matches correctly proposed). In some examples, the request from the digital assistant service206to the data management service204includes an identifier that uniquely identifies the ML model, for which metrics are requested, and the data management service204retrieves the metrics for the particular ML model from the datastore212. The metrics can be returned to the digital assistant service206.

In some examples, the digital assistant service206provides text of the query230and the data (e.g., the metrics) as input to the prompt service208, which generates a first few-shot prompt using a prompt builder242. In accordance with implementations of the present disclosure, the first few-shot prompt provides context for a request for the LLM to perform a task. Here, providing context in a few-shot prompt can be referred to as few-shot learning. In natural language processing (NLP), few-shot learning (also referred to as in-context learning and/or few-shot prompting) is a prompting technique that enables a LLM to process examples before attempting a task. In the context of the present disclosure, the task includes providing explanatory text and, in some instances, UI code.

In some examples, the first few-shot prompt is generated using a prompt template. An example prompt template is provided as:context=f″″″Given the below query question:{“user_query_question”}″″″user_prompt=f″″″is this question asking about benchmark of the model?Respond by “yes” or “no”.″″″prompt=[{“role”: “system”, “content”: context}, {“role”: “user”, “content”: user_prompt}]response=get_completion (prompt)

Listing 1: Example Prompt Template

In the example of Listing 1, {“user_query_question”} is variable containing “what is the benchmark of this model?” query from the user (e.g., the query230). In some examples, the task is to determine whether data associated with the query230, if any, can be graphically represented. The first few-shot prompt is provided to the LLM system210, which returns a response that is provided back to the application plug-in240.

In further detail, the LLM executed by the LLM system210processes the first few-shot prompt and determines whether the query is related to data that can be graphically represented in a GUI. For example, and with reference to the non-limiting example use case introduced above, the query230can be “What is the benchmark of this model?” with reference to a ML model that matches entities (e.g., bank statement line-items to invoices). In response, it can be determined that the query230relates to data (e.g., metrics) that can be graphically represented. In some implementations, if the response returned from the first few-shot prompt is no, a second few-shot prompt is generated and input to the LLM system210. In some implementations, if the response returned from the first few-shot prompt is yes, a second few-shot prompt and a third few-shot prompt are generated and input to the LLM system210.

In some implementations, the LLM generates explanatory text responsive to the second few-shot prompt, and the explanatory text is returned as a LLM response. In some examples, the second few-shot prompt includes the query230. The explanatory text is returned to the digital assistant220. In some examples, the first few-shot prompt is generated using a prompt template. An example prompt template is provided as:context=f″″″Given the benchmark data as below:{“active_model_benchmark_data”}″″″user_prompt=f″″″what is the benchmark of this model?″″″prompt=[{“role”: “system”, “content”: context}, {“role”:“user”, “content”: user_prompt}]response=get_completion (prompt)

Listing 2: Example Prompt Template

In some examples, a third few-shot prompt is generated and input to the LLM system210to perform a task. In some examples, the task includes generating UI code that can be executed to render a graphical representation of the data. For example, if the response to the first few-shot prompt is yes, the application plug-in240retrieves metric data for the particular ML model from the data management service204, which is provided to the prompt service208to generate a third few-shot prompt. In some examples, the metrics can be provided as:{{“company_code”: “ALL”,“active_correctly_proposed”: 796,“active_incorrectly_proposed”: 32,“active_not_proposed”: 191,“selected_correctly_proposed”: 808,“selected_incorrectly_proposed”: 32,“selected_not_proposed”: 179,“performance_gain”: 1.177625122669279}}

Listing 3: Example Metrics

In some examples, the third few-shot prompt is generated using a prompt template. An example prompt template is provided as:context=f″″″Given the benchmark data as below:{“active_model_benchmark_data”}″″″user_prompt=f″″″create UI5 chart variable by javascript which “Dimension”is by text “Company Code” and binding “company_code”, and“Measure” by text “Performance Gain” and binding“performance_gain”, data path in bound to“company_code_breakdown”″″″prompt=[{“role”: “system”, “content”: context}, {“role”; “user”, “content”: user_prompt}]response=get_completion (prompt)

Listing 4: Example Few-Shot Prompt Template

In some examples, the LLM system210returns a response to the third few-shot template, which includes UI code that is provided to the digital assistant220. Example UI code is provided as:

Listing 5: Example UI Code

In some implementations, the explanatory text is provided to the digital assistant220, which displays the textual explanation. In some examples, if the LLM response includes the code234, the digital assistant220triggers opening of the popover container222with window224to execute the code234(UI code) and display the graphical representation of the data.

FIG.3depicts an example UI300in accordance with implementations of the present disclosure. The example ofFIG.3includes a digital assistant302(e.g., UI of a digital assistant) and a popover container304. In the example ofFIG.3, an example query is provided as “What is the benchmark of this model?” input by a user to the digital assistant302. The LLM response includes explanatory text302adisplayed in the digital assistant302and a graphical representation304adisplayed in the popover container304.

FIG.4depicts an example process400that can be executed in accordance with implementations of the present disclosure. In some examples, the example process400is provided using one or more computer-executable programs executed by one or more computing devices.

A query is received (402). For example, and as described herein, the user218can input the query230into the digital assistant220. Data for the query is retrieved (404). For example, and as described herein, in response to the query230, the digital assistant service206requests and receives data responsive to the query230from the data management service204. A first few-shot prompt is generated (406) and a LLM is queried using the first few-shot prompt (408). For example, and as described herein, the prompt service208generates a first few-shot prompt based on the data and the first few-shot prompt is input to the LLM system210.

It is determined whether a graphical representation of the data is to be displayed (410). For example, and as described herein, a LLM executed by the LLM system210provides a prediction indicating whether the data is capable of being represented in a graphical representation. If a graphical representation of the data is not to be displayed, a second few-shot prompt is generated (412) and is used to query the LLM (414). For example, and as described herein, the prompt service208generates a second few-shot prompt to request that the LLM provide explanatory text and the second few-shot prompt is input to the LLM system210, which provides explanatory text. Explanatory text is displayed in the digital assistant (416). For example, and as described herein, explanatory text is displayed in the digital assistant220.

If a graphical representation of the data is to be displayed, a second few-shot prompt and a third few-shot prompt are generated (418) and are used to query the LLM (420). For example, and as described herein, the prompt service208generates a second few-shot prompt to request that the LLM provide explanatory text and the second few-shot prompt is input to the LLM system210, which provides explanatory text. As another example, and as described herein, the prompt service208generates a third few-shot prompt to request that the LLM provide UI code and the third few-shot prompt is input to the LLM system210, which provides UI code. Explanatory text is displayed in the digital assistant (422) and the graphical representation is displayed in the popover container (424). For example, and as described herein, explanatory text is displayed in the digital assistant220and the popover container222is created and the code234is executed to display a graphical representation of the data within the popover container222.

Referring now toFIG.5, a schematic diagram of an example computing system500is provided. The system500can be used for the operations described in association with the implementations described herein. For example, the system500may be included in any or all of the server components discussed herein. The system500includes a processor510, a memory520, a storage device530, and an input/output device540. The components510,520,530,540are interconnected using a system bus550. The processor510is capable of processing instructions for execution within the system500. In some implementations, the processor510is a single-threaded processor. In some implementations, the processor510is a multi-threaded processor. The processor510is capable of processing instructions stored in the memory520or on the storage device530to display graphical information for a user interface on the input/output device540.

The memory520stores information within the system500. In some implementations, the memory520is a computer-readable medium. In some implementations, the memory520is a volatile memory unit. In some implementations, the memory520is a non-volatile memory unit. The storage device530is capable of providing mass storage for the system500. In some implementations, the storage device530is a computer-readable medium. In some implementations, the storage device530may be a floppy disk device, a hard disk device, an optical disk device, or a tape device. The input/output device540provides input/output operations for the system500. In some implementations, the input/output device540includes a keyboard and/or pointing device. In some implementations, the input/output device540includes a display unit for displaying graphical user interfaces.