BUILDING MANAGEMENT SYSTEM WITH BUILDING EQUIPMENT SERVICE AND PARTS RECOMMENDATIONS

A method includes fine-tuning at least one large language model (LLM) using building domain data comprising information regarding equipment types, equipment parameters, and output conditions, facilitating generation of an input query for the at least one LLM by providing an interactive interface configured to guide input of a relevant equipment type, a relevant equipment parameter, a relevant output condition, and a request type by a user and generating the input query based on the input of the relevant equipment type, the relevant equipment parameter, the relevant output condition, and the request type, and providing a response to the input query as an output of the at least one LLM by using the input query as an input to the LLM.

BACKGROUND

The present disclosure relates generally to a building system of a building. The present disclosure relates more particularly to systems for managing and processing data of the building system.

Various interactions between building systems, components of building systems, users, technicians, and/or devices managed by users or technicians can rely on timely generation and presentation of data relating to the interactions, including for performing service operations. However, it can be difficult to generate the data elements to precisely identify proper response actions or sequences of response actions, as well as options for modified response actions, depending on various factors associated with items of equipment to be serviced, technical issues with the items of equipment, and the availability of timely, precise data to use for supporting the service operations.

SUMMARY

One or more aspects relate to building management systems and methods for use with building management systems. A method can include fine-tuning at least one large language model (LLM) using building domain data comprising information regarding equipment types, equipment parameters, and output conditions, facilitating generation of an input query for the at least one LLM by providing an interactive interface configured to guide input of a relevant equipment type, a relevant equipment parameter, a relevant output condition, and a request type by a user and generating the input query based on the input of the relevant equipment type, the relevant equipment parameter, the relevant output condition, and the request type, and providing a response to the input query as an output of the at least one LLM by using the input query as an input to the LLM.

In some embodiments, the request type is at least one of a request for a service recommendation, a request for a product recommendation, or a request for virtual assistance with technical service. The relevant output condition is a supply water temperature output by an equipment unit, a supply air temperature output by an equipment unit, a resource consumption of an equipment unit, or a resource generation of an equipment unit, an indoor air condition of a space served by the equipment unit. In some embodiments, the equipment parameters comprise equipment settings. In some embodiments, the equipment types comprise a plurality of types of chillers. In some embodiments, the equipment types comprise chillers, boilers, cooling towers, air handling units, variable air volume boxes, variable refrigerant flow units, and rooftop units. Providing the interactive interface can include generating conversational prompts based on natural language inputs by a user, the conversational prompts generated using a generative artificial intelligence model and configured to prompt the user to input the relevant equipment type, the relevant equipment parameter, the relevant output condition, and the request type of the user.

In some embodiments, the interactive interface comprises a plurality of fields configured to receive separate selections of the relevant equipment type, the relevant equipment parameter, the relevant output condition, and the request type of the user. Providing the interactive interface can be based on data indicative of equipment available in a building management system associated with the user. In some embodiments, the building domain data includes engineering data, equipment operational data, warranty data, service data, and sales data.

Another implementation of the present disclosure is a system. The system includes a plurality of chillers and a processing system programmed to provide an interactive interface configured to guide input of a relevant chiller parameter, a relevant output condition, an identification of a relevant chiller of the plurality of chillers, and a request type by a user, generate an input query for a large language model based on the input of the identification of the relevant chiller, the relevant chiller parameter, the relevant output condition, and the request type, and output a recommendation relating to the relevant chiller in accordance with the request type as output of the large language model by using the input query as an input to the LLM.

DETAILED DESCRIPTION

Referring generally to the FIGURES, systems and methods in accordance with the present disclosure can implement various systems to precisely generate data relating to operations to be performed for managing building systems and components and/or items of equipment, including heating, ventilation, cooling, and/or refrigeration (HVAC-R) systems and components. For example, various systems described herein can be implemented to more precisely generate data for various applications including, for example and without limitation, virtual assistance for supporting building managers and technician in determining appropriate service, maintenance, parts, updates, etc. that may be advantageous for particular building equipment; virtual assistance for supporting technicians responding to service requests; generating technical reports corresponding to service requests; facilitating diagnostics and troubleshooting procedures; recommendations of services to be performed; and/or recommendations for replacement parts, products or tools to use or install as part of service operations. Various such applications can facilitate both asynchronous and real-time service operations, including by generating text data for such applications based on data from disparate data sources that may not have predefined database associations amongst the data sources, yet may be relevant at specific steps or points in time during service operations.

In some systems, service operations can be supported by text information, such as predefined text documents such as service, diagnostic, and/or troubleshooting guides. Various such text information may not be useful for specific service requests and/or technicians performing the service. For example, the text information may correspond to different items of equipment or versions of items of equipment to be serviced. The text information, being predefined, may not account for specific technical issues that may be present in the items of equipment to be serviced.

AI and/or machine learning (ML) systems, including but not limited to LLMs, can be used to generate text data and data of other modalities in a more responsive manner to real-time conditions, including generating strings of text data that may not be provided in the same manner in existing documents, yet may still meet criteria for useful text information, such as relevance, style, and coherence. For example, LLMs can predict text data based at least on inputted prompts and by being configured (e.g., trained, modified, updated, fine-tuned) according to training data representative of the text data to predict or otherwise generate.

However, various considerations may limit the ability of such systems to precisely generate appropriate data for specific conditions. For example, due to the predictive nature of the generated data, some LLMs may generate text data that is incorrect, imprecise, or not relevant to the specific conditions. Using the LLMs may require a user to manually vary the content and/or syntax of inputs provided to the LLMs (e.g., vary inputted prompts) until the output of the LLMs meets various objective or subjective criteria of the user. The LLMs can have token limits for sizes of inputted text during training and/or runtime/inference operations (and relaxing or increasing such limits may require increased computational processing, API calls to LLM services, and/or memory usage), limiting the ability of the LLMs to be effectively configured or operated using large amounts of raw data or otherwise unstructured data.

Systems and methods in accordance with the present disclosure can use machine learning models, including LLMs and other generative AI systems, to capture data, including but not limited to unstructured knowledge from various data sources, and process the data to accurately generate outputs, such as completions responsive to prompts, including in structured data formats for various applications and use cases. The system can implement various automated and/or expert-based thresholds and data quality management processes to improve the accuracy and quality of generated outputs and update training of the machine learning models accordingly. The systems and method herein can enable real-time messaging and/or conversational or other interfaces for users to provide field data regarding equipment to the system (including presenting targeted queries to users that are expected to elicit relevant responses for efficiently receiving useful response information from users), for example implemented such that the system guides the user into providing inputs (e.g., prompts, questions, queries, criteria, equipment types, parameters, values, identified problems, etc.) particularly well-suited to eliciting high quality outputs from the models (e.g., LLMs, generative AI systems, etc.). Such interfaces can also provide outputs to guide users, such as service technicians, through relevant service, diagnostic, troubleshooting, acquisition, and/or repair processes.

This can include, for example, receiving data from technician service reports in various formats, including various modalities and/or multi-modal formats (e.g., text, speech, audio, image, and/or video). The system can facilitate automated, flexible customer report generation, such as by processing information received from service technicians and other users into a standardized format, which can reduce the constraints on how the user submits data while improving resulting reports. The system can couple unstructured service data to other input/output data sources and analytics, such as to relate unstructured data with outputs of timeseries data from equipment (e.g., sensor data; report logs) and/or outputs from models or algorithms of equipment operation, which can facilitate more accurate analytics, prediction services, diagnostics, and/or fault detection. The system can perform classification or other pattern recognition or trend detection operations to facilitate more timely assignment of technicians, scheduling of technicians based on expected times for jobs, and provisioning of trucks, tools, and/or parts. The system can perform root cause prediction by being trained using data that includes indications of root causes of faults or errors, where the indications are labels for or otherwise associated with (unstructured or structure) data such as service requests, service reports, service calls, etc. The system can receive, from a service technician in the field evaluating the issue with the equipment, feedback regarding the accuracy of the root cause predictions, as well as feedback regarding how the service technician evaluated information about the equipment (e.g., what data did they evaluate; what did they inspect; did the root cause prediction or instructions for finding the root cause accurately match the type of equipment, etc.), which can be used to update the root cause prediction model.

For example, the system can provide a platform for fault detection and servicing processes in which a machine learning model is configured based on connecting or relating unstructured data and/or semantic data, such as human feedback and written/spoken reports, with time-series product data regarding items of equipment, so that the machine learning model can more accurately detect causes of alarms or other events that may trigger service responses. For instance, responsive to an alarm for a chiller, the system can more accurately detect a cause of the alarm, and generate a prescription (e.g., for a service technician) for responding to the alarm; the system can request feedback from the service technician regarding the prescription, such as whether the prescription correctly identified the cause of the alarm and/or actions to perform to respond to the cause, as well as the information that the service technician used to evaluate the correctness or accuracy of the prescription; the system can use this feedback to modify the machine learning models, which can increase the accuracy of the machine learning models.

In some instances, significant computational resources (or human user resources) can be required to process data relating to equipment operation, such as time-series product data and/or sensor data, to detect or predict faults and/or causes of faults. In addition, it can be resource-intensive to label such data with identifiers of faults or causes of faults, which can make it difficult to generate machine learning training data from such data. Systems and methods in accordance with the present disclosure can leverage the efficiency of language models (e.g., GPT-based models or other pre-trained LLMs) in extracting semantic information (e.g., semantic information identifying faults, causes of faults, and other accurate expert knowledge regarding equipment servicing) from the unstructured data in order to use both the unstructured data and the data relating to equipment operation to generate more accurate outputs regarding equipment servicing. As such, by implementing language models using various operations and processes described herein, building management and equipment servicing systems can take advantage of the causal/semantic associations between the unstructured data and the data relating to equipment operation, and the language models can allow these systems to more efficiently extract these relationships in order to more accurately predict targeted, useful information for servicing applications at inference-time/runtime. While various implementations are described as being implemented using generative AI models such as transformers and/or GANs, in some embodiments, various features described herein can be implemented using non-generative AI models or even without using AI/machine learning, and all such modifications fall within the scope of the present disclosure.

The system can enable a generative AI-based service wizard interface. For example, the interface can include user interface and/or user experience features configured to provide a question/answer-based input/output format, such as a conversational interface, that directs users through providing targeted information for accurately generating predictions of root cause, presenting solutions, or presenting instructions for repairing or inspecting the equipment to identify information that the system can use to detect root causes or other issues. The system can use the interface to present information regarding parts and/or tools to service the equipment, as well as instructions for how to use the parts and/or tools to service the equipment.

In various implementations, the systems can include a plurality of machine learning models that may be configured using integrated or disparate data sources. This can facilitate more integrated user experiences or more specialized (and/or lower computational usage for) data processing and output generation. Outputs from one or more first systems, such as one or more first algorithms or machine learning models, can be provided at least as part of inputs to one or more second systems, such as one or more second algorithms or machine learning models. For example, a first language model can be configured to process unstructured inputs (e.g., text, speech, images, etc.) into a structure output format compatible for use by a second system, such as a root cause prediction algorithm or equipment configuration model.

The system can be used to automate interventions for equipment operation, servicing, fault detection and diagnostics (FDD), and alerting operations. For example, by being configured to perform operations such as root cause prediction, the system can monitor data regarding equipment to predict events associated with faults and trigger responses such as alerts, service scheduling, and initiating FDD or modifications to configuration of the equipment. The system can present to a technician or manager of the equipment a report regarding the intervention (e.g., action taken responsive to predicting a fault or root cause condition) and requesting feedback regarding the accuracy of the intervention, which can be used to update the machine learning models to more accurately generate interventions.

The system can be used to provide a virtual assistant that guides users to effective prompts (questions, etc.) to provide as an input to a generative AI model or other large language model in order to receive an output from the model, with the effective prompts expected to cause the model to provide accurate, effective technical information to the user. The model can be fine-tuned via a training process on equipment engineering data (e.g., manuals, operating procedures, specification sheets), operational data (e.g., sensor data, meter data, setpoints, settings, metrics such as a connected equipment performance index, runtime, downtime, etc.), warranty data (e.g., warranty claims, data from warranty management systems), service data (e.g., service orders, reports, etc.), and/or replacement part data (e.g., sales data, inventories, part descriptions, feedback on parts, etc.), for example relating to a particular category of equipment (e.g., building equipment, chillers, air handling units, rooftop units, variable refrigerant flow systems, etc.), such that the model is trained to output specialized, technical information based on specialized, technical prompts. A virtual assistant adapted for a particular type of equipment or facility can thereby be provided according to teachings herein.

I. Machine Learning Models for Building Management and Equipment Servicing

FIG.1depicts an example of a system100. The system100can implement various operations for configuring (e.g., training, updating, modifying, transfer learning, fine-tuning, etc.) and/or operating various AI and/or ML systems, such as neural networks of LLMs or other generative AI systems. The system100can be used to implement various generative AI-based building equipment servicing operations.

For example, the system100can be implemented for operations associated with any of a variety of building management systems (BMSs) or equipment or components thereof. A BMS can include a system of devices that can control, monitor, and manage equipment in or around a building or building area. The BMS can include, for example, a HVAC system, a security system, a lighting system, a fire alerting system, any other system that is capable of managing building functions or devices, or any combination thereof. The BMS can include or be coupled with items of equipment, for example and without limitation, such as heaters, chillers, boilers, air handling units, sensors, actuators, refrigeration systems, fans, blowers, heat exchangers, energy storage devices, condensers, valves, or various combinations thereof.

The items of equipment can operate in accordance with various qualitative and quantitative parameters, variables, setpoints, and/or thresholds or other criteria, for example. In some instances, the system100and/or the items of equipment can include or be coupled with one or more controllers for controlling parameters of the items of equipment, such as to receive control commands for controlling operation of the items of equipment via one or more wired, wireless, and/or user interfaces of controller.

Various components of the system100or portions thereof can be implemented by one or more processors coupled with or more memory devices (memory). The processors can be a general purpose or specific purpose processors, an application specific integrated circuit (ASIC), one or more field programmable gate arrays (FPGAs), a group of processing components, or other suitable processing components. The processors may be configured to execute computer code and/or instructions stored in the memories or received from other computer readable media (e.g., CDROM, network storage, a remote server, etc.). The processors can be configured in various computer architectures, such as graphics processing units (GPUs), distributed computing architectures, cloud server architectures, client-server architectures, or various combinations thereof. One or more first processors can be implemented by a first device, such as an edge device, and one or more second processors can be implemented by a second device, such as a server or other device that is communicatively coupled with the first device and may have greater processor and/or memory resources.

The memories can include one or more devices (e.g., memory units, memory devices, storage devices, etc.) for storing data and/or computer code for completing and/or facilitating the various processes described in the present disclosure. The memories can include random access memory (RAM), read-only memory (ROM), hard drive storage, temporary storage, non-volatile memory, flash memory, optical memory, or any other suitable memory for storing software objects and/or computer instructions. The memories can include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure. The memories can be communicably connected to the processors and can include computer code for executing (e.g., by the processors) one or more processes described herein.

Machine Learning Models

The system100can include or be coupled with one or more first models104. The first model104can include one or more neural networks, including neural networks configured as generative models. For example, the first model104can predict or generate new data (e.g., artificial data; synthetic data; data not explicitly represented in data used for configuring the first model104). The first model104can generate any of a variety of modalities of data, such as text, speech, audio, images, and/or video data. The neural network can include a plurality of nodes, which may be arranged in layers for providing outputs of one or more nodes of one layer as inputs to one or more nodes of another layer. The neural network can include one or more input layers, one or more hidden layers, and one or more output layers. Each node can include or be associated with parameters such as weights, biases, and/or thresholds, representing how the node can perform computations to process inputs to generate outputs. The parameters of the nodes can be configured by various learning or training operations, such as unsupervised learning, weakly supervised learning, semi-supervised learning, or supervised learning.

For example, the first model104can include at least one GPT model. The GPT model can receive an input sequence, and can parse the input sequence to determine a sequence of tokens (e.g., words or other semantic units of the input sequence, such as by using Byte Pair Encoding tokenization). The GPT model can include or be coupled with a vocabulary of tokens, which can be represented as a one-hot encoding vector, where each token of the vocabulary has a corresponding index in the encoding vector; as such, the GPT model can convert the input sequence into a modified input sequence, such as by applying an embedding matrix to the token tokens of the input sequence (e.g., using a neural network embedding function), and/or applying positional encoding (e.g., sin-cosine positional encoding) to the tokens of the input sequence. The GPT model can process the modified input sequence to determine a next token in the sequence (e.g., to append to the end of the sequence), such as by determining probability scores indicating the likelihood of one or more candidate tokens being the next token, and selecting the next token according to the probability scores (e.g., selecting the candidate token having the highest probability scores as the next token). For example, the GPT model can apply various attention and/or transformer based operations or networks to the modified input sequence to identify relationships between tokens for detecting the next token to form the output sequence.

The first model104can include at least one diffusion model, which can be used to generate image and/or video data. For example, the diffusional model can include a denoising neural network and/or a denoising diffusion probabilistic model neural network. The denoising neural network can be configured by applying noise to one or more training data elements (e.g., images, video frames) to generate noised data, providing the noised data as input to a candidate denoising neural network, causing the candidate denoising neural network to modify the noised data according to a denoising schedule, evaluating a convergence condition based on comparing the modified noised data with the training data instances, and modifying the candidate denoising neural network according to the convergence condition (e.g., modifying weights and/or biases of one or more layers of the neural network). In some implementations, the first model104includes a plurality of generative models, such as GPT and diffusion models, that can be trained separately or jointly to facilitate generating multi-modal outputs, such as technical documents (e.g., service guides) that include both text and image/video information.

In some implementations, the first model104can be configured using various unsupervised and/or supervised training operations. The first model104can be configured using training data from various domain-agnostic and/or domain-specific data sources, including but not limited to various forms of text, speech, audio, image, and/or video data, or various combinations thereof. The training data can include a plurality of training data elements (e.g., training data instances). Each training data element can be arranged in structured or unstructured formats; for example, the training data element can include an example output mapped to an example input, such as a query representing a service request or one or more portions of a service request, and a response representing data provided responsive to the query. The training data can include data that is not separated into input and output subsets (e.g., for configuring the first model104to perform clustering, classification, or other unsupervised ML operations). The training data can include human-labeled information, including but not limited to feedback regarding outputs of the models104,116. This can allow the system100to generate more human-like outputs.

In some implementations, the training data includes data relating to building management systems. For example, the training data can include examples of HVAC-R data, such as operating manuals, technical data sheets, configuration settings, operating setpoints, diagnostic guides, troubleshooting guides, user reports, technician reports. In some implementations, the training data used to configure the first model104includes at least some publicly accessible data, such as data retrievable via the Internet.

Referring further toFIG.1, the system100can configure the first model104to determine one or more second models116. For example, the system100can include a model updater108that configures (e.g., trains, updates, modifies, fine-tunes, etc.) the first model104to determine the one or more second models116. In some implementations, the second model116can be used to provide application-specific outputs, such as outputs having greater precision, accuracy, or other metrics, relative to the first model, for targeted applications.

The second model116can be similar to the first model104. For example, the second model116can have a similar or identical backbone or neural network architecture as the first model104. In some implementations, the first model104and the second model116each include generative AI machine learning models, such as LLMs (e.g., GPT-based LLMs) and/or diffusion models. The second model116can be configured using processes analogous to those described for configuring the first model104.

In some implementations, the model updater108can perform operations on at least one of the first model104or the second model116via one or more interfaces, such as application programming interfaces (APIs). For example, the models104,116can be operated and maintained by one or more systems separate from the system100. The model updater108can provide training data to the first model104, via the API, to determine the second model116based on the first model104and the training data. The model updater108can control various training parameters or hyperparameters (e.g., learning rates, etc.) by providing instructions via the API to manage configuring the second model116using the first model104.

Data Sources

The model updater108can determine the second model116using data from one or more data sources112. For example, the system100can determine the second model116by modifying the first model104using data from the one or more data sources112. The data sources112can include or be coupled with any of a variety of integrated or disparate databases, data warehouses, digital twin data structures (e.g., digital twins of items of equipment or building management systems or portions thereof), data lakes, data repositories, documentation records, or various combinations thereof. In some implementations, the data sources112include HVAC-R data in any of text, speech, audio, image, or video data, or various combinations thereof, such as data associated with HVAC-R components and procedures including but not limited to installation, operation, configuration, repair, servicing, diagnostics, and/or troubleshooting of HVAC-R components and systems. Various data described below with reference to data sources112may be provided in the same or different data elements, and may be updated at various points. The data sources112can include or be coupled with items of equipment (e.g., where the items of equipment output data for the data sources112, such as sensor data, etc.). The data sources112can include various online and/or social media sources, such as blog posts or data submitted to applications maintained by entities that manage the buildings. The system100can determine relations between data from different sources, such as by using timeseries information and identifiers of the sites or buildings at which items of equipment are present to detect relationships between various different data relating to the items of equipment (e.g., to train the models104,116using both timeseries data (e.g., sensor data; outputs of algorithms or models, etc.) regarding a given item of equipment and freeform natural language reports regarding the given item of equipment).

The data sources112can include unstructured data or structured data (e.g., data that is labeled with or assigned to one or more predetermined fields or identifiers). For example, using the first model104and/or second model116to process the data can allow the system100to extract useful information from data in a variety of formats, including unstructured/freeform formats, which can allow service technicians to input information in less burdensome formats. The data can be of any of a plurality of formats (e.g., text, speech, audio, image, video, etc.), including multi-modal formats. For example, the data may be received from service technicians in forms such as text (e.g., laptop/desktop or mobile application text entry), audio, and/or video (e.g., dictating findings while capturing video).

The data sources112can include engineering data regarding one or more items of equipment. The engineering data can include manuals, such as installation manuals, instruction manuals, or operating procedure guides. The engineering data can include specifications or other information regarding operation of items of equipment. The engineering data can include engineering drawings, process flow diagrams, refrigeration cycle parameters (e.g., temperatures, pressures), or various other information relating to structures and functions of items of equipment.

In some implementations, the data sources112can include operational data regarding one or more items of equipment. The operational data can represent detected information regarding items of equipment, such as sensor data, logged data, user reports, or technician reports. The operational data can include, for example, service tickets generated responsive to requests for service, work orders, data from digital twin data structures maintained by an entity of the item of equipment, outputs or other information from equipment operation models (e.g., chiller vibration models), or various combinations thereof. Logged data, user reports, service tickets, billing records, time sheets, and various other such data can provide temporal information, such as how long service operations may take, or durations of time between service operations, which can allow the system100to predict resources to use for performing service as well as when to request service.

The data sources112can include, for instance, warranty data. The warranty data can include warranty documents or agreements that indicate conditions under which various entities associated with items of equipment are to provide service, repair, or other actions corresponding to items of equipment, such as actions corresponding to service requests.

The data sources112can include service data. The service data can include data from any of various service providers, such as service reports. The service data can indicate service procedures performed, including associated service procedures with initial service requests and/or sensor data related conditions to trigger service and/or sensor data measured during service processes.

In some implementations, the data sources112can include parts data, including but not limited to parts usage and sales data. For example, the data sources112can indicate various parts associated with installation or repair of items of equipment. The data sources112can indicate tools for performing service and/or installing parts.

The system100can include, with the data of the data sources112, labels to facilitate cross-reference between items of data that may relate to common items of equipment, sites, service technicians, customers, or various combinations thereof. For example, data from disparate sources may be labeled with time data, which can allow the system100(e.g., by configuring the models104,116) to increase a likelihood of associating information from the disparate sources due to the information being detected or recorded (e.g., as service reports) at the same time or near in time.

For example, the data sources112can include data that can be particular to specific or similar items of equipment, buildings, equipment configurations, environmental states, or various combinations thereof. In some implementations, the data includes labels or identifiers of such information, such as to indicate locations, weather conditions, timing information, uses of the items of equipment or the buildings or sites at which the items of equipment are present, etc. This can enable the models104,116to detect patterns of usage (e.g., spikes; troughs; seasonal or other temporal patterns) or other information that may be useful for determining causes of issues or causes of service requests, or predict future issues, such as to allow the models104,116to be trained using information indicative of causes of issues across multiple items of equipment (which may have the same or similar causes even if the data regarding the items of equipment is not identical). For example, an item of equipment may be at a site that is a museum; by relating site usage or occupancy data with data regarding the item of equipment, such as sensor data and service reports, the system100can configure the models104,116to determine a high likelihood of issues occurring before events associated with high usage (e.g., gala, major exhibit opening), and can generate recommendations to perform diagnostics or servicing prior to the events.

Model Configuration

Referring further toFIG.1, the model updater108can perform various machine learning model configuration/training operations to determine the second models116using the data from the data sources112. For example, the model updater108can perform various updating, optimization, retraining, reconfiguration, fine-tuning, or transfer learning operations, or various combinations thereof, to determine the second models116. The model updater108can configure the second models116, using the data sources112, to generate outputs (e.g., completions) in response to receiving inputs (e.g., prompts), where the inputs and outputs can be analogous to data of the data sources112.

For example, the model updater108can identify one or more parameters (e.g., weights and/or biases) of one or more layers of the first model104, and maintain (e.g., freeze, maintain as the identified values while updating) the values of the one or more parameters of the one or more layers. In some implementations, the model updater108can modify the one or more layers, such as to add, remove, or change an output layer of the one or more layers, or to not maintain the values of the one or more parameters. The model updater108can select at least a subset of the identified one or parameters to maintain according to various criteria, such as user input or other instructions indicative of an extent to which the first model104is to be modified to determine the second model116. In some implementations, the model updater108can modify the first model104so that an output layer of the first model104corresponds to output to be determined for applications120.

Responsive to selecting the one or more parameters to maintain, the model updater108can apply, as input to the second model116(e.g., to a candidate second model116, such as the modified first model104, such as the first model104having the identified parameters maintained as the identified values), training data from the data sources112. For example, the model updater108can apply the training data as input to the second model116to cause the second model116to generate one or more candidate outputs.

The model updater108can evaluate a convergence condition to modify the candidate second model116based at least on the one or more candidate outputs and the training data applied as input to the candidate second model116. For example, the model updater108can evaluate an objective function of the convergence condition, such as a loss function (e.g., L1 loss, L2 loss, root mean square error, cross-entropy or log loss, etc.) based on the one or more candidate outputs and the training data; this evaluation can indicate how closely the candidate outputs generated by the candidate second model116correspond to the ground truth represented by the training data. The model updater108can use any of a variety of optimization algorithms (e.g., gradient descent, stochastic descent, Adam optimization, etc.) to modify one or more parameters (e.g., weights or biases of the layer(s) of the candidate second model116that are not frozen) of the candidate second model116according to the evaluation of the objective function. In some implementations, the model updater108can use various hyperparameters to evaluate the convergence condition and/or perform the configuration of the candidate second model116to determine the second model116, including but not limited to hyperparameters such as learning rates, numbers of iterations or epochs of training, etc.

As described further herein with respect to applications120, in some implementations, the model updater108can select the training data from the data of the data sources112to apply as the input based at least on a particular application of the plurality of applications120for which the second model116is to be used for. For example, the model updater108can select data from the parts data source112for the product recommendation generator application120, or select various combinations of data from the data sources112(e.g., engineering data, operational data, and service data) for the service recommendation generator application120. The model updater108can apply various combinations of data from various data sources112to facilitate configuring the second model116for one or more applications120.

In some implementations, the system100can perform at least one of conditioning, classifier-based guidance, or classifier-free guidance to configure the second model116using the data from the data sources112. For example, the system100can use classifiers associated with the data, such as identifiers of the item of equipment, a type of the item of equipment, a type of entity operating the item of equipment, a site at which the item of equipment is provided, or a history of issues at the site, to condition the training of the second model116. For example, the system100combine (e.g., concatenate) various such classifiers with the data for inputting to the second model116during training, for at least a subset of the data used to configure the second model116, which can enable the second model116to be responsive to analogous information for runtime/inference time operations.

Applications

Referring further toFIG.1, the system100can use outputs of the one or more second models116to implement one or more applications120. For example, the second models116, having been configured using data from the data sources112, can be capable of precisely generating outputs that represent useful, timely, and/or real-time information for the applications120. In some implementations, each application120is coupled with a corresponding second model116that is specifically configured to generate outputs for use by the application120. Various applications120can be coupled with one another, such as to provide outputs from a first application120as inputs or portions of inputs to a second application120.

The applications120can include any of a variety of desktop, web-based/browser-based, or mobile applications. For example, the applications120can be implemented by enterprise management software systems, employee or other user applications (e.g., applications that relate to BMS functionality such as temperature control, user preferences, conference room scheduling, etc.), equipment portals that provide data regarding items of equipment, or various combinations thereof. The applications120can include user interfaces, wizards, checklists, conversational interfaces, chatbots, configuration tools, or various combinations thereof. The applications120can receive an input, such as a prompt (e.g., from a user), provide the prompt to the second model116to cause the second model116to generate an output, such as a completion in response to the prompt, and present an indication of the output. The applications120can receive inputs and/or present outputs in any of a variety of presentation modalities, such as text, speech, audio, image, and/or video modalities. For example, the applications120can receive unstructured or freeform inputs from a user, such as a service technician, and generate reports in a standardized format, such as a customer-specific format. This can allow, for example, technicians to automatically, and flexibly, generate customer-ready reports after service visits without requiring strict input by the technician or manually sitting down and writing reports; to receive inputs as dictations in order to generate reports; to receive inputs in any form or a variety of forms, and use the second model116(which can be trained to cross-reference metadata in different portions of inputs and relate together data elements) to generate output reports (e.g., the second model116, having been configured with data that includes time information, can use timestamps of input from dictation and timestamps of when an image is taken, and place the image in the report in a target position or label based on time correlation). As another example, the applications120can receive structured or unstructured inputs from a user, such as a building manager or operator, and provide service recommendations, parts recommendations, and/or other information relating to particular faults, performance degradation, or other operational issues for particular equipment. In some such examples, the applications120provide structured input interfaces and/or conversational prompts configured to request sufficient technical information from the user relating to the equipment of interest (e.g., type, model, particular unit, etc.), a parameter of interest (e.g., supply water temperature, return water temperature, compressor frequency, energy consumption, air temperature, humidity, pressure, airflow, vibration level, chiller performance index, etc.), an issue relating to the parameter (e.g., parameter too high, parameter too low, parameter unstable, value of a parameter, etc.), and a type of information requested (e.g., a request for a service recommendation, a request for a description of possible causes of the issue, a request for a replacement part recommendation, etc.), where the sufficient technical information is synergistic with data used in model training such that quality responses to prompts based on the user-input technical information can be generated by the systems herein.

In some implementations, the applications120include at least one virtual assistant (e.g., virtual assistance for technician services) application120. The virtual assistant application can provide various services to building manager and support technician operations, such as presenting information relating to building equipment and information from service requests, receiving queries regarding diagnosis of equipment-related issues, queries seeking recommendations for resolving such issues, and queries regarding actions to perform to service items of equipment, and presenting responses indicating potential equipment issues, recommended service tasks and/or replacement parts, and actions to perform to service items of equipment. The virtual assistant application can receive information regarding an item of equipment to be serviced, such as sensor data, text descriptions, or camera images, and process the received information using the second model116to generate corresponding responses.

For example, the virtual assistant application120can be implemented in a UI/UX wizard configuration, such as to provide a sequence of requests for information from the user (the sequence may include requests that are at least one of predetermined or dynamically generated responsive to inputs from the user for previous requests). For example, the virtual assistant application120can provide one or more requests for users such as service technicians, facility managers, or other occupants, and provide the received responses to at least one of the second model116or a root cause detection function (e.g., algorithm, model, data structure mapping inputs to candidate causes, etc.) to determine a prediction of a cause of the issue of the item of equipment and/or solutions. The virtual assistant application120can use requests for information such as for unstructured text by which the user describes characteristics of the item of equipment relating to the issue; answers expected to correspond to different scenarios indicative of the issue; and/or image and/or video input (e.g., images of problems, equipment, spaces, etc. that can provide more context around the issue and/or configurations). For example, responsive to receiving a response via the virtual assistant application120indicating that the problem is with temperature in the space, the system100can request, via the virtual assistant application120, information regarding HVAC-R equipment associated with the space, such as pictures of the space, an air handling unit, a chiller, or various combinations thereof.

The virtual assistant application120can include a plurality of applications120(e.g., variations of interfaces or customizations of interfaces) for a plurality of respective user types. For example, the virtual assistant application120can include a first application120for a customer user (e.g., building manager, building operator), and a second application120for a service technician user. The virtual assistant applications120can allow for updating and other communications between the first and second applications120as well as the second model116. Using one or more of the first application120and the second application120, the system100can manage continuous/real-time conversations for one or more users, and evaluate the users' engagement with the information provided (e.g., did the user, customer, service technician, etc., follow the provided steps for responding to the issue or performing service, did the user discontinue providing inputs to the virtual assistant application120, etc.), such as to enable the system100to update the information generated by the second model116for the virtual assistant application120according to the engagement. In some implementations, the system100can use the second model116to detect sentiment of the user of the virtual assistant application120, and update the second model116according to the detected sentiment, such as to improve the experience provided by the virtual assistant application120.

The applications120can include at least one document writer application120, such as a technical document writer. The document writer application120can facilitate preparing structured (e.g. form-based) and/or unstructured documentation, such as documentation associated with service requests. For example, the document writer application120can present a user interface corresponding to a template document to be prepared that is associated with at least one of a service request or the item of equipment for which the service request is generated, such as to present one or more predefined form sections or fields. The document writer application120can use inputs, such as prompts received from the users and/or technical data provided by the user regarding the item of equipment, such as sensor data, text descriptions, or camera images, to generate information to include in the documentation. For example, the document writer application120can provide the inputs to the second model116to cause the second model116to generate completions for text information to include in the fields of the documentation.

The applications120can include, in some implementations, at least one diagnostics and troubleshooting application120. The diagnostics and troubleshooting application120can receive inputs including at least one of a service request or information regarding the item of equipment to be serviced, such as information identified by a service technician. The diagnostics and troubleshooting application120can provide the inputs to a corresponding second model116to cause the second model116to generate outputs such as indications of potential items to be checked regarding the item of equipment, modifications or fixes to make to perform the service, or values or ranges of values of parameters of the item of equipment that may be indicative of specific issues to for the service technician to address or repair.

The applications120can at least one service recommendation generator application120. The service recommendation generator application120can receive inputs such as a service request or information regarding the item of equipment to be serviced, and provide the inputs to the second model116to cause the second model116to generate outputs for presenting service recommendations, such as actions to perform to address the service request.

In some implementations, the applications120can include a product recommendation generator application120. The product recommendation generator application120can process inputs such as information regarding the item of equipment or the service request, using one or more second models116(e.g., models trained using parts data from the data sources112), to determine a recommendation of a part or product to replace or otherwise use for repairing the item of equipment.

Feedback Training

Referring further toFIG.1, the system100can include at least one feedback trainer128coupled with at least one feedback repository124. The system100can use the feedback trainer128to increase the precision and/or accuracy of the outputs generated by the second models116according to feedback provided by users of the system100and/or the applications120.

The feedback repository124can include feedback received from users regarding output presented by the applications120. For example, for at least a subset of outputs presented by the applications120, the applications120can present one or more user input elements for receiving feedback regarding the outputs. The user input elements can include, for example, indications of binary feedback regarding the outputs (e.g., good/bad feedback; feedback indicating the outputs do or do not meet the user's criteria, such as criteria regarding technical accuracy or precision); indications of multiple levels of feedback (e.g., scoring the outputs on a predetermined scale, such as a 1-5 scale or 1-10 scale); freeform feedback (e.g., text or audio feedback); or various combinations thereof.

The system100can store and/or maintain feedback in the feedback repository124. In some implementations, the system100stores the feedback with one or more data elements associated with the feedback, including but not limited to the outputs for which the feedback was received, the second model(s)116used to generate the outputs, and/or input information used by the second models116to generate the outputs (e.g., service request information; information captured by the user regarding the item of equipment).

The feedback trainer128can update the one or more second models116using the feedback. The feedback trainer128can be similar to the model updater108. In some implementations, the feedback trainer128is implemented by the model updater108; for example, the model updater108can include or be coupled with the feedback trainer128. The feedback trainer128can perform various configuration operations (e.g., retraining, fine-tuning, transfer learning, etc.) on the second models116using the feedback from the feedback repository124. In some implementations, the feedback trainer128identifies one or more first parameters of the second model116to maintain as having predetermined values (e.g., freeze the weights and/or biases of one or more first layers of the second model116), and performs a training process, such as a fine tuning process, to configure parameters of one or more second parameters of the second model116using the feedback (e.g., one or more second layers of the second model116, such as output layers or output heads of the second model116).

In some implementations, the system100may not include and/or use the model updater108(or the feedback trainer128) to determine the second models116. For example, the system100can include or be coupled with an output processor (e.g., an output processor similar or identical to accuracy checker316described with reference toFIG.3) that can evaluate and/or modify outputs from the first model104prior to operation of applications120, including to perform any of various post-processing operations on the output from the first model104. For example, the output processor can compare outputs of the first model104with data from data sources112to validate the outputs of the first model104and/or modify the outputs of the first model104(or output an error) responsive to the outputs not satisfying a validation condition.

Connected Machine Learning Models

Referring further toFIG.1, the second model116can be coupled with one or more third models, functions, or algorithms for training/configuration and/or runtime operations. The third models can include, for example and without limitation, any of various models relating to items of equipment, such as energy usage models, sustainability models, carbon models, air quality models, or occupant comfort models. For example, the second model116can be used to process unstructured information regarding items of equipment into predefined template formats compatible with various third models, such that outputs of the second model116can be provided as inputs to the third models; this can allow more accurate training of the third models, more training data to be generated for the third models, and/or more data available for use by the third models. The second model116can receive inputs from one or more third models, which can provide greater data to the second model116for processing.

Question Crafter

Still referring toFIG.1, the system100is shown as including a question crafter117. The prompt crafter117is adapted to craft prompts (e.g., questions, input queries, etc.) for input to the model116and/or104. The prompt crafter117is adapted in model training alongside the model116to craft appropriate prompts predicted to cause the model116to return quality results to a user (e.g., a service or product recommendation as described elsewhere herein), for example a resulting that can be described as relevant, accurate, coherent, etc. In some embodiments, the prompt crafter117is configured to interact with applications120to provide an interactive interface that prompts as user for particular information for use by the prompt crafter117to craft (generate, determine, draft, create, etc.) that includes sufficient information arranged in an appropriate manner to cause the model116(or114) to generate suitable content responsive to a user's desire for relevant information. The prompt creator117can provide various conversational and/or structured interfaces adapted to cause the user to provide appropriate information to enable generation of a well-suited prompt by the prompt creator117. The prompt creator117can employ one or more prompt templates to be populated with information collected from a user to generate prompts, in some embodiments. The prompt creator117can itself be or include a machine learning model (e.g., generative AI model) according to the teachings herein, and/or can be or employ conventional programming using formulaic prompts for generation. In any such embodiments, the prompt crafter117can be adapted based on characteristics of both the model104and the updated model116, such that the prompt crafter117is updated in view of various types of data and information from the data sources112. For example, the prompt crafter117can be configured to generate prompts relating to parameters, variables, entities, equipment, settings, results, etc. represented in the data sources112(e.g., in the engineering data, operational data, warranty data, service data, parts data, etc. as shown inFIG.1).

Automated Service Scheduling and Provisioning

The system100can be used to automate operations for scheduling, provisioning, and deploying service technicians and resources for service technicians to perform service operations. For example, the system100can use at least one of the first model104or the second model116to determine, based on processing information regarding service operations for items of equipment relative to completion criteria for the service operation, particular characteristics of service operations such as experience parameters of scheduled service technicians, identifiers of parts provided for the service operations, geographical data, types of customers, types of problems, or information content provided to the service technicians to facilitate the service operation, where such characteristics correspond to the completion criteria being satisfied (e.g., where such characteristics correspond to an increase in likelihood of the completion criteria being satisfied relative to other characteristics for service technicians, parts, information content, etc.). For example, the system100can determine, for a given item of equipment, particular parts to include on a truck to be sent to the site of the item of equipment. As such, the system100, responsive to processing inputs at runtime such as service requests, can automatically and more accurately identify service technicians and parts to direct to the item of equipment for the service operations. The system100can use timing information to perform batch scheduling for multiple service operations and/or multiple technicians for the same or multiple service operations. The system100can perform batch scheduling for multiple trucks for multiple items of equipment, such as to schedule a first one or more parts having a greater likelihood for satisfying the completion criteria for a first item of equipment on a first truck, and a second one or more parts having a greater likelihood for satisfying the completion criteria for a second item of equipment on a second truck.

II. System Architectures for Generative AI Applications for Building Management System and Equipment Servicing

FIG.2depicts an example of a system200. The system200can include one or more components or features of the system100, such as any one or more of the first model104, data sources112, second model116, applications120, feedback repository124, and/or feedback trainer128. The system200can perform specific operations to enable generative AI applications for building managements systems and equipment servicing, such as various manners of processing input data into training data (e.g., tokenizing input data; forming input data into prompts and/or completions), and managing training and other machine learning model configuration processes. Various components of the system200can be implemented using one or more computer systems, which may be provided on the same or different processors (e.g., processors communicatively coupled via wired and/or wireless connections).

The system200can include at least one data repository204, which can be similar to the data sources112described with reference toFIG.1. For example, the data repository204can include a transaction database208, which can be similar or identical to one or more of warranty data or service data of data sources112. For example, the transaction database208can include data such as parts used for service transactions; sales data indicating various service transactions or other transactions regarding items of equipment; warranty and/or claims data regarding items of equipment; and service data.

The data repository204can include a product database212, which can be similar or identical to the parts data of the data sources112. The product database212can include, for example, data regarding products available from various vendors, specifications or parameters regarding products, and indications of products used for various service operations. The products database212can include data such as events or alarms associated with products; logs of product operation; and/or time series data regarding product operation, such as longitudinal data values of operation of products and/or building equipment.

The data repository204can include an operations database216, which can be similar or identical to the operations data of the data sources112. For example, the operations database216can include data such as manuals regarding parts, products, and/or items of equipment; customer service data; and or reports, such as operation or service logs.

In some implementations, the data repository204can include an output database220, which can include data of outputs that may be generated by various machine learning models and/or algorithms. For example, the output database220can include values of pre-calculated predictions and/or insights, such as parameters regarding operation items of equipment, such as setpoints, changes in setpoints, flow rates, control schemes, identifications of error conditions, or various combinations thereof.

As depicted inFIG.2, the system200can include a prompt management system228. The prompt management system228can include one or more rules, heuristics, logic, policies, algorithms, functions, machine learning models, neural networks, scripts, or various combinations thereof to perform operations including processing data from data repository204into training data for configuring various machine learning models. For example, the prompt management system228can retrieve and/or receive data from the data repository228, and determine training data elements that include examples of input and outputs for generation by machine learning models, such as a training data element that includes a prompt and a completion corresponding to the prompt, based on the data from the data repository228.

In some implementations, the prompt management system228includes a pre-processor232. The pre-processor232can perform various operations to prepare the data from the data repository204for prompt generation. For example, the pre-processor232can perform any of various filtering, compression, tokenizing, or combining (e.g., combining data from various databases of the data repository204) operations.

The prompt management system228can include a prompt generator236. The prompt generator236can generate, from data of the data repository204, one or more training data elements that include a prompt and a completion corresponding to the prompt. In some implementations, the prompt generator236receives user input indicative of prompt and completion portions of data. For example, the user input can indicate template portions representing prompts of structured data, such as predefined fields or forms of documents, and corresponding completions provided for the documents. The user input can assign prompts to unstructured data. In some implementations, the prompt generator236automatically determines prompts and completions from data of the data repository204, such as by using any of various natural language processing algorithms to detect prompts and completions from data. In some implementations, the system200does not identify distinct prompts and completions from data of the data repository204.

Referring further toFIG.2, the system200can include a training management system240. The training management system240can include one or more rules, heuristics, logic, policies, algorithms, functions, machine learning models, neural networks, scripts, or various combinations thereof to perform operations including controlling training of machine learning models, including performing fine tuning and/or transfer learning operations.

The training management system240can include a training manager244. The training manager244can incorporate features of at least one of the model updater108or the feedback trainer128described with reference toFIG.1. For example, the training manager244can provide training data including a plurality of training data elements (e.g., prompts and corresponding completions) to the model system260as described further herein to facilitate training machine learning models.

In some implementations, the training management system240includes a prompts database248. For example, the training management system240can store one or more training data elements from the prompt management system228, such as to facilitate asynchronous and/or batched training processes.

The training manager244can control the training of machine learning models using information or instructions maintained in a model tuning database256. For example, the training manager244can store, in the model tuning database256, various parameters or hyperparameters for models and/or model training.

In some implementations, the training manager244stores a record of training operations in a jobs database252. For example, the training manager244can maintain data such as a queue of training jobs, parameters or hyperparameters to be used for training jobs, or information regarding performance of training.

Referring further toFIG.2, the system200can include at least one model system260(e.g., one or more language model systems). The model system260can include one or more rules, heuristics, logic, policies, algorithms, functions, machine learning models, neural networks, scripts, or various combinations thereof to perform operations including configuring one or more machine learning models268based on instructions from the training management system240. In some implementations, the training management system240implements the model system260. In some implementations, the training management system240can access the model system260using one or more APIs, such as to provide training data and/or instructions for configuring machine learning models268via the one or more APIs. The model system260can operate as a service layer for configuring the machine learning models268responsive to instructions from the training management system240. The machine learning models268can be or include the first model104and/or second model116described with reference toFIG.1.

The model system260can include a model configuration processor264. The model configuration processor264can incorporate features of the model updater108and/or the feedback trainer128described with reference toFIG.1. For example, the model configuration processor264can apply training data (e.g., prompts248and corresponding completions) to the machine learning models268to configure (e.g., train, modify, update, fine-tune, etc.) the machine learning models268. The training manager244can control training by the model configuration processor264based on model tuning parameters in the model tuning database256, such as to control various hyperparameters for training. In various implementations, the system200can use the training management system240to configure the machine learning models268in a similar manner as described with reference to the second model116ofFIG.1, such as to train the machine learning models268using any of various data or combinations of data from the data repository204.

Application Session Management

FIG.3depicts an example of the system200, in which the system200can perform operations to implement at least one application session308for a client device304. For example, responsive to configuring the machine learning models268, the system200can generate data for presentation by the client device304(including generating data responsive to information received from the client device304) using the at least one application session308and the one or more machine learning models268.

The client device304can be a device of a user, such as a technician or building manager. The client device304can include any of various wireless or wired communication interfaces to communicate data with the model system260, such as to provide requests to the model system260indicative of data for the machine learning models268to generate, and to receive outputs from the model system260. The client device304can include various user input and output devices to facilitate receiving and presenting inputs and outputs.

In some implementations, the system200provides data to the client device304for the client device304to operate the at least one application session308. The application session308can include a session corresponding to any of the applications120described with reference toFIG.1. For example, the client device304can launch the application session308and provide an interface to request one or more prompts. Responsive to receiving the one or more prompts, the application session308can provide the one or more prompts as input to the machine learning model268. The machine learning model268can process the input to generate a completion, and provide the completion to the application session308to present via the client device304. In some implementations, the application session308can iteratively generate completions using the machine learning models268. For example, the machine learning models268can receive a first prompt from the application session308, determine a first completion based on the first prompt and provide the first completion to the application session308, receive a second prompt from the application308, determine a second completion based on the second prompt (which may include at least one of the first prompt or the first completion concatenated to the second prompt), and provide the second completion to the application session308.

In some implementations, the model system260includes at least one sessions database312. The sessions database312can maintain records of application session308implemented by client devices304. For example, the sessions database312can include records of prompts provided to the machine learning models268and completions generated by the machine learning models268. As described further with reference toFIG.4, the system200can use the data in the sessions database312to fine-tune or otherwise update the machine learning models268.

Completion Checking

In some implementations, the system200includes an accuracy checker316. The accuracy checker316can include one or more rules, heuristics, logic, policies, algorithms, functions, machine learning models, neural networks, scripts, or various combinations thereof to perform operations including evaluating performance criteria regarding the completions determined by the model system260. For example, the accuracy checker316can include at least one completion listener320. The completion listener320can receive the completions determined by the model system320(e.g., responsive to the completions being generated by the machine learning model268and/or by retrieving the completions from the sessions database312).

The accuracy checker316can include at least one completion evaluator324. The completion evaluator324can evaluate the completions (e.g., as received or retrieved by the completion listener320) according to various criteria. In some implementations, the completion evaluator324evaluates the completions by comparing the completions with corresponding data from the data repository204. For example, the completion evaluator324can identify data of the data repository204having similar text as the prompts and/or completions (e.g., using any of various natural language processing algorithms), and determine whether the data of the completions is within a range of expected data represented by the data of the data repository204.

In some implementations, the accuracy checker316can store an output from evaluating the completion (e.g., an indication of whether the completion satisfies the criteria) in an evaluation database328. For example, the accuracy checker316can assign the output (which may indicate at least one of a binary indication of whether the completion satisfied the criteria or an indication of a portion of the completion that did not satisfy the criteria) to the completion for storage in the evaluation database328, which can facilitate further training of the machine learning models268using the completions and output.

Feedback Training

FIG.4depicts an example of the system200that includes a feedback system400, such as a feedback aggregator. The feedback system400can include one or more rules, heuristics, logic, policies, algorithms, functions, machine learning models, neural networks, scripts, or various combinations thereof to perform operations including preparing data for updating and/or updating the machine learning models268using feedback corresponding to the application sessions308, such as feedback received as user input associated with outputs presented by the application sessions308. The feedback system400can incorporate features of the feedback repository124and/or feedback trainer128described with reference toFIG.1.

The feedback system400can receive feedback (e.g., from the client device304) in various formats. For example, the feedback can include any of text, speech, audio, image, and/or video data. The feedback can be associated (e.g., in a data structure generated by the application session308) with the outputs of the machine learning models268for which the feedback is provided. The feedback can be received or extracted from various forms of data, including external data sources such as manuals, service reports, or Wikipedia-type documentation.

In some implementations, the feedback system400includes a pre-processor400. The pre-processor400can perform any of various operations to modify the feedback for further processing. For example, the pre-processor400can incorporate features of, or be implemented by, the pre-processor232, such as to perform operations including filtering, compression, tokenizing, or translation operations (e.g., translation into a common language of the data of the data repository204).

The feedback system400can include a bias checker408. The bias checker408can evaluate the feedback using various bias criteria, and control inclusion of the feedback in a feedback database416(e.g., a feedback database416of the data repository204as depicted inFIG.4) according to the evaluation. The bias criteria can include, for example and without limitation, criteria regarding qualitative and/or quantitative differences between a range or statistic measure of the feedback relative to actual, expected, or validated values.

The feedback system400can include a feedback encoder412. The feedback encoder412can process the feedback (e.g., responsive to bias checking by the bias checker408) for inclusion in the feedback database416. For example, the feedback encoder412can encode the feedback as values corresponding to outputs scoring determined by the model system260while generating completions (e.g., where the feedback indicates that the completion presented via the application session308was acceptable, the feedback encoder412can encode the feedback by associating the feedback with the completion and assigning a relatively high score to the completion).

As indicated by the dashed arrows inFIG.4, the feedback can be used by the prompt management system228and training management system240to further update one or more machine learning models268. For example, the prompt management system228can retrieve at least one feedback (and corresponding prompt and completion data) from the feedback database416, and process the at least one feedback to determine a feedback prompt and feedback completion to provide to the training management system240(e.g., using pre-processor232and/or prompt generator236, and assigning a score corresponding to the feedback to the feedback completion). The training manager244can provide instructions to the model system260to update the machine learning models268using the feedback prompt and the feedback completion, such as to perform a fine-tuning process using the feedback prompt and the feedback completion. In some implementations, the training management system240performs a batch process of feedback-based fine tuning by using the prompt management system228to generate a plurality of feedback prompts and a plurality of feedback completion, and providing instructions to the model system260to perform the fine-tuning process using the plurality of feedback prompts and the plurality of feedback completions.

Data Filtering and Validation Systems

FIG.5depicts an example of the system200, where the system200can include one or more data filters500(e.g., data validators). The data filters500can include any one or more rules, heuristics, logic, policies, algorithms, functions, machine learning models, neural networks, scripts, or various combinations thereof to perform operations including modifying data processed by the system200and/or triggering alerts responsive to the data not satisfying corresponding criteria, such as thresholds for values of data. Various data filtering processes described with reference toFIG.5(as well asFIGS.6and7) can enable the system200to implement timely operations for improving the precision and/or accuracy of completions or other information generated by the system200(e.g., including improving the accuracy of feedback data used for fine-tuning the machine learning models268). The data filters500can allow for interactions between various algorithms, models, and computational processes.

For example, the data filters500can be used to evaluate data relative to thresholds relating to data including, for example and without limitation, acceptable data ranges, setpoints, temperatures, pressures, flow rates (e.g., mass flow rates), or vibration rates for an item of equipment. The threshold can include any of various thresholds, such as one or more of minimum, maximum, absolute, relative, fixed band, and/or floating band thresholds.

The data filters500can enable the system200to detect when data, such as prompts, completions, or other inputs and/or outputs of the system200, collide with thresholds that represent realistic behavior or operation or other limits of items of equipment. For example, the thresholds of the data filters500can correspond to values of data that are within feasible or recommended operating ranges. In some implementations, the system200determines or receives the thresholds using models or simulations of items of equipment, such as plant or equipment simulators, chiller models, HVAC-R models, refrigeration cycle models, etc. The system200can receive the thresholds as user input (e.g., from experts, technicians, or other users). The thresholds of the data filters500can be based on information from various data sources. The thresholds can include, for example and without limitation, thresholds based on information such as equipment limitations, safety margins, physics, expert teaching, etc. For example, the data filters500can include thresholds determined from various models, functions, or data structures (e.g., tables) representing physical properties and processes, such as physics of psychometrics, thermodynamics, and/or fluid dynamics information.

The system200can determine the thresholds using the feedback system400and/or the client device304, such as by providing a request for feedback that includes a request for a corresponding threshold associated with the completion and/or prompt presented by the application session308. For example, the system200can use the feedback to identify realistic thresholds, such as by using feedback regarding data generated by the machine learning models268for ranges, setpoints, and/or start-up or operating sequences regarding items of equipment (and which can thus be validated by human experts). In some implementations, the system200selectively requests feedback indicative of thresholds based on an identifier of a user of the application session308, such as to selectively request feedback from users having predetermined levels of expertise and/or assign weights to feedback according to criteria such as levels of expertise.

In some implementations, one or more data filters500correspond to a given setup. For example, the setup can represent a configuration of a corresponding item of equipment (e.g., configuration of a chiller, etc.). The data filters500can represent various thresholds or conditions with respect to values for the configuration, such as feasible or recommendation operating ranges for the values. In some implementations, one or more data filters500correspond to a given situation. For example, the situation can represent at least one of an operating mode or a condition of a corresponding item of equipment.

FIG.5depicts some examples of data (e.g., inputs, outputs, and/or data communicated between nodes of machine learning models268) to which the data filters500can be applied to evaluate data processed by the system200including various inputs and outputs of the system200and components thereof. This can include, for example and without limitation, filtering data such as data communicated between one or more of the data repository204, prompt management system228, training management system240, model system260, client device304, accuracy checker316, and/or feedback system400. For example, the data filters500(as well as validation system600described with reference toFIG.6and/or expert filter collision system700described with reference toFIG.7) can receive data outputted from a source (e.g., source component) of the system200for receipt by a destination (e.g., destination component) of the system200, and filter, modify, or otherwise process the outputted data prior to the system200providing the outputted data to the destination. The sources and destinations can include any of various combinations of components and systems of the system200.

The system200can perform various actions responsive to the processing of data by the data filters500. In some implementations, the system200can pass data to a destination without modifying the data (e.g., retaining a value of the data prior to evaluation by the data filter500) responsive to the data satisfying the criteria of the respective data filter(s)500. In some implementations, the system200can at least one of (i) modify the data or (ii) output an alert responsive to the data not satisfying the criteria of the respective data filter(s)500. For example, the system200can modify the data by modifying one or more values of the data to be within the criteria of the data filters500.

In some implementations, the system200modifies the data by causing the machine learning models268to regenerate the completion corresponding to the data (e.g., for up to a predetermined threshold number of regeneration attempts before triggering the alert). This can enable the data filters500and the system200selectively trigger alerts responsive to determining that the data (e.g., the collision between the data and the thresholds of the data filters500) may not be repairable by the machine learning model268aspects of the system200.

The system200can output the alert to the client device304. The system200can assign a flag corresponding to the alert to at least one of the prompt (e.g., in prompts database224) or the completion having the data that triggered the alert.

FIG.6depicts an example of the system200, in which a validation system600is coupled with one or more components of the system200, such as to process and/or modify data communicated between the components of the system200. For example, the validation system600can provide a validation interface for human users (e.g., expert supervisors, checkers) and/or expert systems (e.g., data validation systems that can implement processes analogous to those described with reference to the data filters500) to receive data of the system200and modify, validate, or otherwise process the data. For example, the validation system600can provide to human expert supervisors, human checkers, and/or expert systems various data of the system200, receive responses to the provided data indicating requested modifications to the data or validations of the data, and modify (or validate) the provided data according to the responses.

For example, the validation system600can receive data such as data retrieved from the data repository204, prompts outputted by the prompt management system228, completions outputted by the model system260, indications of accuracy outputted by the accuracy checker316, etc., and provide the received data to at least one of an expert system or a user interface. In some implementations, the validation system600receives a given item of data prior to the given item of data being processed by the model system260, such as to validate inputs to the machine learning models268prior to the inputs being processed by the machine learning models268to generate outputs, such as completions.

In some implementations, the validation system600validates data by at least one of (i) assigning a label (e.g., a flag, etc.) to the data indicating that the data is validated or (ii) passing the data to a destination without modifying the data. For example, responsive to receiving at least one of a user input (e.g., from a human validator/supervisor/expert) that the data is valid or an indication from an expert system that the data is valid, the validation system600can assign the label and/or provide the data to the destination.

The validation system600can selectively provide data from the system200to the validation interface responsive to operation of the data filters500. This can enable the validation system600to trigger validation of the data responsive to collision of the data with the criteria of the data filters500. For example, responsive to the data filters500determining that an item of data does not satisfy a corresponding criteria, the data filters500can provide the item of data to the validation system600. The data filters500can assign various labels to the item of data, such as indications of the values of the thresholds that the data filters500used to determine that the item of data did not satisfy the thresholds. Responsive to receiving the item of data from the data filters500, the validation system600can provide the item of data to the validation interface (e.g., to a user interface of client device304and/or application session308; for comparison with a model, simulation, algorithm, or other operation of an expert system) for validation. In some implementations, the validation system600can receive an indication that the item of data is valid (e.g., even if the item of data did not satisfy the criteria of the data filters500) and can provide the indication to the data filters500to cause the data filters500to at least partially modify the respective thresholds according to the indication.

In some implementations, the validation system600selectively retrieves data for validation where (i) the data is determined or outputted prior to use by the machine learning models268, such as data from the data repository204or the prompt management system228, or (ii) the data does not satisfy a respective data filter500that processes the data. This can enable the system200, the data filters500, and the validation system600to update the machine learning models268and other machine learning aspects (e.g., generative AI aspects) of the system200to more accurately generate data and completions (e.g., enabling the data filters500to generate alerts that are received by the human experts/expert systems that may be repairable by adjustments to one or more components of the system200).

FIG.7depicts an example of the system200, in which an expert filter collision system700(“expert system”700) can facilitate providing feedback and providing more accurate and/or precise data and completions to a user via the application session308. For example, the expert system700can interface with various points and/or data flows of the system200, as depicted inFIG.7, where the system200can provide data to the expert filter collision system700, such as to transmit the data to a user interface and/or present the data via a user interface of the expert filter collision system700that can accessed via an expert session708of a client device704. For example, via the expert session708, the expert session700can enable functions such as receiving inputs for a human expert to provide feedback to a user of the client device304; a human expert to guide the user through the data (e.g., completions) provided to the client device304, such as reports, insights, and action items; a human expert to review and/or provide feedback for revising insights, guidance, and recommendations before being presented by the application session308; a human expert to adjust and/or validate insights or recommendations before they are viewed or used for actions by the user; or various combinations thereof. In some implementations, the expert system700can use feedback received via the expert session as inputs to update the machine learning models268(e.g., to perform fine-tuning).

In some implementations, the expert system700retrieves data to be provided to the application session308, such as completions generated by the machine learning models268. The expert system700can present the data via the expert session708, such as to request feedback regarding the data from the client device704. For example, the expert system700can receive feedback regarding the data for modifying or validating the data (e.g., editing or validating completions). In some implementations, the expert system700requests at least one of an identifier or a credential of a user of the client device704prior to providing the data to the client device704and/or requesting feedback regarding the data from the expert session708. For example, the expert system700can request the feedback responsive to determining that the at least one of the identifier or the credential satisfies a target value for the data. This can allow the expert system708to selectively identify experts to use for monitoring and validating the data.

In some implementations, the expert system700facilitates a communication session regarding the data, between the application session308and the expert session708. For example, the expert session700, responsive to detecting presentation of the data via the application session308, can request feedback regarding the data (e.g., user input via the application session308for feedback regarding the data), and provide the feedback to the client device704to present via the expert session708. The expert session708can receive expert feedback regarding at least one of the data or the feedback from the user to provide to the application session308. In some implementations, the expert system700can facilitate any of various real-time or asynchronous messaging protocols between the application session308and expert session708regarding the data, such as any of text, speech, audio, image, and/or video communications or combinations thereof. This can allow the expert system700to provide a platform for a user receiving the data (e.g., customer or field technician) to receive expert feedback from a user of the client device704(e.g., expert technician). In some implementations, the expert system700stores a record of one or more messages or other communications between the sessions308,708in the data repository204to facilitate further configuration of the machine learning models268based on the interactions between the users of the sessions308,708.

Building Data Platforms and Digital Twin Architectures

Referring further toFIGS.1-7, various systems and methods described herein can be executed by and/or communicate with building data platforms, including data platforms of building management systems. For example, the data repository204can include or be coupled with one or more building data platforms, such as to ingest data from building data platforms and/or digital twins. The client device304can communicate with the system200via the building data platform, and can feedback, reports, and other data to the building data platform. In some implementations, the data repository204maintains building data platform-specific databases, such as to enable the system200to configure the machine learning models268on a building data platform-specific basis (or on an entity-specific basis using data from one or more building data platforms maintained by the entity).

For example, in some implementations, various data discussed herein may be stored in, retrieved from, or processed in the context of building data platforms and/or digital twins; processed at (e.g., processed using models executed at) a cloud or other off-premises computing system/device or group of systems/devices, an edge or other on-premises system/device or group of systems/devices, or a hybrid thereof in which some processing occurs off-premises and some occurs on-premises; and/or implemented using one or more gateways for communication and data management amongst various such systems/devices. In some such implementations, the building data platforms and/or digital twins may be provided within an infrastructure such as those described in U.S. patent application Ser. No. 17/134,661 filed Dec. 28, 2020, Ser. No. 18/080,360, filed Dec. 13, 2022, Ser. No. 17/537,046 filed Nov. 29, 2021, and Ser. No. 18/096,965, filed Jan. 13, 2023, and Indian patent application No. 202341008712, filed Feb. 10, 2023, the disclosures of which are incorporated herein by reference in their entireties.

III. Generative AI-Based Systems and Methods for Equipment Servicing

As described above, systems and methods in accordance with the present disclosure can use machine learning models, including LLMs and other generative AI models, to ingest data regarding building management systems and equipment in various unstructured and structured formats, and generate completions and other outputs targeted to provide useful information to users. Various systems and methods described herein can use machine learning models to support applications for presenting data with high accuracy and relevance.

Equipment Service Management Responsive to Fault Detection Using Machine Learning Models

FIG.8depicts an example of a method800. The method800can be performed using various devices and systems described herein, including but not limited to the systems100,200or one or more components thereof. Various aspects of the method800can be implemented using one or more devices or systems that are communicatively coupled with one another, including in client-server, cloud-based, or other networked architectures.

At805, a fault condition of an item of equipment can be detected. The fault condition can be detected responsive to manual and/or automated monitoring of various data sources regarding the item of equipment. In some implementations, the fault condition is detected responsive to an alarm notification from an alarm of the equipment or coupled with the equipment. For example, sensor data of the equipment or from a sensor directed to the equipment can be monitored by the alarm, and evaluated according to one or more alarm conditions (e.g., threshold values) to trigger the alarm notification. The fault condition can be detected responsive to user input indicative of the fault condition, or images or other data received indicative of the fault condition.

At810, the fault condition can be validated. For example, the fault condition can be validated to determine whether the alarm notification corresponds to a false alarm. In some implementations, the fault condition can be validated by verifying the data used to detect the fault condition at a second point in time (e.g., subsequent to a first point in time at which the fault condition was initially detected), such as by evaluating the one or more alarm conditions using data regarding the equipment at the second point in time; this may include using the same or different data than the data used to initially detect the fault condition to validate the fault condition. The fault condition can be validated by providing the alarm notification to a device of a user, and requesting a confirmation (or indication of false alarm) from the user via the device. Responsive to the fault condition being identified as a false alarm, the equipment can be continued to be monitored.

At815, a cause of the fault condition can be identified, such as by performing a root cause analysis. In some implementations, the cause is detected using a function that includes one or more algorithms, tables, simulations, or machine learning models described herein. For example, at least one of an identifier of the equipment, the fault condition, user text or speech identifying the fault condition (e.g., notes from any of a variety of entities, such as a facility manager, on-site technician, etc.), or data regarding the equipment used to detect the fault condition can be applied as input to the function to enable the function to determine an indication of a cause of the fault condition. For example, the function can include a table mapping various such inputs to one or more causes of fault conditions. The function can include a machine learning model configured using various forms of data described herein. For example, the machine learning model can include one or more classifiers, language models, or combinations thereof that are trained using data that includes information indicative of fault conditions and associated causes of fault conditions.

At820, a prescription is generated based on the cause of the fault condition. For example, one or more of the cause of the fault condition, the fault condition, and an identifier of the equipment can be provided to a language model to cause the language model to generate the prescription. The prescription can have a natural language format. The prescription can indicate one or more actions for a service technician to perform to verify, service, and/or repair the fault condition, such as instructions for tools and/or parts to use for the item of equipment. The language model can include any of various models described herein that are configured, using training data representative of prescriptions. The prescription can be generated for presentation using various output modalities, such as text, speech, audio, image, and/or video, including in real-time, conversational, or asynchronous formats.

In some implementations, generating the prescription includes conditioning or guiding the language model to generate the prescription based on a class of at least one of the service technician or the site at which the item of equipment is present. For example, the language model can have its configuration (e.g., training, etc.) modified according to labels of identifiers or classes of technicians, sites, types of equipment, or other characteristics relating to the item of equipment and/or the service technician, which can enable the prescription to be generated in a manner that is more accurate and/or relevant to the service to be performed.

At825, a warranty is evaluated based on one or more items (e.g., the equipment, parts or tools for servicing the equipment) identified by the prescription. For example, the warranty can be retrieved from various sources, such as a contract database associated with the entity that maintains the site, according to an identifier of the type of equipment, from the service request, or various combinations thereof. The prescription (or the service request) can be parsed to identify one or more items, such as items of equipment, identified by the prescription. For example, the item of equipment for which the service request is generated can be identified from the prescription, and compared with the warranty (e.g., using natural language processing algorithms, etc.) to identify one or more warranty conditions assigned to the item of equipment. The warranty conditions can indicate, for example, timing criteria for authorizing and/or payment for servicing the item of equipment by a vendor or supplier of the item of equipment. Responsive to the warranty conditions being satisfied (e.g., a termination of the warranty not being met), various actions can be performed to trigger servicing of the item of equipment. In some implementations, one or more warranty conditions are evaluated prior to, during, and or subsequent to generation of the prescription, such as to allow the prescription to be generated to incorporate one or more outputs of the evaluation of the warranty (or avoid computational resources for generating the prescription responsive to the warranty conditions not being satisfied).

At830, scheduling of deployment of at least one of a service technician or one or more parts identified by the prescription can be performed. In some implementations, the prescription can identify the service technician, such as to select the service technician from a plurality of candidate service technicians according to an expertise that the service technician is labeled with and which corresponds to the item of equipment. Scheduling deployment of the one or more parts can including identifying a provider of the one or more parts and assigning the one or more parts to a vehicle (e.g., trucks) for delivering the one or more parts to the site of the item of equipment. By using the language model to generate the prescription—which identifies the one or more parts—the one or more parts that are delivered to the site can be more accurately identified, which can reduce resource usage and/or wasted space or weight on the vehicle. In some implementations, scheduling deployment includes generating a service ticket indicative of the service to be performed, such as to identify the service technician, the parts, and/or the item of equipment.

Depending on the determined prescription, the scheduling can include automated servicing of the item of equipment, such as to provide commands to adjust parameters of the item of equipment to a controller of the item of equipment. The scheduling can include providing instructions for performing remote service, such as to provide instructions to a service technician to use on-site tools and/or parts, or manual adjustment of the item of equipment, to service the item of equipment (e.g., to avoid a truck deployment or truck roll to the site).

At835, an application session for a service operation corresponding to the service request (and the prescription) can be provided. In some implementations, the application session is provided via a device of the service technician. For example, the device can provide one or more credentials to access the application session (e.g., credentials that uniquely identify the service technician). The application session can present information to the service technician in any of various conversational, messaging, graphical, real-time, and/or asynchronous formats. The application session can receive one or more prompts from the device (e.g., from a user input device of the device), and provide the one or more prompts to the language model to cause the language model to provide corresponding completions responsive to the one or more prompts. For example, the device can receive text or image data (among other formats) as inputs provided by actions of the user (e.g., via an input interface of the device; by the user controlling a camera of the device), and provide the inputs as prompts to the language model. The application session can present the completions via the device to facilitate guiding the service technician through the actions to perform to service the item of equipment. In some implementations, the application session automatically (e.g., responsive to detecting a condition for escalating the guidance to a human expert) or manually (e.g., responsive to user input requesting guidance from a human expert) can establish a communication session between the device and a device of a human expert to provide further guidance to the service technician; the language model can provide various information such as the service request, prescription, and/or communications between the user and the language model via the application session to the device of the human expert, and can label various portions of the communications as potential causes of the escalation. The application session can be implemented as a virtual assistant, such as to provide information such as instruction manuals or technical reports regarding the item of equipment, responsive to requests from the service technician inputted at the device of the service technician.

At840, operation of the item of equipment can be updated responsive to one or more actions performed by the service technician. For example, various parameters of operation of the item of equipment, such as setpoints, can be updated according to the one or more actions.

In some implementations, information from the service request, prescription, and application session processes can be used to perform analytics regarding entities that maintain sites and items of equipment (e.g., to evaluate customer churn). For example, information including unstructured data (e.g., service reports) regarding items of equipment and entity engagement or disengagement (e.g., deals) can be correlated to identify patterns regarding ways that service can be performed to maintain or increase the likelihood of increasing performance of one or more items of equipment of the entity, completion of deals or of maintaining engagement with the entity.

Virtual Assistant for Service and Parts Recommendations for Chillers and/or Other Equipment.

Referring now toFIG.9, a flowchart of a method900is shown, according to some embodiments. The method900can be performed using various devices and systems described herein, including but not limited to the systems100,200or one or more components thereof. Various aspects of the method900can be implemented using one or more devices or systems that are communicatively coupled with one another, including in client-server, cloud-based, or other networked architectures.

At step902, at least one AI model is fine-tuned using building domain data including information regarding equipment types, equipment parameters, and output conditions. Step902can be executed according to disclosure above relating to updating of model104to provide model116based on data sources112, among other teachings herein. Fine-tuning the at least one AI model with building domain data provides the AI model with domain-specific intelligence that enables the AI model to handle domain-specific prompts and generate output with detailed, domain-specific recommendations and information, for example details of relevance to service technicians for servicing, installing, configuring, etc. building equipment. For example, where an initial AI model may be adapted to handle queries relating generally to building equipment (e.g., relating generally to chillers), a fine-tuned AI model output from step902may be adapted to handle queries relating to particular models of chillers while providing responses which are accurately-tailored a model identified in a query. At least one AI model fine-tuned for building domain applications is thereby provided as an output of step902. The at least one AI model can be a model as described in U.S. application Ser. No. 18/419,449 filed Jan. 22, 2024 and/or U.S. Application No. 18,633,068 filed Apr. 11, 2024, the entire disclosures of which are incorporated by reference herein.

At step904, an interactive interface is provided. The interactive interface is configured to guide input (by a user) of sufficient information for crafting an appropriate input query for the fine-tuned AI model. In some embodiments, such information includes a relevant equipment type (e.g., chiller, air handling unit, particular model of equipment, equipment serial number or unique identifier, etc.), a relevant equipment parameter (e.g., setting, setpoint, compressor frequency, chilled water setpoint, etc.), a relevant output condition (e.g., a measurable output condition, a supply water temperature, a return water temperature, an indoor air condition of a space served by the equipment, an energy consumption of the equipment, a resource consumption of the equipment, a resource generation or load served by the equipment, etc.), and a request type (e.g., a request for a service recommendation, a request for a root cause analysis, a request for a product recommendation such as a replacement part recommendation, a request for a recommendation for changing operating parameters, etc.).

In some embodiments, step904provides the interactive interface as a conversational interface (e.g., chat bot), for example using at least one generative AI model. Step904can include receiving textual and/or audio input from a user in a natural language format, identifying relevant information from the textual and/or audio input (e.g., identifying any equipment type, equipment parameter, output condition, or request type indicated by the input), identifying any missing information, and generating conversational feedback prompting the user to input the missing information (e.g., to input any of the equipment type, equipment parameter, output condition, or request type to the extent not included in the user's initial input).

For example, in some embodiments, responsive to a natural language (e.g., unstructured) user input, the at least one generative AI model can determine that the user has provided insufficient information for the at least generative AI model to fully identify the equipment type, equipment parameter, output condition, or other detail being referred to by the user's request. For example, a user may submit a natural language input that refers to a chiller, and step904can include receiving such an input, determining that the input is missing (e.g., lacks, could be further clarified by, etc.) details relating to the type of chiller (e.g., model, manufacturer, etc.). The at least one generative AI model can then prompt the user for such information, either directly (e.g., “What model of chiller are you referring to?”) or by asking questions to narrow down the set of possible types (e.g., “What color is the chiller?”; “How big is the chiller?”; etc.). Such conversational interactions by the at least one generative AI model can be generated in a manner informed by the model-fine-tuning, i.e., such that the at least one generative AI model is configured to prompt the user to provided sufficient information for the at least one generative AI model to determine that a user is referring to a particular category of information (e.g., equipment type, parameter, variable, etc.) on which the at least one generative AI model was fine-tuned.

Accordingly, conversational prompts can be generated and user inputs processed iteratively until the system has collected sufficient information for crafting an appropriate prompt, for example a relevant equipment type, a relevant equipment parameter, a relevant output condition, and a request type as shown inFIG.9(with the present disclosure that other combinations of information may be gathered in step904as may be suitable for prompt generation in various use cases). In this context, “relevant” refers to the equipment type, equipment parameter, or output condition intended to be referred to by the user and on which the generative AI model has been fine-tuned, i.e., relevant to both (or either in some embodiments) the user and to the at least one fine-tuned generative AI model. In this respect, the at least one generative AI model may be configured to inquire with the user (e.g., via a chatbot interaction), until the user and the chatbot are in agreement with respect to which equipment type, equipment parameter, and/or output condition is/are relevant to addressing a user's query. In a scenario where such a resolution cannot be reached (e.g., where the user attempts to ask about an equipment type on which the at least one generative AI model has not been fine-tuned), the at least one generative AI model can output an indication to the user that a relevant input cannot be determined in step904or that a response to the user's inquiry may not be reliable in that scenario.

In some embodiments, step904provides the interactive interface as a structured input interface. For example, an input field can be provided for each of a set of requested input information, for example for each of equipment type, an equipment parameter, output condition, and request type. In some embodiments, the interactive interface includes drop-down menus or other guided selection widgets for accepting user selection of relevant types, parameters, conditions, requests, etc., to at least partially define the user query via structured input (while a remainder of the user query may be provided via unstructured natural language). In some embodiments, the drop-down menu or other list is automatically populated by the at least one generative AI model to reflect options (e.g., types of equipment, parameters, variables, output conditions) for which the at least one generative AI model has been sufficiently fine-tuned to generate results to a user query associated with such options. Such a list can be generated by prompting the at least one generative AI model to describe the list of options (e.g., list of equipment types) for which the at least one generative AI-model has been fine-tuned. Process904can thereby self-enforce selection of options for which the at least one generative AI-model has been fine-tuned to provide reliable and/or accurate responses to user queries.

In some embodiments, the drop-down menu or other list can be generated or filtered based on equipment, points, sensors, etc. available in a building management system or connected equipment system associated with the user. For example, if a user has an account tied to a connected equipment platform that manages a set of chillers, the options selectable in the interactive interface can be automatically adapted to correspond to the types of chillers and related parameters, conditions, etc. corresponding to the set of chillers associated with the user's account (e.g., filtered to include those on which the at least one generative AI model was fine-tuned) Step904can thereby be based on data indicating equipment, sensors, devices, points, features, etc. of a building management system or connected equipment platform associated with the user, to facilitate selection of relevant equipment type or other relevant option for inclusion in a query for the at least one generative AI model to facilitate responsiveness to a user request.

In some embodiments, the interface provided itself allows unstructured input, while at least one AI model operates to determine which structured inputs correspond to the user's unstructured inputs (e.g., the structured inputs being the relevant equipment type, relevant equipment parameter, relevant output condition, etc. referred to above in some embodiments). Such a process can involve allowing free-text inputs, and then associated the free-text input with an option from a list generated as described above (e.g., from the equipment associated with a user account; from fine-tuning of the at least one generative AI model). Such a process can enforce selection of relevant equipment type, relevant equipment parameter, relevant output condition, etc. during user interaction with the interactive interface provided

At step906, an input query is generated based on the user inputs collected in step904, for example based on the input of the relevant equipment type, the relevant equipment parameter, the relevant output condition, and the request type as shown in the example ofFIG.9. In some embodiments, the input query can be generated by selecting a prompt template (e.g., based on the request type) and filling in the prompt template with the information collected from the user in step904. In some embodiments, the input query is generated using a machine learning model, for example using first generative AI model trained to generate a prompt predicted to cause a second generative AI model (or other large language model) to provide a relevant, accurate, coherent, etc. result. Step906can include providing an input query that is appropriately tuned to exploit the domain-specific adaptation of the fine-tuned model output by step902. Synergies between input query generation in step906and model fine-tuning in step902can be used to provide improved reliability, relevancy, specificity, precision, etc. of outputs of the AI models contemplated herein, in various embodiments. For example, because the user inputs collected in step904are collected so as to reflect relevant details relating to content on which the at least one AI model is fine-tuned in step902, as described above, the input query can be populated in step906such that the input query requests information that the at least one AI model has be fine-tuned to provide. In some embodiments, the input query generated in step906matches a format of a query in a data pair used in model finetuning, for example following a generative fine-tuning approach as described in U.S. Provisional Application No. 63/470,754 filed Jun. 2, 2023, the entire disclosure of which is incorporated by reference herein.

At step908, the input query from step906is provided as an input to at least one AI model (e.g., the fine-tuned AI model from step902). Step908can include prompting the fine-tuned AI model (e.g., model116) to generate response to the input query.

At step910, a recommendation or other response is output from the at least one AI model (e.g., from the fine-tuned AI model from step902) responsive to the input query. The recommendation or other response can provide different information depending on the request type obtained from the user, and can relate to the other domain-specific details input by the user (e.g., the equipment type, the parameter of interest, the output condition of interest, etc.). In some scenarios, the output provides a recommendation for a service action to be performed on the identified equipment. In some scenarios, the output provides a control action that can be taken with respect to the equipment (e.g., an adjustment of operating setpoints, etc.). In some scenarios, the output provides a description of a cause of an issue (e.g., fault, etc.) indicated in the user's query. In some embodiments, process900includes executing an automated action to resolve an identified issue or to implement a recommendation generated in step910(e.g., causing maintenance to be performed, changing operation of equipment in accordance with a recommended change, causing execution of a procedure to further test equipment status, providing a self-healing operation of equipment to resolve a fault condition, etc.). Step910can be executed by providing and using a response according to teachings of U.S. application Ser. No. 18/419,449 filed Jan. 22, 2024; Ser. No. 18/419,442 filed Jan. 22, 2024; Ser. No. 18/419,456 filed Jan. 22, 2024; Ser. No. 18/633,024 filed Apr. 11, 2024; Ser. No. 18/633,040 filed Apr. 11, 2024; Ser. No. 18/633,049 filed Apr. 11, 2024; Ser. No. 18/633,068 filed Apr. 11, 2024; and/or Ser. No. 18/633,086, the entire disclosures of which are incorporated by reference herein.

Process900can thereby provide a virtual assistant for evaluating and resolving building-domain-specific (e.g., chiller-specific, etc.) user questions and queries by generating prompts according to domain-specific information types and providing such prompts to at least one AI model fine-tuned with domain-specific intelligence such that relevant, precise, and accurate domain-specific recommendations and interventions can be provided. Such an approach can eliminate, for example, usage of computational resources and time delays that may otherwise be associated with a user experimenting with different prompts to the fine-tuned AI model before arriving at a prompt that provides a sufficient result and the numerous executions of the fine-tuned AI model required to provide such an approach. The approach of process900can also increase user confidence in results of process900as compared to an approach without step904-906because, rather than presenting the user with a poor recommendation or response when the user inputs insufficient information, the process first prompts the user for the sufficient information and thereby provides a quality, reliable response. Various such technical advantages are provided by the teachings herein.

Referring now toFIG.10, a block diagram of a system1000is shown, according to some embodiments. The system1000is shown as including a chiller1002(or other type of equipment in other embodiments), at least one additional chiller or other equipment1003(e.g., such that the system1000includes multiple chillers, a set of chillers, a set of building plant equipment, a set of HVAC equipment, a set of building devices, a set of plant devices, etc.), a remote processing system1004communicable with the chiller1002and the at least one additional chiller or other equipment1003, and a remote user interface device1006(e.g., a personal computing device, laptop, smartphone, tablet, headset, etc. configured to display a graphical user interface) communicable with the remote processing system1004. The chiller1002is shown as including a local processing system1008and a local user interface1010(e.g., touchscreen, display screen and keyboard or other input device, etc., for example mechanically coupled to the chiller1002). The at least one additional chiller or other equipment1003can also include a local processing system and local user interface.

The various features and functionality disclosed herein can be executed on and by the local processing system1008and/or the remote processing system1004and/or distributed between such systems in any combination or divisional of processing and memory steps to enable the features described above. For example, process900as shown inFIG.9can be executed by one, both, or a combination of the local processing system1008and the remote processing system1004. In some embodiments model fine-tuning and hosting of the at least one AI-model (e.g., in step902and for use in steps908and910) are provided by the remote processing system1004while providing of an interactive interface to guide user inputs and/or generation of a user query based on such inputs is performed by the local processing system1008using the local user interface1010(e.g., steps904and906). In some embodiments, user interactions as described for process900are provided via the remote user interface device1006. The system1000can thereby provide for execution of process900(and/or various other processes described herein) for a chiller1002and/or multiple chillers and/or other equipment.

In various implementations, the steps and operations described herein may be performed on one processor or in a combination of two or more processors. For example, in some implementations, the various operations could be performed in a central server or set of central servers configured to receive data from one or more devices (e.g., edge computing devices/controllers) and perform the operations. In some implementations, the operations may be performed by one or more local controllers or computing devices (e.g., edge devices), such as controllers dedicated to and/or located within a particular building or portion of a building. In some implementations, the operations may be performed by a combination of one or more central or offsite computing devices/servers and one or more local controllers/computing devices. All such implementations are contemplated within the scope of the present disclosure. Further, unless otherwise indicated, when the present disclosure refers to one or more computer-readable storage media and/or one or more controllers, such computer-readable storage media and/or one or more controllers may be implemented as one or more central servers, one or more local controllers or computing devices (e.g., edge devices), any combination thereof, or any other combination of storage media and/or controllers regardless of the location of such devices.