Real-time monitoring of machine learning models in service orchestration plane

A computer-implemented method, system and computer program product for performing real-time monitoring of machine learning models. Real-time model state data and metadata (e.g., operating dataset) of the machine learning models located within an orchestration plane of a network are collected by agents located within the machine learning models. The portion of the collected real-time model state data and metadata that is to be provided to the user by the service orchestrator (configured to monitor the machine learning models in the service orchestration plane via the use of agents in the machine learning models) is selected and marked. The marked collected real-time model state data and metadata are then provided to the user by the service orchestrator. In this manner, real-time monitoring of the machine learning models in the orchestration plane, such as the service orchestration plane, of a broadband cellular network (e.g., fifth generation broadband cellular network) is achieved.

TECHNICAL FIELD

The present disclosure relates generally to machine learning models, and more particularly to real-time monitoring of machine learning models in a service orchestration plane of a network, such as a broadband cellular network (e.g., fifth generation broadband cellular network).

BACKGROUND

Machine learning is the study of computer algorithms that improve automatically through experience and by the use of data. Machine learning algorithms build a model (“machine learning model”) based on sample data, known as “training data,” in order to make predictions or decisions without being explicitly programmed to do so. Machine learning algorithms are used in a wide variety of applications, such as email filtering and computer vision, where it is difficult to develop conventional algorithms to perform the needed tasks.

A machine learning model is a file that has been trained to recognize certain types of patterns. A machine learning model may be trained over a set of data, providing it an algorithm that it can use to reason over and learn from these data. There are various types of machine learning models that use a variety of machine learning algorithms, such as linear regression, logistic regression, decision tree, support-vector machine (SVM), Naive Bayes, k-nearest neighbors algorithm (k-NN), k-means clustering, random forest, etc.

SUMMARY

In one embodiment of the present disclosure, a computer-implemented method for performing real-time monitoring of machine learning models comprises collecting real-time model state data and metadata of the machine learning models located within an orchestration plane of a network by agents located within the machine learning models. The method further comprises marking a portion of the collected real-time model state data and metadata that was selected to be provided to a user of a computing device. The method additionally comprises providing the marked collected real-time model state data and metadata to the user of the computing device.

Other forms of the embodiment of the computer-implemented method described above are in a system and in a computer program product.

The foregoing has outlined rather generally the features and technical advantages of one or more embodiments of the present disclosure in order that the detailed description of the present disclosure that follows may be better understood. Additional features and advantages of the present disclosure will be described hereinafter which may form the subject of the claims of the present disclosure.

DETAILED DESCRIPTION

As stated in the Background section, machine learning is the study of computer algorithms that improve automatically through experience and by the use of data. Machine learning algorithms build a model (“machine learning model”) based on sample data, known as “training data,” in order to make predictions or decisions without being explicitly programmed to do so. Machine learning algorithms are used in a wide variety of applications, such as email filtering and computer vision, where it is difficult to develop conventional algorithms to perform the needed tasks.

A machine learning model is a file that has been trained to recognize certain types of patterns. A machine learning model may be trained over a set of data, providing it an algorithm that it can use to reason over and learn from these data. There are various types of machine learning models that use a variety of machine learning algorithms, such as linear regression, logistic regression, decision tree, support-vector machine (SVM), Naive Bayes, k-nearest neighbors algorithm (k-NN), k-means clustering, random forest, etc.

In a cognitive system, which uses cognitive computing, natural language processing and machine learning to enable people and machines to interact more naturally to extend and magnify human expertise and cognition, there could be many different machine learning models with different functions and operation feature sets to produce different outcomes. By having a variety of machine learning models, the cognitive system is enhanced in its ability to enable people and machines to interact more naturally.

Such a cognitive system may utilize a broadband cellular network, such as the fifth generation technology standard for the broadband cellular network (“5G”). In such an architecture, dissimilar machine learning models reside in the service orchestration plane. The service orchestration plane introduces a parent level of abstraction that alleviates the need for other services to manage interaction details required to ensure that service operations are executed in a specific sequence.

As discussed above, in the service orchestration plane, there are multiple dissimilar machine learning models, each operating with different training datasets, using different algorithms to train the model using the different training datasets. Once the machine learning model is selected to be utilized, the outcome of the machine learning model needs to be evaluated in real-time as to the accuracy of the prediction, including in situations involving automated audits of the machine learning model. Unfortunately, there is not currently a means for performing real-time monitoring of the machine learning models in the service orchestration plane. As a result, the performance of such machine learning models in the service orchestration plane may be unknown.

The embodiments of the present disclosure provide a means for performing real-time monitoring of the machine learning models in the service orchestration plane.

In some embodiments of the present disclosure, the present disclosure comprises a computer-implemented method, system and computer program product for performing real-time monitoring of machine learning models. In one embodiment of the present disclosure, real-time model state data and metadata (e.g., type of machine learning model, an operating dataset, features of a quorum configuration, attributes of the machine learning models, etc.) of the machine learning models located within an orchestration plane of a network are collected by agents located within the machine learning models. In one embodiment, such agents utilize the simple network management protocol (SNMP) and are referred to herein as the “SNMP agents.” The portion of the collected model state data and metadata that is to be provided to the user by the service orchestrator (configured to monitor the machine learning models in the service orchestration plane via the use of agents in the machine learning models) is selected and marked. In one embodiment, such information (model state data and metadata) may be selected based on input received from an expert. In one embodiment, such information (model state data and metadata) may be selected based on prior information previously captured by the SNMP agents. In one embodiment, the selected information (model state data and metadata) to be provided to the user by the service orchestrator is marked by setting a value to a flag associated with such information. The marked collected real-time model state data and metadata are then provided to the user by the service orchestrator. In this manner, real-time monitoring of the machine learning models in the orchestration plane, such as the service orchestration plane, of a broadband cellular network (e.g., fifth generation broadband cellular network) is achieved.

In the following description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without such specific details. In other instances, well-known circuits have been shown in block diagram form in order not to obscure the present disclosure in unnecessary detail. For the most part, details considering timing considerations and the like have been omitted inasmuch as such details are not necessary to obtain a complete understanding of the present disclosure and are within the skills of persons of ordinary skill in the relevant art.

Referring now to the Figures in detail,FIG.1illustrates an embodiment of the present disclosure of a communication system100for practicing the principles of the present disclosure. Communication system100includes computing devices101A-101C (identified as “Computing Device A,” “Computing Device B,” and “Computing Device C,” respectively, inFIG.1) connected to a server102via a network103. Computing devices101A-101C may collectively or individually be referred to as computing devices101or computing device101, respectively. It is noted that both computing devices101and the users of computing devices101may be identified with element number101.

Computing device101may be any type of computing device (e.g., portable computing unit, Personal Digital Assistant (PDA), laptop computer, mobile device, tablet personal computer, smartphone, mobile phone, navigation device, gaming unit, and the like) configured with the capability of connecting to network103and consequently communicating with other computing devices101and server102.

Network103may be, for example, a broadband cellular network, such as the fifth generation (5G) broadband cellular network.

Server102, as used herein, may be computer hardware or software that provides functionality for other programs or devices. In one embodiment, server102provides various functionalities, often called “services,” such as sharing data or resources among multiple computing devices101or performing computation for a computing device101.

In one embodiment, server102is a web server configured to offer a social networking and/or microblogging service thereby enabling users of computing devices101to send and read other users' posts. “Posts,” as used herein, include any one or more of the following: text (e.g., comments, sub-comments and replies), audio, video images, etc.

In one embodiment, server102is configured to host websites (website is a collection of relevant webpages that is addressed to a Uniform Resource Locator (URL)) and serve contents to the World Wide Web. For example, server102may host a website in which its collection of relevant webpages are accessed by a user of computing device101, such as via a web browser (software application for accessing information on the World Wide Web) on computing device101. Furthermore, server102is configured to process incoming network requests over HTTP (Hypertext Transfer Protocol) and several other related protocols.

In one embodiment, as shown inFIG.1, broadband cellular network103, such as the 5G broadband cellular network, has an infrastructure of a radio access network (RAN)104, which consists of various types of facilities, including small cells, towers, masts and dedicated in-building and home systems that connect mobile users and wireless devices to the main core network. Furthermore, as shown inFIG.1, network103further includes the infrastructure of a core network105, which is the mobile exchange and data network that manages all of the mobile voice, data and internet connections. In one embodiment, core network105includes distributed servers across the network.

In one embodiment, core network105includes a device, referred to herein as the “service orchestrator”106, configured to monitor the machine learning models in the service orchestration plane of network103. In one embodiment, service orchestrator106is configured to perform orchestration on network103. “Orchestration,” as used herein, refers to the automated configuration, management and coordination of computer systems, applications and services. In one embodiment, in connection with such orchestration, service orchestrator106includes a service orchestration plane that includes multiple dissimilar machine learning models, each operating with different training datasets, using different algorithms to train the model using the different training datasets. The service orchestration plane introduces a parent level of abstraction that alleviates the need for other services to manage interaction details required to ensure that service operations are executed in a specific sequence. A further discussion regarding the service orchestration plane is provided further below in connection withFIG.2.

In one embodiment, service orchestrator106is configured to monitor the machine learning models in the service orchestration plane of network103via the use of SNMP (simple network management protocol) agents in the machine learning models as discussed in further detail below. A description of the hardware configuration of service orchestrator106is provided further below in connection withFIG.3.

System100is not to be limited in scope to any one particular network architecture. System100may include any number of computing devices101, servers102, networks103, RANs104, core networks105and service orchestrators106.

A discussion regarding the service orchestration plane is provided below in connection withFIG.2.

FIG.2is a diagram of the software components of the service orchestration plane200in accordance with an embodiment of the present disclosure.

Referring toFIG.2, service orchestration plane200includes machine learning models (MLMs)201A-201C (identified as “Machine Learning Model1,” “Machine Learning Model2,” and “Machine Learning Model3,” respectively, inFIG.2). Machine learning models201A-201C may collectively or individually be referred to as machine learning models201or machine learning model201, respectively. A machine learning model201, as used herein, is a file that has been trained to recognize certain types of patterns. In one embodiment, machine learning models201are hosted on different environments (e.g., edge cloud, core cloud locations over a virtual machine infrastructure).

In one embodiment, service orchestration plane200includes dissimilar machine learning models201, each operating with different training datasets, using different algorithms to train the model using the different training datasets. In one embodiment, the performance of such machine learning models201A-201C is monitored using agents, referred to herein as the simple network management protocol (SNMP) agents202A-202C, respectively (identified as “SNMP Agent1,” “SNMP Agent2,” and “SNMP Agent3,” respectively, inFIG.2). SNMP agents202A-202C may collectively or individually be referred to as SNMP agents202or SNMP agent202, respectively. SNMP, as used herein, refers to an Internet Standard protocol for collecting and organizing information about managed devices on IP networks and for modifying that information to change device behavior. In one embodiment, such a protocol is utilized by agents202. As a result, such agents are referred to herein as “SNMP agents.”

In one embodiment, SNMP agents202are configured to collect real-time model state data and metadata of machine learning models201. In one embodiment, each agent202(e.g., SNMP agent202A) residing within machine learning model201(e.g., machine learning model201A) collects real-time model state data and metadata for that machine learning model201. For example, SNMP agent202A collects real-time model state data and metadata for machine learning model201A. In another example, SNMP agent202B collects real-time model state data and metadata for machine learning model201B. In a further example, SNMP agent202C collects real-time model state data and metadata for machine learning model201C. WhileFIG.2illustrates three machine learning models201and three SNMP agents202, it is noted that service orchestration plane200may include any number of machine learning models201and SNMP agents202.

In one embodiment, the collected real-time model state data and metadata include the type of machine learning model (e.g., binary classification, multiclass classification and regression), an operating dataset, features of a quorum configuration, attributes of the machine learning models, etc. A quorum configuration, as used herein, refers to a cluster of physical servers that should be active at any given time. Features of such a quorum configuration may include the particular physical services that are currently active. Furthermore, examples of attributes of the machine learning models, include, but not limited to, modeling tasks, predictions, algorithm used (e.g., decision tree, random forest, k-nearest neighbors (k-NN), etc.), errors, input data attributes, etc.

In one embodiment, SNMP agents202collect such information using Internet of Things (IoT) sensors embedded with software for the purpose of monitoring and collecting information (e.g., features of a quorum configuration, attributes of the machine learning models) and exchanging such data with SNMP agents202.

In one embodiment, SNMP agents202collect such information via log files that are generated by machine learning models201. A log file, as used herein, refers to a file that records events that occur in machine learning models201, such as modeling tasks, predictions, errors, etc.

In one embodiment, SNMP agents202collect metadata (data about data) generated by its associated machine learning model201, such as features and model functions used as input, settings and other inputs used, performance of the training, test and validation, type and amount of resources required to train, type of model, operating data set, version of data set, etc.

Referring again toFIG.2, service orchestration plane200further includes a master service203configured to connect SNMP agents202with services.

Furthermore, as shown inFIG.2, service orchestration plane200includes an SNMP manager204configured to manage SNMP agents202in terms of when to collect real-time model state data and metadata of their associated machine learning models201as well as when to send such collected information that is marked to be provided to SNMP manager204.

A further description of these and other functions is provided below in connection with the discussion of the method for performing real-time monitoring of machine learning models201(FIG.2) residing within service orchestration plane200(FIG.2) of a broadband cellular network (e.g., network103ofFIG.1).

Prior to the discussion of the method for performing real-time monitoring of machine learning models201residing within service orchestration plane200of broadband cellular network103, a description of the hardware configuration of service orchestrator106(FIG.1) is provided below in connection withFIG.3.

Referring now toFIG.3,FIG.3illustrates an embodiment of the present disclosure of the hardware configuration of service orchestrator106(FIG.1) which is representative of a hardware environment for practicing the present disclosure.

Service orchestrator106has a processor301connected to various other components by system bus302. An operating system303runs on processor301and provides control and coordinates the functions of the various components ofFIG.3. An application304in accordance with the principles of the present disclosure runs in conjunction with operating system303and provides calls to operating system303where the calls implement the various functions or services to be performed by application304. Application304may include, for example, a program for performing real-time monitoring of machine learning models201(FIG.2) as discussed further below in connection withFIGS.4-7.

Referring again toFIG.3, read-only memory (“ROM”)305is connected to system bus302and includes a basic input/output system (“BIOS”) that controls certain basic functions of service orchestrator106. Random access memory (“RAM”)306and disk adapter307are also connected to system bus302. It should be noted that software components including operating system303and application304may be loaded into RAM306, which may be service orchestrator's106main memory for execution. Disk adapter307may be an integrated drive electronics (“IDE”) adapter that communicates with a disk unit308, e.g., disk drive. It is noted that the program for performing real-time monitoring of machine learning models201, as discussed further below in connection withFIGS.4-7, may reside in disk unit308or in application304.

Service orchestrator106may further include a communications adapter309connected to bus302. Communications adapter309interconnects bus302with an outside network (e.g., network103ofFIG.1) to communicate with other devices, such as computing devices101, etc.

In one embodiment, application304of service orchestrator106includes the software components of service orchestration plane200. The functions discussed above performed by such components are not generic computer functions. As a result, service orchestrator106is a particular machine that is the result of implementing specific, non-generic computer functions.

In one embodiment, the functionality of such software components of service orchestrator106, including the functionality for performing real-time monitoring of machine learning models, may be embodied in an application specific integrated circuit.

As stated above, in a cognitive system, which uses cognitive computing, natural language processing and machine learning to enable people and machines to interact more naturally to extend and magnify human expertise and cognition, there could be many different machine learning models with different functions and operation feature sets to produce different outcomes. By having a variety of machine learning models, the cognitive system is enhanced in its ability to enable people and machines to interact more naturally. Such a cognitive system may utilize a broadband cellular network, such as the fifth generation technology standard for the broadband cellular network (“5G”). In such an architecture, dissimilar machine learning models reside in the service orchestration plane. The service orchestration plane introduces a parent level of abstraction that alleviates the need for other services to manage interaction details required to ensure that service operations are executed in a specific sequence. As discussed above, in the service orchestration plane, there are multiple dissimilar machine learning models in the service orchestration plane, each operating with different training datasets, using different algorithms to train the model using the different training datasets. Once the machine learning model is selected to be utilized, the outcome of the machine learning model needs to be evaluated in real-time as to the accuracy of the prediction, including in situations involving automated audits of the machine learning model. Unfortunately, there is not currently a means for performing real-time monitoring of the machine learning models in the service orchestration plane. As a result, the performance of such machine learning models in the service orchestration plane may be unknown.

The embodiments of the present disclosure provide a means for monitoring the performance of machine learning models in the service orchestration plane of a broadband cellular network (e.g., fifth generation broadband cellular network) using agents (e.g., SNMP agents) within the machine learning models to collect real-time model state data and metadata of the machine learning models as discussed below in connection withFIGS.4-7.FIG.4is a flowchart of a method for performing real-time monitoring of the machine learning models residing within the service orchestration plane of a broadband cellular network.FIG.5is a flowchart of a method for providing the marked collected real-time model state data and metadata to the simple network management protocol (SNMP) manager.FIG.6is a flowchart of an alternative method for providing the marked collected real-time model state data and metadata to the SNMP manager.FIG.7is a flowchart of a further alternative method for providing the marked collected real-time model state data and metadata to the SNMP manager.

As stated above,FIG.4is a flowchart of a method400for performing real-time monitoring of the machine learning models201(FIG.2) residing within the service orchestration plane200(FIG.2) of a broadband cellular network (e.g., network103ofFIG.1) in accordance with an embodiment of the present disclosure.

Referring toFIG.4, in conjunction withFIGS.1-3, in step401, SNMP agents201are initialized, such as by initializing itself or by having master service203initialize SNMP agents201. In one embodiment, in its initialization, SNMP agents201load the necessary variables and configuration settings for collecting real-time model state data and metadata of machine learning models201.

In step402, SNMP manager204is initialized. In one embodiment, SNMP manager204is initialized by master service203. In one embodiment, in its initialization, SNMP manager204initiates all configuration parsing approaches and performs service authentication to the subscribed monitoring services. Furthermore, in its initialization, metadata mappers and data structures are loaded. Metadata mappers, as used herein, refer to the software components used in the process of metadata mapping, which is a way of associating equivalent metadata values or fields from one system with content in another system. A data structure, as used herein, is a data organization, management, and storage format that enables efficient access and modification.

In step403, SNMP agents202connect with the services. In one embodiment, SNMP agents202are connected with the services by master service203. In one embodiment, SNMP agents202connect with the services in case the machine learning models201are in a subscription list of SNMP-based monitoring.

In step404, SNMP agents202access the model functions in machine learning models201. Such “model functions,” as used herein, refer to the core set of functions used for making a prediction based on the input received by machine learning model201. Such model functions are utilized by an algorithm (machine learning algorithm) to generate an output based on the input received by machine learning model201. In one embodiment, SNMP agent202accesses the model functions in machine learning model201after machine learning model201grants permission to SNMP agent202to access such model functions. In one embodiment, by accessing such model functions, SNMP agent202is able to collect real-time model state data and metadata of machine learning model201, such as errors, predictions, etc.

In step405, SNMP agents202initiate a polling thread along with discovery response management interfaces for upcoming requests. A “polling thread,” as used herein, refers to a sequential flow of instructions to repeatedly determine if a signal or request, such as a request to begin monitoring machine learning model201to capture its real-time model state data and metadata, has been issued. A “discovery response management interface,” as used herein, refers to an interface, such as with SNMP manager204, that is used for SNMP manager204to issue requests for SNMP agents202to begin collecting the machine learning model's real-time model state data and metadata.

In step406, SNMP agents202collect real-time model state data and metadata of machine learning models201within service orchestration plane200. As discussed above, such real-time model state data and metadata may include the type of machine learning model, an operating dataset, features of a quorum configuration, attributes of the machine learning models, etc. In one embodiment, SNMP agents202utilize a monitor function to target information about internal events, machine learning model decision point values, multi-pass information, metadata, etc. In one embodiment, such information may only be collected by SNMP agents202if its associated flag is enabled. For example, if the value of the flag is the logical value of one, then the information associated with such a flag may be collected. Otherwise, such information may not be collected.

For each of the machine learning models201in which real-time model state data and metadata were collected, the following steps (407-409) occurs.

In step407, machine learning model201selects which model state data and metadata out of the collected real-time model state data and metadata are eligible to be provided to SNMP manager204(and ultimately to the user of computing device101), such as via SNMP GET/GETNEXT/GETBULK/SET or TRAP operations. That is, machine learning model201selects a portion of the collected real-time model state data and metadata to be provided to SNMP manager204(and ultimately to the user of computing device101). In one embodiment, such information (model state data and metadata) may be selected based on input received from an expert. In one embodiment, such information (model state data and metadata) may be selected based on prior information previously captured by SNMP agents202and provided to SNMP manager204.

In step408, the collected real-time model state data and metadata that are to be provided to SNMP manager204(i.e., the real-time model state data and metadata selected in step407) are marked by SNMP agent202. In one embodiment, such information is marked by setting a value to a flag associated with such information. For example, if the value of the flag is the logical value of one, then the information associated with such a flag may be marked to be provided to SNMP manager204.

In step409, the marked collected real-time model state data and metadata are provided to SNMP manager204, and ultimately to the user of computing device101, such as in one of the following ways as discussed inFIGS.5-7.

FIG.5is a flowchart of a method500for providing the marked collected real-time model state data and metadata to SNMP manager204in accordance with an embodiment of the present disclosure.

Referring toFIG.5, in conjunction withFIGS.1-4, in step501, machine learning models201discover SNMP manager204. In one embodiment, machine learning models201discover SNMP manager204via the TRAP operation. A “TRAP” operation, as used herein, is an alert message sent from machine learning model201to a central collector, SNMP manager204.

In step502, upon discovering the COMPLETE signal, SNMP agents202send the marked collected real-time model state data and metadata to SNMP manager204. In one embodiment, the COMPLETE signal is issued by SNMP manager204to SNMP agents202(those agents of machine learning models201which discovered SNMP manager204) to begin sending the marked collected real-time model state data and metadata to SNMP manager204after SNMP manager204receives an indication from machine learning model201of being discovered.

In step503, SNMP manager204transfers the received marked collected real-time model state data and metadata to the user of computing device101, such as on a demand basis. In one embodiment, such information is in the form of tuples.

An alternative method for providing the marked collected real-time model state data and metadata to SNMP manager204is discussed below in connection withFIG.6.

FIG.6is a flowchart of an alternative method600for providing the marked collected real-time model state data and metadata to SNMP manager204in accordance with an embodiment of the present disclosure.

Referring toFIG.6, in conjunction withFIGS.1-4, in step601, SNMP manager204sends a request to SNMP agents202for the marked collected real-time model state data and metadata. In one embodiment, such a request is issued by SNMP manger204via the SNMP GETNEXT operation.

In step602, SNMP manager204receives the marked collected real-time model state data and metadata from SNMP agents202. In one embodiment, such information is sent to SNMP manager204by SNMP agents202over the medium access control (MAC) interface with SNMP manager204as a protocol data unit (PDU). In one embodiment, SNMP manager204is identified via its universally unique identifier (UUID) (128-bit number used to identify SNMP manager204).

In step603, SNMP manager204transfers the received marked collected real-time model state data and metadata to the user of computing device101, such as on a demand basis. In one embodiment, such information is in the form of tuples.

A further alternative method for providing the marked collected real-time model state data and metadata to SNMP manager204is discussed below in connection withFIG.7.

FIG.7is a flowchart of a further alternative method700for providing the marked collected real-time model state data and metadata to SNMP manager204in accordance with an embodiment of the present disclosure.

In step702, SNMP manager204sends an instruction to the located SNMP agents202to provide the marked collected real-time model state data and metadata to SNMP manager204. In one embodiment, such an instruction is sent via the SNMP GETNEXT operation.

In step703, SNMP manager204receives the marked collected real-time model state data and metadata from SNMP agents202. In one embodiment, SNMP agents202start the data proactively after setting the SNMP TRAP operation in response to the ACTIVATE signal being initialized. In one embodiment, SNMP manager204initializes the ACTIVATE signal. In one embodiment, a parallel polling thread (sequential flow of instructions to repeatedly determine if a signal has been issued) will be fork()ed (process which creates a copy of itself) by SNMP manager204to locate the DISABLE SIGNAL. In this manner, using the polling thread, SNMP manger204will repeatedly determine if the DISABLE SIGNAL has been issued by SNMP agents202. Upon detecting the DISABLE SIGNAL, SNMP manager204sends the TRAP_DISABLE signal to SNMP agents202to stop the proactive data sharing with SNMP manager204. In one embodiment, the TRAP_DISABLE signal is sent after execution of the SNMP GET operation. In one embodiment, the polling for the SNMP GETNEXT operation is activated for future messages.

In step704, SNMP manager204transfers the received marked collected real-time model state data and metadata to the user of computing device101, such as on a demand basis. In one embodiment, such information is in the form of tuples.

In this manner, the performance of machine learning models in the service orchestration plane of a broadband cellular network (e.g., fifth generation broadband cellular network) may be monitored.

As a result of the foregoing, embodiments of the present disclosure provide a means to obtain better internal details from the machine learning models, which are useful in understanding the performance of machine learning models in a multi-domain orchestration and programmability framework.

Furthermore, the embodiments of the present disclosure provide a way to communicate with the machine learning models and receive detailed information from the machine learning models on a standard interface which provides better flexibility of processing verification in the machine learning space.

Furthermore, the principles of the present disclosure improve the technology or technical field involving machine learning models. As discussed above, in a cognitive system, which uses cognitive computing, natural language processing and machine learning to enable people and machines to interact more naturally to extend and magnify human expertise and cognition, there could be many different machine learning models with different functions and operation feature sets to produce different outcomes. By having a variety of machine learning models, the cognitive system is enhanced in its ability to enable people and machines to interact more naturally. Such a cognitive system may utilize a broadband cellular network, such as the fifth generation technology standard for the broadband cellular network (“5G”). In such an architecture, dissimilar machine learning models reside in the service orchestration plane. The service orchestration plane introduces a parent level of abstraction that alleviates the need for other services to manage interaction details required to ensure that service operations are executed in a specific sequence. As discussed above, in the service orchestration plane, there are multiple dissimilar machine learning models in the service orchestration plane, each operating with different training datasets, using different algorithms to train the model using the different training datasets. Once the machine learning model is selected to be utilized, the outcome of the machine learning model needs to be evaluated in real-time as to the accuracy of the prediction, including in situations involving automated audits of the machine learning model. Unfortunately, there is not currently a means for performing real-time monitoring of the machine learning models in the service orchestration plane. As a result, the performance of such machine learning models in the service orchestration plane may be unknown.

Embodiments of the present disclosure improve such technology by providing real-time monitoring of the machine learning models in the service orchestration plane. In one embodiment, real-time model state data and metadata (e.g., type of machine learning model, an operating dataset, features of a quorum configuration, attributes of the machine learning models, etc.) of the machine learning models located within an orchestration plane of a network are collected by agents located within the machine learning models. In one embodiment, such agents utilize the simple network management protocol (SNMP) and are referred to herein as the “SNMP agents.” The portion of the collected model state data and metadata that is to be provided to the user by the service orchestrator (configured to monitor the machine learning models in the service orchestration plane via the use of agents in the machine learning models) is selected and marked. In one embodiment, such information (model state data and metadata) may be selected based on input received from an expert. In one embodiment, such information (model state data and metadata) may be selected based on prior information previously captured by the SNMP agents. In one embodiment, the selected information (model state data and metadata) to be provided to the user by the service orchestrator is marked by setting a value to a flag associated with such information. The marked collected real-time model state data and metadata are then provided to the user by the service orchestrator. In this manner, real-time monitoring of the machine learning models in the orchestration plane, such as the service orchestration plane, of a broadband cellular network (e.g., fifth generation broadband cellular network) is achieved. Furthermore, in this manner, there is an improvement in the technical field involving machine learning models.

The technical solution provided by the present disclosure cannot be performed in the human mind or by a human using a pen and paper. That is, the technical solution provided by the present disclosure could not be accomplished in the human mind or by a human using a pen and paper in any reasonable amount of time and with any reasonable expectation of accuracy without the use of a computer.