Patent Description:
Platforms for development and execution of models are known. However, the capabilities of these platforms to provide a multi-platform modeling environment in which models can be executed locally and/or on distributed systems are insufficient.

<CIT> relates to a distributed, multi-model, self-learning platform for machine learning.

The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosure. The summary is not an extensive overview of the disclosure. It is neither intended to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure. The following summary merely presents some concepts of the disclosure in a simplified form as a prelude to the description below.

Aspects of the disclosure relate to systems, methods, and computing devices configured to manage the processing and execution of models that may have been developed on a variety of platforms. A multi-model execution module specifying a sequence of models to be executed may be determined. A multi-platform model processing and execution management engine may execute the multi-model execution module internally, or outsource the execution to a distributed model execution orchestration engine. A model data monitoring and analysis engine may monitor the internal and/or distributed execution of the multi-model execution module, and may further transmit notifications to various computing systems. According to an aspect of the present disclosure, there is provided a method as set out in claim <NUM>.

Other features and advantages of the disclosure will be apparent from the additional description provided herein.

A more complete understanding of the present invention and the advantages thereof may be acquired by referring to the following description in consideration of the accompanying drawings, in which like reference numbers indicate like features, and wherein:.

In the following description of the various embodiments, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration, various embodiments of the disclosure that may be practiced. It is to be understood that other embodiments may be utilized.

As mentioned above, aspects of the disclosure relate to systems, devices, computer-implemented methods, and computer-readable media for managing the processing and execution of models that may have been developed on a variety of platforms. <FIG> illustrates an example of a suitable computing system <NUM> that may be used according to one or more illustrative embodiments. The computing system <NUM> is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality contained in the disclosure. The computing system <NUM> should not be interpreted as having any dependency or requirement relating to any one or combination of components shown in the illustrative computing system <NUM>.

The disclosure is operational with numerous other special purpose computing systems or configurations. Examples of computing systems, environments, and/or configurations that may be suitable for use with the disclosed embodiments include, but are not limited to, personal computers (PCs), server computers, hand-held or laptop devices, mobile devices, tablets, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like and are configured to perform the functions described herein. The mobile devices, for example, may have virtual displays or keyboards.

With reference to <FIG>, the computing system <NUM> may include a computing device (e.g., server <NUM>) wherein the processes discussed herein may be implemented. The server <NUM> may have a processor <NUM> for controlling the overall operation of the server <NUM> and its associated components, including random-access memory (RAM) <NUM>, read-only memory (ROM) <NUM>, input/output module <NUM>, and memory <NUM>. Processor <NUM> and its associated components may allow the server <NUM> to receive one or models from one or more platforms, process these models to generate standardized models, receive one or more multi-model execution modules utilizing one or more of the standardized models, and execute the multi-model execution module locally or outsource the multi-model execution module to a distributed model execution orchestration engine.

Server <NUM> may include a variety of computer-readable media. Computer-readable media may be any available media that may be accessed by server <NUM> and include both volatile and non-volatile media, removable and non-removable media. For example, computer-readable media may comprise a combination of computer storage media and communication media.

Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, random access memory (RAM), read only memory (ROM), electronically erasable programmable read only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information that can be accessed by server <NUM>.

Computing system <NUM> may also include optical scanners (not shown). Exemplary usages include scanning and converting paper documents, such as correspondence, data, and the like to digital files.

Although not shown, RAM <NUM> may include one or more applications representing the application data stored in RAM <NUM> while the server <NUM> is on and corresponding software applications (e.g., software tasks) are running on the server <NUM>.

Input/output module <NUM> may include a microphone, keypad, touch screen, and/or stylus through which a customer of server <NUM> may provide input, and may also include one or more of a speaker for providing audio output and a video display device for providing textual, audiovisual and/or graphical output.

Software may be stored within memory <NUM> and/or storage to provide instructions to processor <NUM> for enabling server <NUM> to perform various functions. For example, memory <NUM> may store software used by the server <NUM>, such as an operating system <NUM>, application programs <NUM>, and an associated database <NUM>. Also, some or all of the computer executable instructions for server <NUM> may be embodied in hardware or firmware.

Server <NUM> may operate in a networked environment supporting connections to one or more remote computing devices, such as computing devices <NUM>, <NUM>, and <NUM>. The computing devices <NUM>, <NUM>, and <NUM> may be personal computing devices or servers that include many or all of the elements described above relative to the server <NUM>. The computing devices <NUM>, <NUM>, <NUM> may be a mobile computing devices or servers that include many or all of the elements described above relative to server <NUM>.

The network connections depicted in <FIG> include a local area network (LAN) <NUM> and a wide area network (WAN) <NUM>, but may also include other networks. When used in a LAN networking environment, server <NUM> may be connected to the LAN <NUM> through a network interface (e.g., LAN interface <NUM>) or adapter in the communications module <NUM>. When used in a WAN networking environment, the server <NUM> may include a modem <NUM> or other means for establishing communications over the WAN <NUM>, such as the Internet <NUM> or other type of computer network. It will be appreciated that the network connections shown are illustrative and other means of establishing a communications link between the computing devices may be used. Various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like may be used, and the system may be operated in a client-server configuration to permit a customer to retrieve web pages from a web-based server. Any of various conventional web browsers may be used to display and manipulate on web pages.

Additionally, one or more application programs <NUM> used by the server <NUM>, according to an illustrative embodiment, may include computer executable instructions for invoking functionality related to communication including, for example, email short message service (SMS), and voice input and speech recognition applications.

Embodiments of the disclosure may include forms of computer-readable media. Computer-readable media include any available media that can be accessed by a server <NUM>. Computer-readable media may comprise storage media and communication media and in some examples may be non-transitory. Storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, object code, data structures, program modules, or other data. Communication media include any information delivery media and typically embody data in a modulated data signal such as a carrier wave or other transport mechanism.

Memory <NUM> may also store data <NUM> used in performance of one or more aspects of this disclosure. For example, data <NUM> may include received from modeling platforms, post-processed standardized models, multi-model execution modules, data sets for the multi-model execution modules, and results from execution of the multi-model execution modules.

Various aspects described herein may be embodied as a method, a data processing system, or as a computer-readable medium storing computer-executable instructions. For example, a computer-readable medium storing instructions to cause a processor to perform steps of a method in accordance with aspects of the disclosed embodiments is contemplated. For instance, aspects of the method steps disclosed herein may be executed on a processor <NUM> on server <NUM>. Such a processor may execute computer-executable instructions stored on a computer-readable medium.

<FIG> illustrates an example of a multi-platform model processing and execution management engine <NUM> that may be used according to one or more illustrative embodiments. Multi-platform model processing and execution management engine <NUM> may include a processor <NUM>. Processor <NUM> is configured to receive data from one or more interfaces. The data received from these interfaces comprise models that have been developed on different modeling platforms. For example, processor <NUM> may receive a first external model <NUM> via interface <NUM>, a second external model <NUM> via interface <NUM>, and/or a third external model <NUM> via interface <NUM>. Each of first external model <NUM>, second external model <NUM>, and third external model <NUM> have been developed on different modeling platforms. For example, these different modeling platforms may include frameworks in R such as glm, glmnet, gbm, xgboost, frameworks in Python such as scikitlearn and xgboost, and standalone tools, such as H2O. Additionally, each of interface <NUM>, interface <NUM>, and interface <NUM> may be different types of interfaces. For example, interface <NUM> may be a graphical user interface that may be accessed via a web service. In another example, interface <NUM> may be an application programmable interface (API) in R, Python, Java, PHP, Ruby, or scala, and/or the like. In yet another example, the interface may be a command line interface (CLI) executed in the shell.

Upon receiving an external model from an interface, processor <NUM> is configured to process the model to generate an internal model. Continuing with the example above, processor <NUM> processes first external model <NUM> to generate first internal model <NUM>, process second external model <NUM> to generate internal model <NUM>, and process third external model <NUM> to generate internal model <NUM>. Each of these internal models are in a standard format such that these internal models may be executed by multi-platform model processing and execution management engine <NUM>, or outsourced for execution by multi-platform model processing and execution management engine <NUM>. Processing of these external models may include translating the model from an external language to an internal language. Processing may additionally or alternatively include verifying that the internal models are functional. For example, calls to the internal model from a computing device may be simulated via processor <NUM>.

In an alternative environment, one or more converters may be utilized to convert the external models to internal models. For example, one or more converters may be located external to multi-platform model processing and execution management engine <NUM>. These one or more converters may receive the external models <NUM>, <NUM>, and <NUM> from interfaces <NUM>, <NUM>, and <NUM>, respectively. Additionally, or alternatively, the one or more converters may receive the external models <NUM>, <NUM>, and <NUM> directly from the modeling engines in which the external models were generated. The one or more converters may process external models <NUM>, <NUM>, and <NUM> to generate internal models <NUM>, <NUM>, and <NUM>, respectively. In one instance, each modeling framework may have its own dedicated converter (or converters). For example, a first converter may be utilized to transform an external model, that was created within the Python modeling framework, to an internal framework used by multi-platform model processing and execution management engine <NUM>, such as JSON. Similarly, a second converter may be utilized to transform an external model that was created within the R modeling framework to an internal framework used by multi-platform model processing and execution management engine <NUM> (such as JSON). The one or more converters may then transmit internal models <NUM>, <NUM>, and <NUM> to processor <NUM> of multi-platform model processing and execution management engine <NUM>.

Once the internal models are generated and/or received from external converters, these models may be stored within one or more storage devices of multi-platform model processing and execution management engine <NUM>. One or more of the internal models may additionally or alternatively be stored on external storage devices. The internal models may be modified on an as-needed basis. Processor <NUM> may be configured to generate and transmit notifications to one or more computing devices subsequent to generation of an internal model. These notifications may indicate that the processing of the corresponding external model is complete. These notifications may additionally include information about and access to the internal model.

As discussed below in reference to <FIG>, one or more multi-model execution modules may be created, each of which may include calls to one or more internal models. In addition to receiving external models created via various modeling platforms, multi-platform model processing and execution management engine <NUM> may outsource the execution of these multi-model execution modules to various model execution environments. In the claimed invention, a first multi-model execution module is deployed on a mobile device, while a second multi-model execution module is deployed on a cluster/cloud environment, such as Hadoop.

<FIG> illustrates additional example aspects of a multi-platform model processing and execution management engine <NUM> that may be used according to one or more illustrative embodiments. Processor <NUM>, internal model <NUM>, internal model <NUM>, and internal model <NUM>, shown in <FIG>, may correspond to processor <NUM>, internal model <NUM>, internal model <NUM>, and internal model <NUM>, shown in <FIG>, respectively. Application <NUM> and application <NUM> may each be executing on computing devices external to multi-platform model processing and execution management engine <NUM>. Application <NUM> and application <NUM> may each communicate with processor <NUM> via one or more interfaces (not shown).

The applications may communicate with processor <NUM> to create one or more multi-model execution modules. For example, application <NUM> may create and configure multi-model execution module <NUM>. Application <NUM> may communicate with processor <NUM> to create and configure multi-model execution module <NUM> and multi-model execution module <NUM>. Each multi-model execution module may comprise one or more datasets and calls to one or more internal models. For example, multi-model execution module <NUM> may include dataset 311a and dataset 311b, and may include calls to both internal model <NUM> and internal model <NUM>. In another example, multi-model execution module <NUM> may include dataset 312a and may include calls to both internal model <NUM> and internal model <NUM>. In yet another example, multi-model execution module <NUM> may include dataset 313a and may include a call to internal model <NUM>.

A many-to-many relationship exists between the applications and the internal models. That is, a single application (for example, application <NUM>) may create multiple multi-model execution modules that result in calls to many different models, and each internal model (for example, internal model <NUM>) may be called by many applications via the various multi-model execution modules created by these applications. Each of the applications may use the different internal models differently and with different inputs. Certain applications may utilize one or more internal models on a daily basis, whereas other applications may utilize those one or more internal models on a less-frequent basis (i.e., weekly, bi-weekly, monthly, etc.). In another application, certain applications may utilize one or more internal models for batch jobs (i.e., running the internal models multiple times using multiple sets of data), whereas other applications may utilize those one or more internal models in a one-time use case.

A multi-model execution module may include a sequence of calls to various internal models. The sequence of calls may be pre-configured by the application associated with the multi-model execution module. Alternatively, the sequence of calls may be dynamically determined during the execution of the multi-model execution module (discussed in detail below in reference to <FIG>). Each call to an internal model within the multi-model execution module may be associated with a different dataset. For example, in multi-model execution module <NUM>, the call to internal model <NUM> may be associated with dataset 311a, and the call to internal model <NUM> may be associated with dataset 311b. The datasets may be stored within the multi-model execution module, processor <NUM>, a storage device internal to multi-platform model processing and execution management engine <NUM>, an external storage device, and/or on a cloud-based storage device. The calls to the internal models may include the actual datasets themselves, or may alternatively include information identifying the location of the datasets. In the case of the latter, the dataset may be retrieved from its location during the execution of the internal model. In one instance, one or more of the datasets may be created and propagated with data during the configuration of the multi-model execution module. In another instance, one or more of the datasets may be dynamically created and/or propagated with data during the execution of the multi-model execution module. Processor <NUM> may execute multiple multi-model execution modules simultaneously.

In one instance, a multi-model execution module may be configured to be executed locally by processor <NUM>. For example, application <NUM> may create and configure multi-model execution module <NUM> on processor <NUM>. Once the configuration of multi-model execution module <NUM> is complete, multi-model execution module <NUM> may be locally executed by processor <NUM>. Execution of multi-model execution module <NUM> locally by processor <NUM> may include one or more local calls from processor <NUM> to internal models. For example, during execution of multi-model execution module <NUM>, processor <NUM> may call internal model <NUM>. Processor <NUM> may transmit dataset 311a to internal model <NUM> as part of the call to the internal model <NUM>. Alternatively, processor <NUM> may transmit instructions to internal model <NUM> to access dataset <NUM> as needed during the execution of internal model <NUM>. Dataset 311a may include data to be utilized by internal model <NUM> during execution of internal model <NUM>. Internal model <NUM> may store the results of the execution to dataset 311a. These results may also be transmitted from internal model <NUM> to processor <NUM> (where they may be stored), or to another internal model.

Once execution of internal model <NUM> is complete, and the results have been received by multi-model execution module <NUM> (and/or stored in dataset 311a), processor <NUM> calls internal model <NUM>. Alternatively, internal model <NUM> may be configured by multi-model execution module <NUM> to call internal model <NUM> once execution of internal model <NUM> is complete. The call to internal model <NUM> includes dataset 311b. Dataset 311b includes data that is to be used in the execution of internal model <NUM>. A portion of the data in dataset 311b is output data generated by execution of one or more additional internal models. Internal model <NUM> may store the results of the execution of internal model <NUM> to dataset 311b. These results may also be transmitted from internal model to processor <NUM> (where they may be stored), or to another internal model. Once execution of internal model <NUM> is complete, and the results have been received by multi-model execution module <NUM> (and/or stored in dataset 311a), processor <NUM> may call one or more additional models as specified within multi-model execution module <NUM>. If no additional internal models are to be called, processor <NUM> aggregates the results of multi-model execution module <NUM> (i.e., the data produced by the execution of the internal models). Processor <NUM> may then transmit a notification to application <NUM> (or the corresponding computing device) indicating that execution of multi-model execution module <NUM> is complete. Additionally, or alternatively, processor <NUM> may transmit the aggregated results of the execution of multi-model execution module <NUM> to the corresponding computing device. In one instance, processor <NUM> may process the aggregated results data prior to transmitting the data to the corresponding computing device. Processor <NUM> may be configured to similarly locally execute multi-model execution module <NUM> and/or multi-model execution module <NUM>.

As noted above, in other instances, processor <NUM> may be configured to outsource the execution of one or more multi-model execution modules. For example, processor <NUM> may be configured to outsource the execution of multi-model execution module <NUM> to a distributed model execution orchestration engine (discussed below in reference to <FIG>).

<FIG> illustrates computing environment <NUM> comprising a plurality of multi-platform model processing and execution management engines. Each of multi-platform model processing and execution management engine 400a, multi-platform model processing and execution management engine 400b, and multi-platform model processing and execution management engine 400c may be an instantiation of multi-platform model processing and execution management engines <NUM>. Application <NUM> may be an instantiation of application <NUM> or <NUM>. Although only one application is shown for purposes of illustratively clarity, multiple applications be present in environment <NUM> (for example, as shown in <FIG>).

Application <NUM> may create one or more multi-model execution modules on each of multi-platform model processing and execution management engine 400a, multi-platform model processing and execution management engine 400b, and multi-platform model processing and execution management engine 400b. As discussed above in reference to <FIG>, creation of the multi-model execution modules may include the transmittal of data between application <NUM> and the multi-platform model processing and execution management engines.

In a first instance, application <NUM> may create a first multi-model execution module on multi-platform model processing and execution management engine 400a. The first multi-model execution module may be an interactive multi-model execution module. In this instance, application <NUM> may transmit data (or location of data) needed for the creation of the first multi-model execution module to multi-platform model processing and execution management engine 400a, and may further transmit data (or the location of data) needed for the execution of the first multi-model execution module. For example, application <NUM> may transmit, to multi-platform model processing and execution management engine 400a, one or more datasets that are to be utilized during execution of the first multi-model execution module on multi-platform model processing and execution management engine 400a. Once the first multi-model execution module has been created on multi-platform model processing and execution management engine 400a, multi-platform model processing and execution management engine 400a may initiate execution of the first multi-model execution module. To execute the first multi-model execution module, multi-platform model processing and execution management engine 400a may utilize datasets that have been transmitted from application <NUM>. Additionally, or alternatively, multi-platform model processing and execution management engine 400a may utilize datasets that are stored on external storage devices, such as database <NUM>. Execution of the first multi-model execution module may include calls to one or more internal models. Instantiations of these one or more internal models may also be stored on external storage devices, such as database <NUM>. Accordingly, during execution of the first multi-model execution module, multi-platform model processing and execution management engine 400a may transmit data to (such as datasets to be utilized as inputs for one or more internal models), and may receive data from (such as internal model data) the external storage devices. Once execution of the first multi-model execution module is complete, multi-platform model processing and execution management engine 400a may transmit the results of the execution of the first multi-model execution module to application <NUM>.

In one variation of the first instance, application <NUM> may create and store the first multi-model execution module on an external storage device, such as database <NUM>. Application <NUM> may then transmit execution instructions to the multi-platform model processing and execution management engine 400a. The execution instructions may include instructions that trigger multi-platform model processing and execution management engine 400a to retrieve the first multi-model execution module from the external storage device, and to execute the first multi-model execution module. Again, once execution of the first multi-model execution module is complete, multi-platform model processing and execution management engine 400a may transmit the results of the execution of the first multi-model execution module to application <NUM>.

In a second instance, application <NUM> may create a second multi-model execution module on multi-platform model processing and execution management engine 400b. The second multi-model execution module may be a batch multi-model execution module. Batch multi-model execution modules may include multiple executions of a same sequence of internal models. Each execution of the sequence of internal models may utilize different datasets. In this instance, application <NUM> may transmit data (or location of data) needed for the creation of the first multi-model execution module to multi-platform model processing and execution management engine 400b, and may further transmit the location of the datasets needed for the execution of the second multi-model execution module. In one example, a first execution of the sequence of internal models may require a first set of datasets, and a second execution of the sequence of internal models may require a second set of datasets (the first and second execution are for illustrative purposes only, and execution of the batch multi-model execution module may include thousands of executions of the sequence of internal models). The datasets of the first set of datasets and the second set of datasets may be stored on different external storage devices. The locations of each of these datasets may be transmitted from application <NUM> during creation of the second multi-model execution module. As previously noted, execution of the second multi-model execution module may include thousands of executions of a same sequence of internal models, each of the executions utilizing different datasets. As each execution of the sequence of internal models is completed, multi-platform model processing and execution management engine 400b may store the results of that execution on external storage devices, such as database <NUM>. The particular external storage devices to be utilized for storage may be specified within the second multi-model execution module, or may be dynamically determined by multi-platform model processing and execution management engine 400b. The results of different executions of the sequence of internal models may be stored on the same external storage device, or may be stored on different external storage devices. Multi-platform model processing and execution management engine 400b may tag the results so that the results identify the particular dataset(s) used during the execution of the sequence of internal models.

In a third instance, application <NUM> may create a third multi-model execution module on multi-platform model processing and execution management engine 400c. The third multi-model execution module may be an outsourced multi-model execution module. In this instance, application <NUM> may transmit data (or location of data) needed for the creation of the third multi-model execution module to multi-platform model processing and execution management engine 400c, and may further transmit data (or the location of data) needed for the execution of the third multi-model execution module to multi-platform model processing and execution management engine 400c. For example, application <NUM> may transmit, to multi-platform model processing and execution management engine 400c, the location of one or more datasets that are to be utilized during execution of the third multi-model execution module. Once the third multi-model execution module has been created on multi-platform model processing and execution management engine 400c, multi-platform model processing and execution management engine 400c may outsource execution of the third multi-model execution module to distributed model execution orchestration engine 420c. This is discussed in detail below in reference to <FIG>.

<FIG> illustrates an example environment for an outsourced execution of a multi-model execution module. Multi-platform model processing and execution management engine <NUM> may be configured to transmit multi-model execution module data (and corresponding datasets) and internal model data to the distributed model execution orchestration engine <NUM>. Transmittal of the multi-model execution module data may comprise transmittal of the multi-model execution module, a portion of the multi-model execution module, or information identifying a location of the multi-model execution module. Transmittal of the multi-model execution module data may further include transmittal of one or more datasets of the multi-model execution module, and/or information identifying locations of the one or more datasets of the datasets. Transmittal of the internal model data may comprise transmittal of one or more internal models, portions of one or more internal models, and/or information identifying the location of one or more internal models.

Distributed model execution orchestration engine <NUM> may be configured to receive the multi-model execution module and internal model data from multi-platform model processing and execution management engine <NUM>. Distributed model execution orchestration engine <NUM> may further be configured to orchestrate execution of the multi-model execution module across a plurality of distributed processing engines, such as processing engine <NUM>, processing engine <NUM>, and/or processing engine <NUM>. One or more of the execution and orchestration features discussed below may be performed by a controller located on distributed model execution orchestration engine <NUM> and associated with the multi-platform model processing and execution management engine. Distributed model execution orchestration engine may distribute the execution of the multi-model execution module based on a plurality of different factors, such as processing capabilities of the various processing engines, locations of the datasets needed for the internal models, and/or availabilities of the various processing engines. For example, the execution of the multi-model execution module may require calls to three internal models. A first dataset needed for a first internal model of the three internal models may be located on processing engine <NUM>. Accordingly, distributed model execution orchestration engine <NUM> may transmit data to processing engine <NUM>, wherein the data may include instructions to execute the first internal model using data stored on processing engine <NUM>. The data transmitted to processing engine <NUM> may include the first internal model, or information identifying a location of the first internal model. A second dataset needed for a second internal model of the three internal models may be located on processing engine <NUM>. Accordingly, distributed model execution orchestration engine <NUM> may transmit data to processing engine <NUM>, wherein the data may include instructions to execute the second internal model using data stored on processing engine <NUM>. The data transmitted to processing engine <NUM> may include the second internal model, or information identifying a location of the second internal model. A third dataset needed for a third internal model of the three internal models may be located on processing engine <NUM>. Accordingly, distributed model execution orchestration engine <NUM> may transmit data to processing engine <NUM>, wherein the data may include instructions to execute the third internal model using data stored on processing engine <NUM>. The data transmitted to processing engine <NUM> may include the third internal model, or information identifying a location of the third internal model.

The processing engines may intermittently return status updates to distributed model execution orchestration engine <NUM>. In turn, distributed model execution orchestration engine <NUM> may intermittently forward these status updates to multi-platform model processing and execution management engine <NUM>. The processing engines may further transmit results of the execution of the internal models to distributed model execution orchestration engine <NUM>. In one instance, distributed model execution orchestration engine <NUM> may forward these results to multi-platform model processing and execution management engine <NUM> as they are transmitted to distributed model execution orchestration engine <NUM>. In another instance, distributed model execution orchestration engine <NUM> may wait and aggregate all the results from the various processing engines, and then transmit the aggregated results to multi-platform model processing and execution management engine <NUM>. As discussed above in reference to <FIG>, multi-platform model processing and execution management engine <NUM> may transmit these results to one or more external computing devices.

<FIG> illustrates a sequence of steps that is executed by a processor (for example, processor <NUM>) of multi-platform model processing and execution management engine <NUM> during execution of a multi-model execution module. As discussed above in reference to <FIG>, a multi-model execution module may comprise a series of calls to different internal models. Each of the calls is associated with a dataset that is to be utilized by a particular internal model when it is called. Each of the calls further results in the generation of new output data, resulting from the execution of the corresponding internal model.

At step <NUM>, the processor determines the first internal model to be called. The first internal model to be called may be identified within the multi-model execution module. The processor then retrieves the dataset that is to be used by the first internal model during its execution. In one instance, the dataset may already be stored within the multi-model execution module. In another instance, the multi-model execution module may identify the location of the dataset, and the processor may retrieve the dataset from the identified location.

At step <NUM>, the processor calls the first internal model. Calling the first internal model comprises transmitting data to the first internal model. The data includes an instruction and a dataset, wherein the instruction indicates that the first internal model is to execute using the transmitted dataset. The instructions further indicate that the dataset output from the first internal model is to be transmitted to the processor. Calling the first internal model triggers the execution of the first internal model. At step <NUM>, the processor receives a dataset output from the first internal model. In one instance, the first internal model may transmit the dataset itself to the processor. In another instance, the first internal model may transmit the location of the dataset to the processor, and the processor may retrieve the dataset from the location.

At step <NUM>, the processor updates one or more downstream datasets based on the dataset received from the first internal model. As noted above, the multi-model execution module may comprise a sequence of calls to various internal models. In certain instances, a dataset output by a first internal model during execution of the multi-model execution module (or a portion thereof) may be used as input data for a subsequent internal model during execution of the multi-model execution module. In these instances, when the processor receives the dataset from the first internal model, the processor may determine whether the dataset (or a portion thereof) is to be used during any subsequent calls of the multi-model execution module. If so, the processor propagates these downstream input datasets with data from the dataset returned by the first internal model.

At step <NUM>, the processor determines if additional internal models are to be called during execution of the multi-model execution module. In one instance, the multi-model execution module may indicate that no additional internal models are to be called. In another instance, the multi-model execution module may indicate the internal model that is to be called next. In another instance, the multi-model execution module may indicate that the determination of whether or not another internal model to be called (and the identity of that internal model) is to be dynamically determined based on the dataset returned by the first internal model. In this instance, the processor analyses the dataset returned by the first internal model and automatically determine which internal model, if any, is to be called next. For example, the processor compares one or more values of the dataset to one or more threshold values, and select the next internal model based on a result of the comparison. In one instance, the processor may compare a first value of the dataset to a first threshold value. If the first value of the dataset is above the first threshold value, the processor automatically determines that a second internal model is to be called as the next internal model; if the first value of the dataset is below the first threshold value, the processor may automatically determine that a third internal model is to be called as the next internal model. If, at step <NUM>, the processor determines (based on an explicit indication in the multi-model execution module or an analysis of the dataset returned by the first internal model) that a particular internal model is to be called next, the processor may proceed to step <NUM>.

At step <NUM>, the processor calls the next internal model. Similar to calling the first internal model, calling the next internal model comprises transmitting data to the next internal model. The data may include an instruction and a dataset, wherein the instruction indicates that the next internal model is to execute using the transmitted dataset. The instructions may further indicate that the dataset output from the next internal model is to be transmitted to the processor. Calling the next internal model may trigger the execution of the next internal model. At step <NUM>, the processor may receive a dataset output from the next internal model. In one instance, the next internal model may transmit the dataset itself to the processor. In another instance, the next internal model may transmit the location of the dataset to the processor, and the processor may retrieve the dataset from the location. At step <NUM>, the processor may update one or more downstream datasets based on the dataset received from the next internal model. The processor may then return to step <NUM>, where the processor may determine whether additional internal models are to be called during execution of the multi-model execution module.

If, at step <NUM>, the processor determines that no additional internal models are to be called, the processor may proceed to step <NUM>. At step <NUM>, the processor aggregates each of the datasets returned from the internal models called during execution of the multi-model execution module. In certain instances, the multi-model execution module may specify that only a subset of the aggregated data is to be stored as the final output data. In these instances, the multi-model execution module may process the aggregated data to filter out the unnecessary data. At step <NUM>, the processor may send the results of the execution of the multi-model execution module to one or more computing devices. The results may comprise all or a subset of the aggregated data.

<FIG> illustrates an example operating environment of a multi-platform model processing and execution management engine. Multi-model execution module <NUM>, internal model <NUM>, and distributed model execution orchestration engine <NUM> may correspond to multi-model execution module <NUM>, internal model <NUM>, and distributed model execution orchestration engine <NUM>, respectively. One or more elements of multi-platform model processing and execution management engine <NUM> may transmit data to and/or receive data from model data monitoring and analysis engine <NUM>. For example, multi-model execution module <NUM> and/or internal model <NUM> may transmit data to or receive data from model data monitoring and analysis engine <NUM>. In another example, a processor (not shown) within multi-platform model processing and execution management engine <NUM> may transmit data to or receive data from model data monitoring and analysis engine <NUM>. Additionally, distributed model execution orchestration engine <NUM> and model data monitoring and analysis engine <NUM> may be configured to exchange data.

Model data monitoring and analysis engine <NUM> may be configured to monitor data generated within multi-platform model processing and execution management engine <NUM>. One or more computing systems, such as computing system <NUM> and/or computer system <NUM>, may be utilized to configure model data monitoring and analysis engine <NUM>.

For example, as discussed above in reference to <FIG> and <FIG>, multi-model execution module <NUM> may receive data from one or more internal models. Multi-model execution module <NUM> may be configured to transmit the data received from internal models to model data monitoring and analysis engine <NUM>. Additionally, or alternatively, model data monitoring and analysis engine <NUM> may be configured to periodically transmit requests for data to multi-model execution module <NUM>. The periodic requests may be automatically transmitted by model data monitoring and analysis engine <NUM>, and may be sent every few seconds, every minute, every hour, daily, weekly, and/or the like. The particular intervals at which the requests from model data monitoring and analysis engine <NUM> are to be automatically transmitted may be configured by computing systems, such as computing system <NUM> and/or computer system <NUM>. In response, multi-model execution module <NUM> may transmit any new data received from internal models (i.e., since a last request for data was received from model data monitoring and analysis engine <NUM>) to model data monitoring and analysis engine <NUM>. Additionally or alternatively, a similar exchange of data may occur between model data monitoring and analysis engine <NUM> and one or more internal models of multi-platform model processing and execution management engine <NUM>, such as internal model <NUM>. As discussed above with respect to <FIG> and <FIG>, the execution of internal models may result in a generation of one or more datasets. The internal models may be configured to transmit the generated datasets to model data monitoring and analysis engine <NUM>. Additionally or alternatively, model data monitoring and analysis engine <NUM> may be configured to periodically transmit requests for data to internal model <NUM>. In response, internal model <NUM> may transmit any new generated datasets (i.e., since a last request for data was received from model data monitoring and analysis engine <NUM>) to model data monitoring and analysis engine <NUM>.

As discussed above in reference to <FIG>, distributed model execution orchestration engine <NUM> may be configured to receive output data from one or more processing engines that execute internal models. Distributed model execution orchestration engine <NUM> may be configured to transmit these datasets as they are received from the processing engines to model data monitoring and analysis engine <NUM>. Additionally or alternatively, model data monitoring and analysis engine <NUM> may be configured to periodically transmit requests for data to distributed model execution orchestration engine <NUM>. In response, distributed model execution orchestration engine <NUM> may transmit any new datasets received from the processing engines (i.e., since a last request for data was received from model data monitoring and analysis engine <NUM>) to model data monitoring and analysis engine <NUM>.

Model data monitoring and analysis engine <NUM> is configured to aggregate and analyze the model data received from distributed model execution orchestration engine <NUM> and multi-platform model processing and execution management engine <NUM>. The specific type(s) of analysis to be performed on the data may vary based on the source of data, the type of data, etc., and may be configured by external computing systems, such as computing system <NUM> and/or computer system <NUM>. For example, computing system <NUM> may access model data monitoring and analysis engine <NUM> via one or more interfaces (not shown). Computing system <NUM> may create one or more analysis modules (not shown) within model data monitoring and analysis engine <NUM>. Computing system <NUM> may create a first analysis module within model data monitoring and analysis engine <NUM>. Computing system <NUM> may configure the first analysis module to periodically request model data from one or more sources. For example, computing system <NUM> may configure the first analysis module to request first model data from distributed model execution orchestration engine <NUM> at a first time interval, second model data from multi-model execution module <NUM> at a second time interval, and/or third model data from internal model <NUM> at a third time interval.

Computing device <NUM> may further configure the first analysis module to perform one or more analysis functions on the received model data. For example, computing device <NUM> may configure the first analysis module to perform stability analysis on the third model data received from internal model <NUM>. The stability analysis may track the outputs of the internal model <NUM> over pre-determined time intervals, and determine whether the outputs are deviating from an expected output, or whether the outputs indicate that internal model <NUM> is degrading and requires updating. For example, internal model <NUM> may be forecasted to degrade at a first rate, and the stability analysis may include analyzing the output data to determine if the actual degradation of internal model <NUM> is tracking or exceeding the forecasted degradation of internal model <NUM>. Computing device <NUM> may configure the first analysis module to send automatic alerts to computing device <NUM> (or another computing device). In one instance, computing device <NUM> may configure the first analysis module to send an automatic alert upon detection of an unexpected deviation of the outputs. Additionally or alternatively, computing device <NUM> may configure the first analysis module to send an automatic alert upon determining that the outputs have drifted beyond a specified value or range of values. For example, computing device <NUM> may configure the first analysis module to send an automatic alert upon determining that the outputs (or values produced during and/or as a result of analysis of the outputs) fall within (or outside) a predefined range of values, above a threshold, below a threshold, and the like.

In another example, computing device <NUM> may be configured to create a second analysis module within model data monitoring and analysis engine <NUM>. Computing device <NUM> may configure the second analysis module to automatically retrieve all of the datasets of multi-model execution module <NUM>. For example, multi-model execution module <NUM> may include calls to multiple internal models, and the second analysis module may be configured to retrieve each of the input datasets and output datasets of each of the multiple internal models. Computing device <NUM> may configure the second analysis module to perform a traceability analysis on these datasets. For example, the second analysis module may analyze each of these datasets to determine the effects of particular datasets and/or internal models on the final output of the multi-model execution module. For example, if the output of the multi-model execution module largely deviated from an expected output (or expected range of outputs), the second analysis module may analyze the datasets and internal models to determine which one of (or combination of) the datasets and internal models had a substantial effect on the output. The analysis may include re-running of the models using various what-if scenarios. For example, one or more input datasets may be changed, and the multi-model execution module (or a portion thereof) may be re-executed using the modified input datasets. This process may be repeated a number of times, until the second analysis module is able to identify the one or more factors driving the outlying output.

Additionally, or alternatively, computing device <NUM> may configure the second analysis module to automatically monitor and send alerts to computing device <NUM> (or another computing device) regarding the input datasets and output datasets. Regarding the output datasets, the second analysis module may be configured to automatically request the output datasets from multi-model execution module <NUM> via automated, periodic requests. Computing device <NUM> may configure the particular time periods at which different output datasets are to be requested from multi-model execution module <NUM> within the second analysis module. The analysis of the output datasets received from the multi-model execution module <NUM> may be similar to that discussed above with reference to the first analysis module. Further similar to the first analysis module, computing device <NUM> may configure the second analysis module to automatically send alerts to computing device <NUM> when the output values fall within or outside of a predetermined range of values, above a predefined threshold, below a predefined threshold, and/or the like.

Regarding the input datasets, computing device <NUM> may configure the second analysis module to automatically request the input datasets from multi-model execution module <NUM> via automated requests. The automated requests may be a one-off event, or may occur on a periodic basis. The computing device <NUM> may further configure the second analysis module to automatically analyze the input datasets received from the multi-model execution module <NUM>. The second analysis module is configured to determine a current distribution of values of the input datasets. The second analysis module is further configured to compare the current distribution of values to an older distribution of values determined from a prior input dataset received from multi-model execution module <NUM> (or from multi-platform model processing and execution management engine <NUM>). The second analysis module determines, based on the comparison, whether there was a significant shift in the distribution of values. The second analysis module determines if the change in distribution (i.e., the difference between the current distribution of values and the older distribution of values) is within a range of predefined values, above a predefined threshold, below a predefined threshold, etc. If there is a significant shift in the distribution of the values, the second analysis module is configured to automatically send an alert indicating the shift to computing device <NUM> (or another computing device).

The analysis modules within the model data monitoring and analysis engine <NUM> may be executed as a one-time use case, or may be configured to execute periodically. The analysis modules may be configured to automatically transmit notifications and/or data to one or more computing systems. For example, the first analysis module may be configured to transmit data/notifications to computing system <NUM>, and the second analysis module may be configured to transmit data/notifications to computing system <NUM>. The specific data and/or notifications to be transmitted may be configured at the time the analysis modules are configured, and may additionally be dynamically modified on an as-needed basis. In one instance, an analysis module may be configured to transmit a notification upon detecting an unexpected output from an internal model and/or a multi-model execution module. The analysis module may additionally or alternatively be configured to transmit data reporting the results of an analysis. For example, the second analysis module may be configured to transmit data indicating the specific datasets and/or internal models that are substantially affecting the final output of the multi-model execution module. Additionally, the analysis modules may be configured to store all analysis results on one or more storage devices. These stored results may subsequently be used by the same or different analysis modules when performing automated analysis of data.

Claim 1:
A computer-implemented method comprising:
receiving, by a computing device (<NUM>), a first external model generated in a first modelling framework;
receiving, by the computing device (<NUM>), a second external model generated in a second modelling framework distinct from the first modelling framework;
generating, by the computing device (<NUM>), a first model by processing the first external model, wherein the first model is formatted using a common modelling framework;
generating, by the computing device (<NUM>), a second model by processing the second external model, wherein the second model is formatted using the common modelling framework;
transmitting, by the computing device (<NUM>), the first model to a mobile device;
triggering, by the computing device (<NUM>), execution of the first model by the mobile device;
receiving, by the computing device (<NUM>) and from the mobile device, a dataset generated based on execution of the first model by the mobile device;
obtaining, by the computing device (<NUM>), a historical statistical distribution generated based on the dataset;
calculating, by the computing device (<NUM>), a statistical distribution based on the dataset;
determining, by the computing device (<NUM>), a shift in the distribution of values between the historical statistical distribution and the calculated statistical distribution;
transmitting, by the computing device (<NUM>), a notification indicating the shift in distribution of values based on the shift exceeding a threshold value;
automatically determining, by the computing device (<NUM>) and based on the dataset and the shift in distribution of values exceeding the threshold value, the second model for execution;
transmitting, by the computing device (<NUM>), the dataset to a cloud computing device;
triggering, by the computing device (<NUM>), execution of the second model by the cloud computing device, wherein the execution of the second model is based on the dataset;
receiving, by the computing device (<NUM>), a second dataset from the cloud computing device, wherein the second dataset is generated based on the execution of the second model;
generating, by the computing device (<NUM>), an aggregated dataset based on the dataset and the second dataset; and
generating, by the computing device (<NUM>), a subset of the aggregated dataset to be stored as final output data, by processing the aggregated dataset to filter out data not designated for storage.