Secure modular machine learning platform

A secure, modular multi-tenant machine learning platform is configured to: receive untrusted code supplied by a first tenant; perform a security scan of the untrusted code to determine whether the untrusted code satisfies a set of one or more security requirements; responsive to determining that the untrusted code satisfies the security requirement(s): deploy the untrusted code to a runtime execution environment; deploy a machine learning model associated with the first tenant to the runtime execution environment, the untrusted code being configured to perform one or more functions using the machine learning model; receive a set of untrusted code supplied by a second tenant; perform a security scan of the untrusted code to determine whether the untrusted code satisfies the security requirement(s); and responsive to determining that the untrusted code does not satisfy the security requirement(s): refraining from deploying the untrusted code to a runtime execution environment.

INCORPORATION BY REFERENCE; DISCLAIMER

The following application is hereby incorporated by reference: U.S. Provisional Patent Application 63,330,645, filed Apr. 13, 2022. The applicant hereby rescinds any disclaimer of claims scope in the parent application(s) or the prosecution history thereof and advise the USPTO that the claims in the application may be broader that any claim in the parent application(s)

TECHNICAL FIELD

The present disclosure relates to machine learning. In particular, the present disclosure relates to software-as-a-service (SaaS) machine learning platforms.

BACKGROUND

Entities (e.g., businesses, organizations, individuals, etc.) are increasingly turning to machine learning to solve complex problems. Many of these entities have invested heavily (in time, money, and computing resources, for example) in creating customized data science models that solve use cases and implement business logic that is of particular interest to those entities. Such bespoke approaches may be necessary because commercially available solutions generally do not provide the desired entity-specific functionality.

For example, even if a commercially available SaaS data model has access to all the relevant data, an entity-specific model may require exporting raw data and reingesting processed machine learning model scores. The process of exporting and reingesting the entity's unique intellectual property introduces concerns about security, data access control, data staleness, etc. that must be addressed by the external system(s) (e.g., a data lake) through which the model is fed.

In some cases, an entity's custom-built machine learning model produces output that can serve as input to a SaaS product. For example, an entity may develop a customized churn model that accounts for its subscription business' unique data patterns. A third-party integration tool may allow a downstream SaaS product to consume and process the churn model's decision outcomes (e.g., decisions about customers' propensity to churn).

In addition, an entity may require or prefer to use a machine learning system that is not shared with other entities. For example, a business may prefer to keep a machine learning system separate from other businesses, in order to protect trade secrets and/or other sensitive information (e.g., employee data, earnings data, etc.). A typical approach is for the entity to host a machine learning system in its own network. However, a machine learning system may require training data and/or production data that is hosted in another location, such as a cloud storage service. The data storage location may be physically and/or logically remote (e.g., requiring data to traverse many network nodes) from the machine learning system.

The physical and/or logical distance between the data and the machine learning system can introduce various performance and/or security concerns. As one example, the physical and/or logical distance between the data and the machine learning system can introduce significant latency in transmitting data to and from the machine learning system, thus reducing overall machine learning performance. As another example, transmitting sensitive data over untrusted nodes in a public network may increase the risk that the data is intercepted by a malicious actor or otherwise compromised. As another example, if a network path between the machine learning system and the data storage goes down, the machine learning system may become inoperable due to lack of access to the data.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation and to provide a thorough understanding, numerous specific details are set forth. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in a different embodiment. In some examples, well-known structures and devices are described with reference to a block diagram form, in order to avoid unnecessarily obscuring the present invention.

The following table of contents is provided for reference purposes only and should not be construed as limiting the scope of one or more embodiments.1. GENERAL OVERVIEW2. EXAMPLE SYSTEM2.1. PIPELINE ARCHITECTURE2.1.1. CONTAINER SUPPLY2.1.1. MODEL SUPPLY2.1.2. SCAN AND CERTIFICATION PIPELINE2.1.3. BUILD AND DEPLOY PIPELINE2.1.4. TRAINING PIPELINE2.1.5. SCORING PIPELINE2.2. NETWORK ISOLATION2.3. DATA STORAGE2.4. USER INTERFACE2.5. TENANTS2.6. MACHINE LEARNING3. MACHINE LEARNING4. MACHINE LEARNING MODEL/PROFILE CONFIGURATION5. TENANT ISOLATION6. DEPLOYING UNTRUSTED CODE7. TRAINING A MACHINE LEARNING MODEL8. APPLYING A MACHINE LEARNING MODEL9. EXAMPLE GRAPHICAL USER INTERFACES10. COMPUTER NETWORKS AND CLOUD NETWORKS11. MICROSERVICE APPLICATIONS11.1. TRIGGERS11.2. ACTIONS12. HARDWARE OVERVIEW13. MISCELLANEOUS; EXTENSIONS
1. General Overview

In one or more embodiments, a machine learning platform is a software-as-a-service (SaaS) platform configured to host machine learning systems for multiple tenants. The machine learning platform is modular in that it allows each tenant to supply its own machine learning model(s) and/or code that uses the machine learning model(s). The machine learning platform is secure in at least two senses: first, in that it analyzes tenant-supplied models and/or other untrusted code for vulnerabilities before deployment to a runtime execution environment hosted by the platform; and second, in that it hosts each tenant's machine learning system(s) in a location that is physically and/or logically isolated from other tenants (e.g., in separate runtime execution environments and/or physical machines), thus protecting each tenant's data, while keeping the physical and/or logical distance between the data and the machine learning systems comparatively low (e.g., as compared with entity-hosted solutions). One or more embodiments described herein may be referred to as a “bring-your-own-model” (BYOM) and/or “bring-your-own-inference” (BYOI) approach.

One or more embodiments are configured to consume an entity's model as a secure service deployed in a scalable, secure, isolated architecture—thereby providing a technical improvement over systems that attempt to use third-party integrations between bespoke models and the data source(s) used by those models. Reducing the physical and/or logical distance between the model and the data allows data to be processed (e.g., scored) in a near-real-time pipeline using each entity's respective customized, purpose-built machine learning models. This approach helps ensure that data is no longer stale, so that large data pipelines do not need to be maintained and hydrated. An entity's machine learning model can be accessed directly, helping to provide a return on the entity's investment in data science. Moreover, one or more embodiments provide cost savings by reducing costs associated with data transfer and latency.

A SaaS product according to one or more embodiments provides extensible machine learning framework, in which each entity's machine learning models are consumed, orchestrated through flexible data and model pipelines, and deployed in a scalable and extensible architecture that integrates tightly and seamlessly into the SaaS product. An entity's machine learning models may be deployed in an isolated network that protects itself from potentially malicious zero-trust code by identifying and neutralizing threats in the entity's code, thus protecting both the orchestrator deployments and other entities' respective models.

2. Example System

FIGS.1A-1Eillustrates an example of a secure modular machine learning platform100in accordance with one or more embodiments. As illustrated inFIG.1A-1E, the platform100includes: a supply pipeline106and various components thereof; a scan and certification pipeline128and various components thereof; a build and deploy pipeline136and various components thereof; a training pipeline148and various components thereof; and a scoring pipeline156and various components thereof. In one or more embodiments, the platform100may include more or fewer components than the components illustrated inFIGS.1A-1E. The components illustrated inFIGS.1A-1Emay be local to or remote from each other. The components illustrated inFIGS.1A-1Emay be implemented in software and/or hardware. Each component may be distributed over multiple applications and/or machines. Multiple components may be combined into one application and/or machine. Operations described with respect to one component may instead be performed by another component.

For ease of discussion, examples are described herein with reference to components produced by one or more specific vendors. For example, some examples include one or more components produced by Oracle International Corporation, one or more components produced by Docker Inc., etc. One or more embodiments are not limited to the specific components produced by the specific vendors used in these examples.

Additional embodiments and/or examples relating to computer networks are described below in the section titled “Computer Networks and Cloud Networks.”

2.1. Pipeline Architecture

As noted above, the platform100includes one or more of: a supply pipeline106; a scan and certification pipeline128; a build and deploy pipeline136; a training pipeline148; and/or a scoring pipeline156. Examples of each of these pipelines are described in further detail below. One or more of the pipelines (e.g., supply pipeline106and scan and certification pipeline128) may be implemented within a tenant's isolated network within the platform100, i.e., a physical and/or virtual network in the platform100that is isolated from similar networks used by other tenants. Machine learning activities such as training, scoring, etc. may be activated through the respective pipelines. The platform100may be configured to ingest data into the training pipeline from a previous and/or ongoing export into tenant-specific storage (e.g., a tenant-specific object store). For example, the platform may host tenant-specific storage within a tenant-specific compartment that provides physical and/or logical isolation from other tenants.

In examples described herein, untrusted code is provided in container images, such as Docker images and/or other kinds of container images. Alternatively or additionally, untrusted code may be provided in another format that is suitable for deployment to a runtime execution environment. In general, container images can execute in any environment that supports the corresponding container image format; thus, container images can be prepared tenant-side and deployed to the platform100even if the two systems use different operating systems, etc.

In an embodiment, container images and model files follow separate pipelines, or separate paths through the same pipelines; this allows for a tenant to replace an image without needing to resupply the model file, or to replace a model file without resupplying the image.

2.1.1. Container Supply

In an embodiment, the supply pipeline106is configured to ingest a container120that includes untrusted code. As used herein, a “container” refers to a packaged set of code that can be executed in any environment that supports the container format. A container120may also be referred to as an “image” or “container image.” Examples of container formats include, without limitation, Docker containers, Linux containers (LXC), Solaris containers, etc. The code in a container120is “untrusted” because it originates from outside the platform100and may include malicious code (e.g., a virus, malware, etc.) and/or code that is otherwise detrimental to the operating environment (e.g., that consumes excessive computing resources).

For the supply pipeline106to ingest a container120, a tenant102may supply credentials110(e.g., via a user interface104) to allow the supply pipeline106to retrieve a container120from a separate container repository108that is external to the platform100. The supply pipeline106may include a credential manager116configured to secure the tenant's credentials110using envelope encryption and/or another kind of security. The credential manager116may be configured to store the secured credentials in a credential repository122. The supply pipeline106may include a tenant repository connector114configured (e.g., using an executable job) to connect to the container repository108using the supplied credentials110and obtain the container120from the repository108. For example, the container repository108may be a Docker Repository, and the tenant credentials110may include information needed to access the Docker Repository, such as a uniform resource locator (URL) of the Docker Registry, a username, and a password. The supply pipeline106may be configured to store the container120in an internal container repository124that is part of the platform100's infrastructure, such as an object store and/or a local registry.

2.1.1. Model Supply

In an embodiment, the supply pipeline106is configured to ingest a machine learning model112into the platform100. A tenant102may upload a representation of the machine learning model112via a user interface104. For example, the tenant102may upload a trained model112using an upload API118of the platform100that is accessible via the interface104. The platform100may store the tenant-supplied model112in an internal model repository126. For example, the platform100may store the tenant-supplied model112as a serialized binary or pickle file. As used herein, “pickle” (or “PKL”) refers to the Python pickle module, which is configured to serialize and deserialize Python object structures. Other formats may be used.

In an embodiment, one or more model files supplied by the tenant include(s) information about one or more functions that can be applied to a target item/entity, to perform scoring and/or other machine learning functions. Thus, while the model file(s) may not include an immediately operational model, the platform100can recreate an operational model from the model file.

2.1.2. Scan and Certification Pipeline

In an embodiment, the scan and certification pipeline128is configured perform security scans on a container120. An internal (also referred to as “local”) container repository124may function as a “staging” repository for the container120. The scan and certification pipeline128may include one or more security scanners130configured to execute one or more security scans on the container120. A security scan may examine the container120for one or more of viruses, ransomware, malware, spyware, etc. A security scan may generate one or more security scan scores. One or more security scans may include an anti-virus scan configured to generate a report and/or a security scan score. The scan and certification pipeline128may be configured to compare a security scan score with a predetermined threshold value that indicates an acceptable level of security. If the security scan score does not satisfy the threshold value, then the scan and certification pipeline128may generate a message (e.g., a report indicating one or more security issues) and transmit the message to the tenant102(e.g., via the interface104). In an embodiment, if each of the security scan scores satisfies the corresponding threshold value, then the scan and certification pipeline128certifies the container120by moving it from a staging area to a main/production area. To move the container120from staging, the scan and certification pipeline128may perform one or more of: moving the container120to a different repository (not shown); moving the container120to a different area of the same repository124; and/or changing a flag or other indicator in the repository124to indicate that the container120is no longer in staging.

In an embodiment, the scan and certification pipeline128is configured perform security scans on a tenant-supplied model112. An internal (also referred to as “local”) model repository126may function as a “staging” repository for the model112. The scan and certification pipeline128may include one or more security scanners134configured to execute one or more security scans on the container120. A security scanner134used to scan a model112may be the same or different from a security scanner130used to scan a container120. A security scan may examine the model112for one or more of viruses, ransomware, malware, spyware, etc. A security scan may generate one or more security scan scores. One or more security scans may include an anti-virus scan configured to generate a report and/or a security scan score. The scan and certification pipeline128may be configured to compare a security scan score with a predetermined threshold value that indicates an acceptable level of security. If the security scan score does not satisfy the threshold value, then the scan and certification pipeline128may generate a message (e.g., a report indicating one or more security issues) and transmit the message to the tenant102(e.g., via the interface104). In an embodiment, if each of the security scan scores satisfies the corresponding threshold value, then the scan and certification pipeline128certifies the model112by moving it from a staging area to a main/production area. To move the model112from staging, the scan and certification pipeline128may perform one or more of: moving the model112to a different repository (not shown); moving the model112to a different area of the same repository126; and/or changing a flag or other indicator in the repository126to indicate that the model126is no longer in staging.

2.1.3. Build and Deploy Pipeline

In an embodiment, the build and deploy pipeline136is configured to prepare a container for deployment. Operation of the build and deploy pipeline136may depend on what the tenant102provides. If the tenant102provides a model112without a tenant-supplied container, a container bundler142may bundle the model112with a default container140, to generate a bundled container144. If the tenant102provides a container120, an endpoint verifier138may scan the container120to verify that it exposes the necessary endpoints for supplying inputs to and obtaining outputs from the model112. The container bundler142may bundle the verified model112with the tenant-supplied container120, to obtain a bundled container144.

For example, if a model pickle file is provided, the build and deploy pipeline136may package the file into a Docker image (e.g., a default Docker image that was previously prepared and scanned for vulnerabilities) and generate a new container image bundled with the pickle file. If a Docker image and model pickle file are both provided, then the build and deploy pipeline136may ensure that the image exposes the necessary endpoints, then package the file into the tenant-supplied Docker image.

The build and deploy pipeline136may be configured to store the bundled container144in the internal model repository126, internal container repository124, and/or another storage location.

In an embodiment, the build and deploy pipeline136is configured to deploy the bundled container144to a runtime execution environment146. For example, the runtime execution environment146may include a Kubernetes cluster within a virtual cloud network associated with the tenant102. The runtime execution environment146may be unique to a specific tenant102. Alternatively, the runtime execution environment146may be shared by multiple tenants of the platform100. Examples of runtime execution environments are described in further detail below. To deploy the bundled container144, the platform100may use an automation server. For example, the platform100may use Jenkins, which is a free, open-source automation server produced by Oracle International Corporation. The build and deploy pipeline136may also be configured to activate or “spin up” the bundled container144in the runtime execution environment146to which it was deployed.

2.1.4. Training Pipeline

In an embodiment, the training pipeline148configured to train a machine learning model. Training may occur after the bundled container144, which includes the machine learning model, is deployed to the runtime execution environment146and before using the machine learning model on production data. Alternatively or additionally, training may occur on an ongoing basis, in a feedback loop that refines the machine learning model based on results obtained using production data.

In the training pipeline,148a scheduler154may be configured to trigger an orchestrator152to obtain information about the machine learning model. The orchestrator152is configured to activate or “spin up” an enterprise integrator model (not shown) for the pipeline. The enterprise integrator model is configured to activate or “spin up” a training job150(e.g., a Kubernetes job) to perform the training. The training job150executes to train the model and continues operating until one or more completion criteria is/are satisfied (e.g., all training data has been processed). The orchestrator152may be configured to poll for the status of any training job(s)150.

In an embodiment, the scoring pipeline156is configured to perform scoring using a trained machine learning model in the runtime execution environment146. Scoring generates one or more insights by applying the trained machine learning model to production data. Scoring is only one example of how a machine learning model may be used; other examples include, but are not limited to, generating one or more predictions and/or using the output of the model to continue training the model.

To perform scoring, an orchestrator (not shown) may initiate the scoring pipeline156(e.g., in a corresponding Kubernetes pod). The scoring pipeline156may be configured to pull data from one or more sources (e.g., one or more data platforms162that may be external to the platform100), via a data platform API160, and store the data in a tenant-specific data repository164(e.g., an object store associated with the tenant102). The scoring pipeline156(e.g., code executing in a Kubernetes pod) is configured to obtain the data from the data repository164and apply the machine learning model to the data within the runtime execution environment146. The scoring pipeline156may be configured to store the output of the machine learning model (e.g., newly scored data) in a repository within the platform and/or transmit the output to the external data source.

2.2. Network Isolation

In an embodiment, the secure modular machine learning platform100is configured to isolate tenants' untrusted code from each other. The platform100may be configured to execute an orchestrator on an existing cluster (e.g., a Kubernetes cluster), separate from the tenant-specific clusters. Tenant-specific clusters may be configured to execute only tenant-provided containers and/or machine learning models. The orchestrator may be configured to transmit instructions to the pipelines, to perform their respective functions.

In an embodiment, the orchestrator is configured to trigger tenant-specific machine learning models in tenant-specific clusters. Tenant-supplied code executing in tenant-specific clusters may be configured to utilize the respective tenant-specific machine learning model(s). The platform may be configured so that all communication with tenant-specific clusters (e.g., a call to initiate a scoring process) must come from the orchestrator's “master” cluster.

In an embodiment, each tenant's code executes in its own virtual cloud network (VCN). Each tenant's respective VCN may be separated from the others by firewall rules. Each VCN may be configured to accept only incoming data (“ingress”) to the tenant-specific cluster. Alternatively or additionally, each cluster may expose only a limited set of ports for specific communication protocols. The platform100may not provide any mechanism to allow different tenants' respective VCNs to communicate with each other.

2.3. Data Storage

In one or more embodiments, the platform includes a data repository. A data repository is any type of storage unit and/or device (e.g., a file system, database, collection of tables, and/or any other storage mechanism) for storing data. The data repository may include multiple different storage units and/or devices. The multiple different storage units and/or devices may or may not be of the same type or located at the same physical site. The data repository may be implemented or executed on the same computing system as one or more other components illustrated inFIG.1and/or on a separate computing system. The data repository may be communicatively coupled to one or more other components via a direct connection or via a network. Information may be implemented across any of the components of the platform other than the data repository.

2.4. User Interface

In one or more embodiments, the platform100includes a user interface104. A user interface104refers to hardware and/or software configured to facilitate communications between a user and one or more components of the platform100. The interface104renders user interface elements and receives input via user interface elements. Examples of interfaces104include a graphical user interface (GUI), a command line interface (CLI), a haptic interface, and a voice command interface. Examples of user interface elements include checkboxes, radio buttons, dropdown lists, list boxes, buttons, toggles, text fields, date and time selectors, command lines, sliders, pages, and forms. Different components of the interface104may be specified in different languages. For example, the behavior of user interface elements may be specified in a dynamic programming language, such as JavaScript. The content of user interface elements may be specified in a markup language, such as hypertext markup language (HTML) or XML User Interface Language (XUL). The layout of user interface elements may be specified in a style sheet language, such as Cascading Style Sheets (CSS). Alternatively, the interface104may be specified in one or more other languages, such as Java, Python, C, or C++.

Some examples of graphical user interfaces are described in further detail below.

In one or more embodiments, a tenant102is a corporation, organization, enterprise, or other entity that accesses a shared computing resource, such as the secure modular machine learning platform100. Multiple tenants may be independent from each other, such that a business or operation of one tenant is separate from a business or operation of another tenant. Some examples of multi-tenant architectures in accordance with one or more embodiments are described in further detail below.

2.6. Machine Learning

In one or more embodiments, a machine learning algorithm is an algorithm that can be iterated to learn a target model that best maps a set of input variables to one or more output variables, using a set of training data. The training data includes datasets and associated labels. The datasets are associated with input variables for the target model. The associated labels are associated with the output variable(s) of the target model. For example, a label associated with a dataset in the training data may indicate whether the dataset is in one of a set of possible data categories. The training data may be updated based on, for example, feedback on the accuracy of the current target model. Updated training data may be fed back into the machine learning algorithm, which may in turn update the target model.

A machine learning algorithm may generate a target model such that the target model best fits the datasets of the training data to the labels of the training data. Specifically, the machine learning algorithm may generate the target model such that when the target model is applied to the datasets of the training data, a maximum number of results determined by the target model match the labels of the training data. Different target models be generated based on different machine learning algorithms and/or different sets of training data.

The machine learning algorithm may include supervised components and/or unsupervised components. Various types of algorithms may be used, such as linear regression, logistic regression, linear discriminant analysis, classification and regression trees, naïve Bayes, k-nearest neighbors, learning vector quantization, support vector machine, bagging and random forest, boosting, backpropagation, and/or clustering.

3. Machine Learning

FIG.2illustrates an example set of operations for machine learning in accordance with one or more embodiments. One or more operations illustrated inFIG.2may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated inFIG.2should not be construed as limiting the scope of one or more embodiments.

In an embodiment, the platform receives a query (Operation202) to obtain some or all of the data available to a tenant. The platform may generate the query responsive to user input that defines one or more criteria for the query. The query may be for training data or scoring data (i.e., data to be scored using a trained model, such as production data).

The platform may determine which kind of data is requested in the query (Operation204). If the query is for training data, the platform may execute the query to obtain the training data (Operation206) and train the machine learning model (Operation208) using the training data. Specifically, the platform supplies training data to an algorithm/model used to train a machine learning model. Training generates a coefficient/weight matrix map, where each coefficient/weight is mapped to a respective feature of the data. The outcome of the trained model is saved into a model file, the format of which depends on the underlying language support. For example, if the underlying model is written in python, the “pickle” (PKL) format may be used. The platform may subsequently receive another query (Operation202) to obtain additional training data and/or scoring data.

If the query is for scoring data, the platform may execute the query to obtain the scoring data (Operation210). The platform may apply the machine learning model to the scoring data212and store the scoring results (e.g., a scoring file) produced by the model (Operation214). The platform may subsequently receive another query (Operation202) to obtain additional training data and/or scoring data.

4. Machine Learning Model/Profile Configuration

FIG.3illustrates an example set of operations for configuring a tenant's machine learning model/profile in accordance with one or more embodiments. One or more operations illustrated inFIG.3may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated inFIG.3should not be construed as limiting the scope of one or more embodiments.

In the example illustrated inFIG.3, the “tenant”302may refer to a user (e.g., an administrator) or process supplying data and/or instructions on behalf of a specific tenant. A tenant's model (referred to here as the BYOM model) may be a Dockerized machine learning algorithm that is configured to consume certain input data in a specified format or formats, process the data according to the packaged algorithm and providing outcomes—for example, a training model file (e.g., a PKL file) or scored model outcomes for a set of input attributes based on the model file.

To create a BYOM model/profile, the tenant302supplies instructions to an orchestrator304, via a system API, to create a BYOM profile (Operation308). The tenant302further provides a machine learning algorithm and/or other representation of a machine learning model (Operation310), an input mapping that maps outputs of the model to inputs of the platform (Operation312), and an output mapping that maps inputs of the model to outputs of the platform (Operation314).

Responsive to the information provided by the tenant302, the orchestrator304instructs the system306to create the attributes required to train or score the model (Operation316). The orchestrator304further instructs the system to create a destination (e.g., a cluster) for the model (Operation318) and an export job to obtain data for the BYOM container(s) (Operation320).

In an embodiment, responsive to the tenant302providing the model input attributes (which may correspond to the features in the model), the system306determines which of the output attributes correspond to predicted values. These output attributes may also be mapped to existing attributes.

FIG.4shows a block diagram that illustrates an example of a cluster configuration400in accordance with one or more embodiments. In one or more embodiments, the cluster configuration400may include more or fewer components than the components illustrated inFIG.4. The components illustrated inFIG.4may be local to or remote from each other. The components illustrated inFIG.4may be implemented in software and/or hardware. Each component may be distributed over multiple applications and/or machines. Multiple components may be combined into one application and/or machine. Operations described with respect to one component may instead be performed by another component.

As illustrated inFIG.4, the cluster configuration400includes: tenant-specific VCNs408A-N; tenant-specific clusters404A-N (e.g., using Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE)); and tenant-specific node pools402A-N. Each VCN408A-N is protected by firewall rules410A-N. Each cluster404A-N is further protected by another layer of firewall rules406A-N, including port restrictions.

In an embodiment, a system VCN412(e.g., an Oracle Data Sciences VCN) includes a system cluster414(e.g., an Oracle Data Sciences OKE Cluster), which hosts an orchestrator416. The system VCN412is configured to communicate unidirectionally with the tenant VCNs408A-N. In addition, the orchestrator416is configured to communicate with a data repository418(e.g., an Autonomous Transaction Processing (ATP) database system).

In an embodiment, each tenant's respective VCN408A-N is hosted on a respective physical machine (i.e., separate bare metal deployments), thus adding physical separation to logical separation.

FIG.5illustrates an example set of operations for deploying a set of untrusted code supplied by a tenant in accordance with one or more embodiments. One or more operations illustrated inFIG.5may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated inFIG.5should not be construed as limiting the scope of one or more embodiments.

Initially, a tenant502supplies the location of a container image and credentials for accessing the image (Operation510) to the platform, via a system API504. The platform stores the credentials, with encryption, in a repository508(Operation512).

When the system API504triggers an import job (Operation514), an automation server506requests credentials from the repository (Operation516) and receives the returned credentials (Operation518). The platform uses the credentials to import the image (e.g., a Docker image) (Operation520), and reports a status back to the repository (Operation522).

A successful import triggers a security scan of the image (Operation524). The platform scans the image (Operation526). Post-scan, the automation server506updates a status of the image (e.g., passed or failed the security scan) to the repository508(Operation528).

Responsive to a tenant502requesting a status via the system API504(Operation530), the platform queries the status from the repository508(Operation533), receives the status from the repository508(Operation534), and reports the status to the tenant502(Operation536).

7. Training a Machine Learning Model

FIG.6illustrates an example set of operations for training a machine learning model in accordance with one or more embodiments. One or more operations illustrated inFIG.6may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated inFIG.6should not be construed as limiting the scope of one or more embodiments.

In the example illustrated inFIG.6, the flow manager602refers to a component that is configured to handle orchestration and scheduling, and the pipeline orchestrator604refers to a component in the platform that is configured to handle data pipelines and orchestration for the data loads.

In an embodiment, the flow manager602exports a dataset query to the pipeline orchestrator604(Operation612). Responsive to receiving the query, the pipeline orchestrator604creates a dataset (Operation614) from the delimiter-separate values (DSV), exports the dataset to a repository608(e.g., an object store) (Operation616), and notifies the flow manager602that the data export is complete (Operation618).

The flow manager602deploys a training container to a training environment610(i.e., a set of components in the platform configured to perform training) (Operation620). The training environment610requests a dataset location from the server API606(Operation622) and receives the location in response (Operation624). The training environment requests the dataset from the provided location (Operation626), receives the dataset (Operation628), and performs training using the dataset as training data (Operation630). The training environment610uploads the resulting output (e.g., a pickle file) to the repository608(Operation632) and provides the location of the model to the server API (Operation634). The training environment610notifies the flow manager602that training is complete (Operation636).

8. Applying a Machine Learning Model

FIGS.7A-7Billustrate an example set of operations for applying a machine learning model in accordance with one or more embodiments. One or more operations illustrated inFIGS.7A-7Bmay be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated inFIGS.7A-7Bshould not be construed as limiting the scope of one or more embodiments.

In an embodiment, a tenant702triggers a scoring job, via the server API704(Operation714). The orchestrator704configures model deployment712(Operation716), if necessary, and model deployment712notifies the orchestrator704when the configuration is complete (Operation718). The orchestrator704then instructs the data platform706to start an export job (Operation720), and the data platform706exports the resulting file to a data repository708) (e.g., an object store). The data platform706notifies the orchestrator704when the export job is complete (Operation723). The orchestrator704starts the scoring pipeline710(Operation724), which requests the exported file from the repository708(Operation726) and receives the returned file (Operation728). The scoring pipeline710performs input mapping on the file (Operation730) and sends the transformed file to model deployment712(Operation732), which performs scoring on the transformed file (Operation734). Model deployment712stores the scoring output to the repository (Operation736). The scoring pipeline710requests the scoring output from the repository708(Operation740), receives the returned scoring output (Operation742), performs output mapping (Operation744), and writes the transformed output to data platform706's streaming API (Operation746). The scoring pipeline710notifies the orchestrator704that scoring is complete (Operation748), and the orchestrator704in turn notifies the tenant702(Operation750).

9. Example Graphical User Interfaces

FIGS.8A-8Oillustrate examples of graphical user interfaces (GUIs) in accordance with one or more embodiments. These examples are provided for illustrative purposes only and should not be construed as limiting the scope of one or more embodiments.

In the examples illustrated inFIGS.8A-8O, the machine learning platform is a software-as-a-service (SaaS) product made available via a web browser interface. The user is assumed to be an administrator associated with a particular tenant.

FIG.8Aillustrates an example of a dashboard GUI800that presents options for configuring and using machine learning on the platform. InFIG.8A, an “algorithms” tab802is currently selected, where the user can select from among various algorithms that use one or more machine learning models. A separate “models” tab804, when selected, allows the user to configure the model(s).

FIGS.8B-8Dillustrates an example of a GUI806for entering information about an algorithm. The screen shown inFIGS.8B-8Dis the first step (“details”) in a 4-step process that includes details, code, attributes, and configuration. As shown inFIGS.8B-3D, on the details screen, the user can enter a name808and description810of the algorithm, assign the algorithm to a family812, and indicate the algorithm's purpose814.

FIG.8Eillustrates an example of a GUI816for supplying untrusted code that uses a machine learning model. In this example, the user can indicate the location818(e.g., a URL) where a Docker container image can be found, as well as a username820and password822for accessing that location. In addition, an upload interface824allows the user to upload a JavaScript Object Notation (JSON) or other kind of file that defines a machine learning model. The model file may use a platform-agnostic schema that can be mapped to inputs and outputs of the algorithm.

FIGS.8F and8Gillustrates an example of a GUI826for mapping inputs828and outputs830, i.e., inputs to the algorithm and outputs of the algorithm resulting from scoring. In this example, the algorithm is configured to use a machine learning model to obtain a height estimate based on an individual's age and weight. InFIG.8G, the age input832and weight input834have been configured, and the height output836is in the process of being configured.

FIG.8Hillustrates an example of a GUI838for configuring parameters840and their allowable values for the algorithm, and hyperparameter tunings842for the machine learning model.

FIG.8Iillustrates an example of a GUI844for the “models” tab846, which in this example presents a list of available models.

FIG.8Jillustrates an example of a GUI848for creating a new model. The screen shown inFIG.8Jis the first step (“Details”) in a 5-step process that includes details, algorithm, query, mapping, and schedule (the latter being concealed by the tooltip850inFIG.8J). On the details screen shown inFIG.8J, the user can enter a name852and description854of the model.

FIG.8Killustrates an example of a GUI856for configuring parameters858of a model, in this case including a lookback window that defines the temporal scope of data to use in the model.

FIG.8Lillustrates an example of a GUI860, in which the algorithm tab862is selected, for selecting an algorithm to use with a model. As shown inFIG.8L, the algorithm is the set of untrusted code that uses the model.

FIG.8Millustrates an example of a GUI864, in which the query tab866is selected, for selecting one or more queries that obtain data for training the model and/or performing scoring using the trained model.

FIG.8Nillustrates an example of a GUI868for mapping inputs870(i.e., mapping attributes of the algorithm to query attributes) and outputs872(i.e., mapping outputs of the algorithm to system variables).

FIG.8Oillustrates an example of a GUI874for setting a schedule876for executing the algorithm that uses the model. In this example, the algorithm executes only on demand. Alternatively, a recurring schedule may be designated. In addition, the GUI874illustrated inFIG.8Oallows the user to provide one or more email addresses878to receive notifications. The user may select to receive notifications only when a failure condition is encountered.

10. Computer Networks and Cloud Networks

In a multi-tenant computer network, tenant isolation may be implemented to ensure that the applications and/or data of different tenants are not shared with each other. Various tenant isolation approaches may be used. Each tenant may be associated with a tenant identifier (ID). Each network resource of the multi-tenant computer network may be tagged with a tenant ID. A tenant may be permitted access to a particular network resource only if the tenant and the particular network resources are associated with the same tenant ID.

For example, each application implemented by the computer network may be tagged with a tenant ID, and tenant may be permitted access to a particular application only if the tenant and the particular application are associated with a same tenant ID. Each data structure and/or dataset stored by the computer network may be tagged with a tenant ID, and tenant may be permitted access to a particular data structure and/or dataset only if the tenant and the particular data structure and/or dataset are associated with a same tenant ID. Each database implemented by the computer network may be tagged with a tenant ID, and tenant may be permitted access to data of a particular database only if the tenant and the particular database are associated with the same tenant ID. Each entry in a database implemented by a multi-tenant computer network may be tagged with a tenant ID, and a tenant may be permitted access to a particular entry only if the tenant and the particular entry are associated with the same tenant ID. However, the database may be shared by multiple tenants.

In one or more embodiments, a subscription list indicates which tenants have authorization to access which network resources. For each network resource, a list of tenant IDs of tenants authorized to access the network resource may be stored. A tenant may be permitted access to a particular network resource only if the tenant ID of the tenant is included in the subscription list corresponding to the particular network resource.

In one or more embodiments, techniques described herein are implemented in a microservice architecture. A microservice in this context refers to software logic designed to be independently deployable, having endpoints that may be logically coupled to other microservices to build a variety of applications. Applications built using microservices are distinct from monolithic applications, which are designed as a single fixed unit and generally include a single logical executable. With microservice applications, different microservices are independently deployable as separate executables. Microservices may communicate using Hypertext Transfer Protocol (HTTP) messages and/or according to other communication protocols via Application Programming Interface (API) endpoints. Microservices may be managed and updated separately, written in different languages, and executed independently from other microservices.

Microservices provide flexibility in managing and building applications. Different applications may be built by connecting different sets of microservices without changing the source code of the microservices. Thus, the microservices act as logical building blocks that may be arranged in a variety of ways to build different applications. Microservices may provide monitoring services that notify a microservices manager (such as If-This-Then-That (IFTTT), Zapier, or Oracle Self-Service Automation (OSSA)) when trigger events from a set of trigger events exposed to the microservices manager occur. Microservices exposed for an application may alternatively or additionally provide action services that perform an action in the application (controllable and configurable via the microservices manager by passing in values, connecting the actions to other triggers and/or data passed along from other actions in the microservices manager) based on data received from the microservices manager. The microservice triggers and/or actions may be chained together to form recipes of actions that occur in optionally different applications that are otherwise unaware of or have no control or dependency on each other. These managed applications may be authenticated or plugged in to the microservices manager, for example, with user-supplied application credentials to the manager, without requiring reauthentication each time the managed application is used alone or in combination with other applications.

Microservices may be connected via a GUI. For example, microservices may be displayed as logical blocks within a window, frame, or other element of a GUI. A user may drag and drop microservices into an area of the GUI used to build an application. The user may connect the output of one microservice into the input of another microservice using directed arrows or any other GUI element. The application builder may run verification tests to confirm that the output and inputs are compatible (e.g., by checking the datatypes, size restrictions, etc.)

The techniques described above may be encapsulated into a microservice, according to one or more embodiments. In other words, a microservice may trigger a notification (into the microservices manager for optional use by other plugged-in applications, herein referred to as the “target” microservice) based on the above techniques and/or may be represented as a GUI block and connected to one or more other microservices. The trigger condition may include absolute or relative thresholds for values, and/or absolute or relative thresholds for the amount or duration of data to analyze, such that the trigger to the microservices manager occurs whenever a plugged-in microservice application detects that a threshold is crossed. For example, a user may request a trigger into the microservices manager when the microservice application detects that a value has crossed a triggering threshold.

A trigger, when satisfied, may output data for consumption by the target microservice. Alternatively or additionally, when satisfied, a trigger may output a binary value indicating that the trigger has been satisfied, and/or may output the name of the field or other context information for which the trigger condition was satisfied. Additionally or alternatively, the target microservice may be connected to one or more other microservices such that an alert is input to the other microservices. Other microservices may perform responsive actions based on the above techniques, including, but not limited to, deploying additional resources, adjusting system configurations, and/or generating GUIs.

A plugged-in microservice application may expose actions to the microservices manager. The exposed actions may receive, as input, data or an identification of a data object or location of data that causes data to be moved into a data cloud.

The exposed actions may receive, as input, a request to increase or decrease existing alert thresholds. The input may identify existing in-application alert thresholds and whether to increase, decrease, or delete the threshold. The input may request the microservice application to create new in-application alert thresholds. The in-application alerts may trigger alerts to the user while logged into the application or may trigger alerts to the user, using default or user-selected alert mechanisms available within the microservice application itself, rather than through other applications plugged into the microservices manager.

The microservice application may generate and provide an output based on input that identifies, locates, or provides historical data, and defines the extent or scope of the requested output. The action, when triggered, causes the microservice application to provide, store, or display the output, for example, as a data model or as aggregate data that describes a data model.

12. Hardware Overview

In one or more embodiments, techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing device(s) may be hard-wired to perform the techniques, and/or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination thereof. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, FPGAs, or NPUs with custom programming to accomplish the techniques. A special-purpose computing device may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other device that incorporates hard-wired and/or program logic to implement the techniques.

For example,FIG.9is a block diagram that illustrates a computer system900upon which one or more embodiments of the invention may be implemented. The computer system900includes a bus902or other communication mechanism for communicating information, and a hardware processor904coupled with bus902for processing information. The hardware processor904may be, for example, a general-purpose microprocessor.

The computer system900further includes a read only memory (ROM)908or other static storage device coupled to the bus902for storing static information and instructions for the processor904. A storage device910, such as a magnetic disk or optical disk, is provided and coupled to the bus902for storing information and instructions.

The computer system900may be coupled via the bus902to a display912, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device914, including alphanumeric and other keys, is coupled to the bus902for communicating information and command selections to the processor904. Another type of user input device is cursor control916, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processor904and for controlling cursor movement on the display912. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

The computer system900may implement techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware, and/or program logic which in combination with the computer system900causes or programs the computer system900to be a special-purpose machine. In one or more embodiments, the techniques herein are performed by the computer system900in response to the processor904executing one or more sequences of one or more instructions contained in the main memory906. Such instructions may be read into the main memory906from another storage medium, such as the storage device910. Execution of the sequences of instructions contained in the main memory906causes the processor904to perform the process steps described herein. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions.

Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires of the bus902. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency (RF) and infrared data communications.

Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor904for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line or other communications medium, using a modem. A modem local to the computer system900can receive the data on the telephone line or other communications medium and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus902. The bus902carries the data to the main memory906, from which the processor904retrieves and executes the instructions. The instructions received by the main memory906may optionally be stored on the storage device910, either before or after execution by processor904.

The computer system900also includes a communication interface918coupled to the bus902. The communication interface918provides a two-way data communication coupling to a network link920that is connected to a local network922. For example, the communication interface918may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface918may be a local area network (LAN) card configured to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, the communication interface918sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

The network link920typically provides data communication through one or more networks to other data devices. For example, the network link920may provide a connection through a local network922to a host computer924or to data equipment operated by an Internet Service Provider (ISP)926. The ISP926in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet”928. The local network922and Internet928both use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link920and through the communication interface918, which carry the digital data to and from the computer system900, are example forms of transmission media.

The computer system900can send messages and receive data, including program code, through the network(s), network link920, and communication interface918. In the Internet example, a server930might transmit a requested code for an application program through the Internet928, ISP926, local network922, and communication interface918.

The received code may be executed by processor904as it is received, and/or may be stored in the storage device910or other non-volatile storage for later execution.

In one or more embodiments, a non-transitory computer-readable storage medium stores instructions which, when executed by one or more hardware processors, cause performance of any of the operations described herein and/or recited in any of the claims.