CROSS-CLOUD RESOURCE MOBILITY OPTIMIZATION

Metrics, including cost and latency, corresponding to public computing system are monitored. A determination to migrate a workload running at a donor computing system may be made based on the monitored metrics. Before migrating the workload, state information corresponding to the workload may be stored to a remote storage that stores data used by the workload, but that is not part of the public computing systems. The recipient computing system may be instructed to retrieve the state information from the remote storage and to revive the workload based on the retrieved state information. The state information may comprise parameter values and factors that were current when the state information was stored. The state information may comprise an image of the workload. The recipient system may be instructed to revive the workload based on the retrieved state information retrieved from the remote storage.

BACKGROUND

The term ‘cloud’ may refer to a set, group, collection, or other plurality of computing resources, components, services, instances, collections, application, and the like that may be accessed by a computing resource, typically via a communication network (a communication network may also be referred to as a cloud). The term ‘cloud’ is typically used in reference to the computing resources without referencing specific items that make up the cloud resources when discussing computing functionality from the perspective of a computing resource that may make use of the functionality.

A cloud computing service provider may make available various computing resources as a service, for example, software, virtual machines, storage, bare metal computing hardware, or even a complete enterprise's infrastructure and development platforms, over a communication network. A cloud service provider may make a public cloud computing resource available to users over a publicly accessible network, such as the Internet. A private cloud computing resource is typically available or accessible only by a given customer, such as an enterprise and its employees. Computing resources may be provided from an enterprise's own on-premises data center or from a data center operated by an independent (e.g., independent from the enterprise customer) cloud services provider. A hybrid cloud may connect an organization's private cloud services and resources of public clouds into an infrastructure that facilitates the organization's applications and workloads in a manner that balances the maximizing of performance and the minimizing of costs across public and private cloud computing resources.

Cloud providers, whether providers of public or private computing resources, may use clustering of servers. A server cluster typically comprises servers that share a single Internet Protocol (“IP”) address. Clustering enhances data protection typically, availability, load balancing, and scalability. A server associated with a cluster may be referred to as a node, which may comprise a hard drive, random access memory, (“RAM”), and central processing unit (“CPU”) resources. In a hybrid cloud environment it is desirable for an organization to use resources of its private cloud as much as possible and use public cloud computing resources to handle spikes in usage demands that would exceed a determined limit, or a capacity, of the organization's private network. Moreover, it is desirable for an organization to optimize (e.g., minimize) costs related to use of public cloud resources. An organization's private cloud computing system, or systems, may comprise active components, modules, storage, services, and other resources that facilitate computing needs of the enterprise. An organization's private cloud computing resources may also comprise idle, or inactive, components, modules, storage, services, and other resources that are essentially held in reserve but are not used until workload increases require additional resources than the active resources already being used and paid for by the enterprise. The enterprise may subscribe to the computing resources of their private network from a computing resources provider/cloud computing provider instead of maintaining the resources and owning them outright. A provider may increase a subscription fee when an enterprise activates idle/inactive resources. The subscription may include a warranty cost associated with given active computing resources, such as storage, a processing components or instances, network bandwidth, and the like. Typically, the more a resource has been used, the more ‘wear’ has been placed on it. Thus, an organization desires to find an optimal balance of use of private and public cloud resources to maximize performance and to facilitate supporting computing workloads of the organization while minimizing costs for computing resources and services whether public or private.

An organization may have multiple providers of public computing resources (e.g., multiple providers of public cloud computing resources) to choose from in handling its workloads for which it uses public cloud resources. Pricing from one public computing provider for a given service may be less that pricing for the same service from another provider, but the pricing between the two providers may change with the previously more costly provider becoming the lower cost provider. However, costs to switch providers, especially costs to transfer large volume of data from one provider to another is prohibitively high.

SUMMARY

In an example embodiment, an example method may comprise monitoring, via a communication network, such as the Internet, by a first computing system comprising a processor, at least one metric corresponding to at least one of a group of computing systems that are coupled with, and that provide computing services via, the communication network. The first computing system may comprise an organization's private network, which may be operated by the organization at its own data center or may be operated for the organization at a data center that is not operated by the organization. The at least one metric may comprise cost, latency, or other information related to use by the organization of the at least one of the group of computing systems, which group may comprise public computing systems that the organization may use, or be able to use, to operate workloads instead of operating the workloads on their private computing system. The method may comprise analyzing, by the first computing system, the at least one metric with respect to a determined migration criterion, and in response to the at least one metric being determined to satisfy the determined migration criterion based on a result of the analyzing, initiating, by the first computing system, a migrating of a computing workload from a second computing system of the group of computing systems to a third computing system of the group of computing systems.

The determined migration criterion may comprise a cost element, or factor; a latency element, or factor; an amount of time for the migration to occur after initiation; and the like. For example, the migration criterion may be used to determine to migrate the workload to a third computing system that has a lower cost to the organization for operating the workload than is currently being paid by the organization to a provider of the second computing system. However, in an aspect, the criteria could also seek to optimize operating of the workload by one of the group of computing systems according to cost as well as latency. In a scenario, a given computing system of the group of computing systems may provide a lower operating cost for handling the workload than the currently used second computing system but if a latency value of a monitored metric associated with the given computing system is significantly higher than a latency associated with the currently used second computing system the migration criterion, or criteria, may not be satisfied. In other words, in the example scenario, using the migration criterion, or criteria, a determination may be made that an increase in latency may not be worth the reduced cost to the organization that corresponds to the workload.

The example method may further comprise storing pre-migration state information corresponding to a state of the workload at the initiating of the migrating of the workload to a remote computing system storage, which may be remote from the first computing system, the second computing system, and the third computing system; and causing the reviving of the workload at the second computing system with the pre-migration state information. The pre-migration state information may correspond to a ‘dirty state’ that corresponds to a state of the workload at the time of the initiating. The ‘dirty-state’ pre-migration state information may include parameter values, coefficients, factors, connection information, metadata, user log-in information, and other data produced by, used by, stored by, or otherwise associated with the workload at the time of the initiating, but may not include a complete copy of the full workload (e.g., the pre-migration state information may not include application code for running a workload virtual machine). In other words, code to start a new workload may not be part of the pre-migrations state information stored to the remote computing system storage, but the pre-migration state information may include information that may be used to configure a new workload virtual machine that is started at the third computing system such that the new workload virtual machine is operating on the third computing system as the previous workload was operating on the second computing system at the time of the initiating of the migration. The computing workload may comprise a computing instance, which may comprise a virtual machine. The computing workload may comprise an application. A workload may comprise an instance of a solution created to solve a problem, by using, configuring, or connecting multiple services offered by a cloud services provider, which may provide services corresponding to a public cloud or services corresponding to a private cloud. A set of services provided by a cloud services provider may comprise, for example, bare-metal computing resources as a service, virtual machine as a service, database as a service, identity as a service, to name a few. A typical cloud services provider may offer hundreds of computer-related services. Embodiments disclosed herein may enable or facilitate movement of one instance of a solution in one cloud to another cloud, which clouds, or corresponding cloud services, may be operated or provided by different services providers or by the same services provider.

The computing workload may be configured to access, and perform operations on, data that is stored by a storage of a fourth computing system that is coupled with the communication network, wherein the fourth computing system comprises the remote computing storage on which the pre-migration state information is stored. For example, the remote computing storage may comprise a storage located geographically close to a location of one or more of the group of computing systems that facilitates storage of data of the organization such that the data need not be migrated with the migration of a workload from one public computing system to another.

The determined migration criterion may be determined according to rules and/or factors. An individual user, such as an employee of the organization, may determine the rules or factors. For example, the rules or factors may be determined to always minimize cost to operate a workload. In another example, the rules or factors may be determined to minimize costs to operate the workload as long as latency in a computing system that a workload is migrated to, such as the third computing system, does not exceed a limit. An artificial intelligence algorithm may determine the criterion, or criteria, based on analyzing past migration activity when cost or latency metrics were considered and based on cost savings or latency changes achieved based on the past analyzing.

The rules and factors may be determined based on user input received by the first computing system, wherein the user input comprises information corresponding to the rules and factors entered or provided via an interface coupled with the first computing system.

The method may further comprise determining, by the first computing system, the rules and factors based on a log of migration operations performed by the first computing system before the analyzing of the at least one metric with respect to the determined migration criterion. An artificial intelligence model algorithm may have been trained using log, or history, information corresponding to previous migration activities, which may include having determined not to perform a migration if monitored metrics did not satisfy a migration criterion, or migration criteria. Thus, the artificial intelligence model algorithm may be trained not only on migration activities that were performed in the past along with corresponding past actual cost or latency improvements or degradations, but the artificial intelligence model algorithm may also be trained on past actual cost or latency improvements or degradations that occurred after monitored metrics were analyzed but a determination was made not to migrate a workload from a public computing system currently being used to another public computing system.

The at least one metric may comprise a spot market cost factor applicable to operation of the computing resource on the second computing system. A spot market may comprise an online, or electronic, platform operated for sellers that may currently use, or that may have purchased the right to use, computing resources of one or more public computing system providers—the sellers may effectively become resellers of public computing resources. Instead of using the computing system resources that have been purchased, the sellers may choose to sell the resources. The spot market may be in the form of an auction available to, or may be in the form of a direct offer from a seller to, an organization's private computing system, or a module that monitors metrics or that determines to migrate a workload. The module may monitor cost or latency metrics from a seller/reseller via the Internet and may automatically determine to migrate a workload based on analyzing of the monitored metrics from the spot market being analyzed according to the migration criterion/criteria.

The first computing system may output, via a user interface of the first computing system, an alert that the at least one metric has been determined to satisfy the determined migration criterion. An alert may be intended to notify personnel of the organization that a determination has been made by a module that has analyzed monitored metric data according to migration criterion/criteria. An alert may comprise a query that requests an approval input from an organization's personnel via a user interface that the migration may proceed, or the alert may merely inform an organization, personnel thereof, or just a module of a private computing system of an organization, that a determination has been made to migrate a workload.

In an example embodiment, an example system may comprise a first computing system, coupled to a communication network, that may comprise a processor configured to monitor, via the communication network, at least one metric corresponding to at least one computing system of computing systems that are coupled to, and that enable computing services via, the communication network. The computing systems of which the one computing system is a member, may comprise a public computing system to which an organization may migrate a workload. The processor may be configured to analyze the at least one metric with respect to a determined migration criterion; and in response to the at least one metric being determined to satisfy the determined migration criterion based on a result of the analyzing, initiate migration of a computing workload from a second computing system of the computing systems to a third computing system of the computing systems.

The processor may be further configured to store pre-migration state information corresponding to a state of the workload at the initiating of the migrating of the workload to a remote computing system storage that is remote from the first computing system, the second computing system, and the third computing system; and to cause the reviving of the workload at the second computing system with the pre-migration state information. The causing of the reviving of the workload may comprise sending a message to the remote computing system storage with an instruction to forward, or to permit the retrieval of, the pre-migration information to the third computing system to which the workload is being migrated. The pre-migration state information may comprise dirty state values that can be used to configure a new virtual machine at the third computing system or the pre-migration state information may comprise a full image of the workload before, or at the time of, the initiating of the migration of the workload.

The workload may be configured to access, or perform operations on, or process, an enterprise's data, which may be referred to as operational data or enterprise data, that is stored by a storage of a fourth computing system that is communicatively coupled with the communication network, and wherein the fourth computing system comprises the remote computing storage on which the pre-migration state information is stored. The enterprise's operational data that may be stored by the storage of the fourth computing system may comprise data that is not the pre-migration data. The enterprise's operational data may comprise databases, video content files, image content files, text content files, and other types of data. The enterprise's operational data may correspond to amounts or sizes of data such that if the operational data had to be migrated with a workload, such migration of the workload would likely not be cost beneficial to the organization corresponds to the workload if a cost associated with transferring the enterprise's operational data from a current computing system on which the workload is currently operating to a computing system to which the workload is to be migrated would exceed cost saving that might otherwise be realized by migrating the workload. Thus, the pre-migration state information may be stored to the same storage on which operational data that the workload uses is stored.

The determined migration criterion may be determined according to rules and factors. The processor may be further configured to present a user interface to receive one or more inputs from a user to determine the rules and factors or to enter the migration criterion or migration criteria. The processor may be further configured to determine the rules and factors based on a log of migration operations performed by the first computing system before the analyzing of the metric with respect to the determined migration criterion.

In an example embodiment, a non-transitory machine-readable medium may comprise executable instructions that, when executed by a processor of a computing device coupled with a communication network, facilitate performance of operations, comprising: monitor, via the communication network, at least one metric corresponding to at least one of a set of computing systems that are coupled with the communication network; analyze the at least one metric with respect to a determined migration criterion; and in response to the at least one metric being determined to satisfy the determined migration criterion based on a result of the analyzing, initiate migration of a computing workload from a second computing system of the set of computing systems to a third computing system of the set of computing systems.

The processor may be further configured to store pre-migration state information corresponding to a state of the workload at the initiating of the migrating of the workload to a remote computing system storage that is remote from the first computing system, the second computing system, and the third computing system; and cause the reviving of the workload at the second computing system with the pre-migration state information.

The workload may be configured to access, and perform operations on, data, such as an enterprise's operational data, that is stored by a storage of a fourth computing system that is coupled with the communication network and that may be geographically located within a defined distance of the second computing system, and wherein the fourth computing system may comprise the remote computing storage on which the pre-migration state information is stored. The pre-migration state information may comprise an image of the workload. The pre-migration state information may comprise state information corresponding to the workload at the initiating of the migration of the workload but not an image of the full workload at the initiating of the migration.

DETAILED DESCRIPTION OF THE DRAWINGS

As a preliminary matter, it will be readily understood by those persons skilled in the art that the present embodiments are susceptible of broad utility and application. Many methods, embodiments, and adaptations of the present application other than those herein described as well as many variations, modifications and equivalent arrangements, will be apparent from or reasonably suggested by the substance or scope of the various embodiments of the present application.

Accordingly, while the present application has been described herein in detail in relation to various embodiments, it is to be understood that this disclosure is illustrative of one or more concepts expressed by the various example embodiments and is made merely for the purposes of providing a full and enabling disclosure. The following disclosure is not intended nor is to be construed to limit the present application or otherwise exclude any such other embodiments, adaptations, variations, modifications and equivalent arrangements, the present embodiments described herein being limited only by the claims appended hereto and the equivalents thereof.

The term “facilitate” as used herein is in the context of a system, device or component “facilitating” one or more actions or operations, in respect of the nature of complex computing environments in which multiple components and/or multiple devices can be involved in some computing operations. Non-limiting examples of actions that may or may not involve multiple components and/or multiple devices comprise transmitting or receiving data, establishing a connection between devices, determining intermediate results toward obtaining a result, etc. In this regard, a computing device or component can facilitate an operation by playing any part in accomplishing the operation. When operations of a component are described herein, it is thus to be understood that where the operations are described as facilitated by the component, the operations can be optionally completed with the cooperation of one or more other computing devices or components, such as, but not limited to, sensors, antennae, audio and/or visual output devices, other devices, etc.

A monitor service may monitor components, resources, or workloads in a hybrid computing system environment. The monitor service can run on a SmartNIC or iDRAC or iSM or VxRail Manager or OpenBMC or any such controller of the computing system. The monitoring service may be one of multiple monitoring service with some of the services acting as clients of a master of the monitoring services, which master may act as a main interface for the others with respect to the computing system, such as may be used in a computing environment that comprises clusters.

Turning now to the figures,FIG.1illustrates computing system2, which may comprise a hybrid computing system. System2may comprise private computing system4operated by an enterprise6, which may be a business, a learning institution, a government agency, and the like. Private computing system4may communicate with one or more public computing systems8A-8n, which may be operated by corresponding cloud computing services providers10A-10n, via communication network12, which may comprise the Internet. Private computing system4may comprise a plurality of computing resources, for example processing resources18A-18n, storage resources20A-20n, and software resources22A-22n. Private computing system4may comprise other resources than processing, storage, and software, which are shown only for purposes of illustration and discussion. Public computing systems8A-8nmay comprise a plurality of computing resources, for example processing resources24A-24n, storage resources26A-26n, and software resources28A-28n. Public computing systems8A+8nmay comprise other resources than processing, storage, and software, which are shown only for purposes of illustration and discussion.

Turning now toFIG.2A, the figure illustrates system2with monitor36, which monitors usage parameters corresponding to public computing system resources8A-8nof public computing service providers10A-10n. The monitored parameters may comprise pricing for one or more different computing resources of computing systems8A-8nof providers10A-10n. The pricing information may be received via communication network12. In addition to receiving pricing information for various computing services from service providers10A-10n, pricing for services offered by providers10A-10nmay be received via communication network12from resellers, or from one or more spot market providers29that facilitate a market, including online, Internet or other electronic market type, for buying and selling public computing resources that have already been sold by providers10A-10n. Monitor36may comprise one or more computing services. In an embodiment, monitor36may comprise a service running on private computing system4. Monitor36may comprise a service running on a third computing system coupled with communication network12that is not private computing system4or one of public computing systems8A-8n, or the monitor may comprise a service running on one of public computing systems8A-8n.

Storage30may comprise storage resources, such as disc drives, solid state drives, tape drives, and the like, that are provided by a storage resources provider that is not one or public computing resource providers10A-10n. Storage30may be collocated at a computing data center that is operated for one of public computing resource providers10A-10n. Storage30may be located at a computing data center that is not operated by one of public computing resource providers10A-10n, but that is located geographically close to a data center operated by one of providers10A-10n. Storage30may facilitate an enterprise storing its enterprise information and operational data32separate from computing resources provided by public computing resource providers10A-10n. If enterprise6, or the enterprise's computing system4, determines to change from one of providers10A-10nto another of providers10A-10nfor the providing of computing resources that augment private computing system4, data32does not have to be transferred from the one public computing resources provider to the other.

Data egress may refer to moving, or transferring, data from one storage to another, including from a storage of one public computing resources provider to another public computing resources provider, or more generally, date egress may refer to the movement from a computing system or from a computing component to another computing system, or component thereof. Data egress costs from public cloud computing systems are usually high, not only in terms of time to reserve space at a target storage but also in terms of the actual time to transfer data from one storage to another, as well as in terms of bandwidth resources of a communication network used to transfer the data. The cost associated with moving data from one storage to another when a user, such as enterprise6, switches providers from one public computing resources provider to another may be prohibitively high insofar as any cost saving an organization my otherwise realize by changing public computing resources providers because differences in computing services pricing may be overwhelmed by data transfer, or data egress, costs. Thus, an organization may keep its data with the public computing provider even though the organization may be able to obtain better pricing with another provider. By an organization, such as enterprise6, storing its data32at third-party storage facility30(the storage is operated by a ‘third-party’ in the sense that storage30is not part of public computing systems8A-8n) the organization may change providers of computing resources dynamically, or almost ‘on-the-fly’, because data32stays at storage30and thus data egress costs are reduced, if not eliminated.

FIG.2Ashows workload34of enterprise6being currently (e.g., at time t0seconds) serviced by resources of public computing system8A provided by provider10A. The term ‘workload’ may refer to a computing application, service, instance, capability, or work that consumes computing resources (e.g., processing, memory, network access, or storage). Databases, tables, containers, microservices, VMs, bare-metal computing resources, including bare-metal computing resource configurations, which bare-metal resources may be offered as a service, and the like may be referred to as workloads. Workloads may comprise mobility solutions, office productivity applications, video conferencing solutions, disaster recovery solutions, analytics, or web content/hosting. Workload34is rendered inFIG.2Ain solid lines to show that the workload is currently being services by computing system8A. Workload34may be facilitated by containerized services, instances, applications, and the like that may be migrated, partially or fully, from one or more virtual machines (“VM”) running on computing system8A to one or more VMs running on any other system, or systems,8B-8n.

Turning now toFIG.2B, workload34is shown in broken lines at to and again in broken lines between to and a time after to that is to plus a transition period tp38. The phrase ‘migration period’ may be used to refer to transition period38. Workload34is shown unattached to a computing system8A-8ninFIG.2B. The transition, or migration, of workload34from computing system8A may have been determined to proceed because of computing system4electronically receiving pricing information corresponding to one or more of computing systems8B-8nfrom monitor36that indicates transitioning the workload away from computing system8A to another computing system8B-8nwould be advantageous for cost or for performance reasons.

FIG.2Cillustrates system2with workload34being rendered in solid lines and centered over, and connected to, provider10B and thus to its corresponding public computing system8B at time t0+tp. to indicate that the workload has been transitioned to use computing system resources from public computing system8B. Data32has not transitioned, or migrated, from storage30. Rather, data32remains at storage30and ‘follows’ workload34without moving from storage30as the workload migrates from donor computing system8A to recipient computing system8B as shown by the broken line connecting enterprise data block32and enterprise workload34at the workload's new, or recipient, public computing system8B.

It will be appreciated that during transition period38, which may be a few minutes, a few seconds, or less than one second, workload34may not be serviced by computing resources of either computing system8A or of computing system8B. For transitioned workload34to resume operating on resources provided by computing system8B, a pre-transition state, or pre-migration state, of the workload at time t0that may comprise settings, parameter values, instruction pointers, one or more steps of an application of the workload being executed, and the like, that existed before the transition from one computing system to the other began may be stored and transferred to new computing system8B so that the transitioned workload can ‘pick up where it left off.’ The terms ‘pre-migration’ and ‘pre-transition’ may be used interchangeably, and the terms ‘migration’ and ‘transition’ may be used interchangeably.

Turning now toFIG.3A, the figure illustrates donor computing system8A and recipient computing system8B arranged for a full VM transfer. At step A1subnet layer39connection between computing systems8A and8B may be established to facilitate transfer of one or more VMs40corresponding to enterprise6shown in previous figures from the current donor computing system8A to the target recipient computing system8B. After subnet layer connection39is established, VMs, which may include application code, may be transferred at step A2via the subnet layer from computing system8A to computing system8B. It will be appreciated that after a transfer begins, the one or more VMs40being transferred may not be serviced by a computing system and thus may be viewed as ‘dormant’. It will be appreciated that data32and storage30, as shown inFIG.2A,2B, or2C, may be connected to, or coupled with, VM40at donor system8A shown inFIG.3Abefore a transfer, but connected to the VM in recipient system8B after the transfer after which the recipient system facilitates execution of the VM.

It will be appreciated that in the case of a bare-metal computing system being migrated, it may not be physical components that are migrated from one location to another, but the use of computing components that may be transferred from one location to another, in which case item40inFIG.3Amay represent a configuration of a bare-metal server such that if donor bare-metal computing system8A is to be transferred and instantiated at recipient, or target, computing system8B, a configuration40representing a state of the donor system at the time of transfer maybe transferred to the recipient system and will be used to create on system8B the bare metal server as it was running on system8A when the configuration40was generated.

InFIG.3B, at step B1one or more pre-transition, or pre-migration, VM information state, or states,42corresponding to one or more VMs that were running on computing system8A at time t0is/are stored to storage30at step B2. A VM40may comprise a shell of a VM as well as a shell of an application that will need to be revived at target/donor computing system8B using information of the pre-transitions state42, corresponding to time t0, that is saved to storage30.

Pre-transition state information42is transferred to computing system8B from storage30at step B3and one or more shell applications of VMs40are revived using the pre-transition state information42at step B4, which may occur at or about time t0+tp. Pre-transition state information42may be referred to as ‘dirty-state’ information to indicate that although state information42was current as of time t0circumstances may change during tp, for example, data from sensors being monitored by a VM40at to, may change during tp. When a VM40is revived at step B4using state information42stored at step B2, VM will be revived to the state it was in at time t0when it was actively operating on donor computing system8A.

InFIG.3C, pre-transition application image44corresponding to the state of a workload to be transferred at time t0is moved from a VM40running on donor computing system8A at t0to storage30at step C1and then at step C2from storage30to the VM40at recipient computing system8B. Instead of just storing pre-transition state information42to storage30, a snapshot of a full VM, including an application, may be stored to storage30before beginning a transition of one or more VMs from computing system8A to computing system8B. After a new VM at recipient computing system8B has been started its state that was current at time t0may be revived using pre-transition application state information44, which may comprise an image, or snapshot, of an application running on computing system8A at to that is to be revived at recipient computing system8B. A revived VM40running on computing system8B may need to gather information to update itself to be current as of to +tp. To facilitate moving application state information44of a full VM, application code may be written to facilitate moving only a minimum of modules of the application that are needed for the application to start in the VM40at computing system8B. Thus, an application may be written so that application image44may use an amount of memory that only transfers essential information and data to facilitate the application being transferred to and run at donor computing system8B and so that connections and scripts are updated in the donor computing system8B. Such minimal, or ‘skinny’ code may optimize latency and cost in storing application image44to storage30and then to donor computing system8B.

Turning now toFIG.4, the figure illustrates a system50, which may comprise a software module51that further comprises other software modules that may interface with external public or private clouds/computing systems. Module51may comprise service provider interfaces modules61A-61nthat correspond to public computing resource service providers10A-10nas described in reference to other figures herein.

Module51may comprise monitor module52. Monitor module52make comprise monitor36as shown in other figures herein, or may receive information from monitor36such as, for example, cost information related to computing resource providers10A-10n. Monitoring module52may comprise a service that keeps track of a cost of holding and executing computational assets in the clouds of providers10A-10n, and consuming cloud-based services. Monitoring module52may operate under control of policy rules engine56, which may specify which assets and services to track (e.g., resources of cloud services providers10A-10n, how often to monitor or sample, how to compute costs per application, user, or time, etc., and how or when to report monitored information.

Module51may include a user interface54, which may provide a dashboard for a user, such as an employee of enterprise6, as shown inFIG.1, to use in evaluating, monitoring, establishing criteria, or reviewing of information relative to costs providers of public computing resources10A-10n. a user may use user interface54to enter criteria that may be used to determine when to transition a workload from one public computing resources provider to another. The criteria may be forwarded to a rules engine module56, which may apply the criteria, or other rules, to cost information received from monitor module52. For example, a cost value corresponding to a public computing resources provider may be received by rules engine56from monitor module52and may cause rules engine56to change, or adapt, automatically, a criteria that a user may have entered using user interface module54.

Rules engine module56may be programmed by the Financial Operations (“FinOps”) user of an enterprise using a policy specification language or other user interfaces via user interface module54. Development Operations (“DevOps”) users of an enterprise may specify restrictions on policies, based on an application's requirements. Other types of users, in addition to FinOps or DevOps, may user interface54to program rules engine module56. Rules engine module56may consider constraints, or criteria, and may create directives for the monitoring and reporting of cost information by a monitoring service facilitated by monitoring module52. If the monitoring leads to determining threshold events that trigger actions, the set of actions may be directed by the rules engine module56to be performed by Policy/Action Execution Engine58. The set of actions directed by rules engine56may comprise ‘recipes’, or programs, that execution engine58executes, under supervision and control of the rules engine56.

Rules engine56may forward criteria, or other rules, that have been determined based on information received from user interface54or based on information received from module52to the execution engine module58. Execution engine58may apply rules received from rules engine module56to cost information received from monitor module52and determine that a workload currently operating on a given public computing system should be migrated, partially or fully, to a different public computing system. Execution engine module58may forward a determination that a workload should be migrated, or transferred, from one public computing system to a different public computing system to migration engine module60to effectuate the migration, or transfer. Actions associated with workload asset and service migration from one public cloud to another, or to a private cloud, may be executed by execution engine module58. Rules engine56may provide actions to be executed by execution engine58at run-time, or may be policies that were defined to rules engine56before the monitoring by a monitoring service facilitated by monitoring module52begins. Actions to be executed by execution engine58may change with time, or with external events, for example a cloud outage/computing system outage of one of service providers10A-10n.

Migration engine module60may cause a snapshot of the workload or a current state of a VM running on the donor public computing system to be stored to storage30. Migration module62may manage establishing a subnet layer between the donor computing system and the recipient computing system for the workload to be transferred it may retrieve the workload snapshot and state information from storage30. Migration engine module60may combine and execute a logical program order for migration of cloud assets and services. Migration module60may also be involved in mapping the migration program parts into specific methods of migration/transfer of a workload, for example, a full VM transfer as described in reference toFIG.3A, a dirty state transfer as described in reference toFIG.3B, or an application state transfer as described in reference toFIG.3C.

Migration module62may forward the VM state information and application snapshot information to the donor public computing system. Storage control module63may provide an interface with a public cloud-adjacent storage service, such as storage30, such that migration module62interacts with module63without having to interface directly with storage30. Similarly, other services can also be implemented, or ‘abstracted,’ using service-specific interfaces. Migration module63may map specific cloud actions (e.g., actions that are specific to, or customized with respect to, a given service provider's10A-10ncorresponding public computing system8A-8n) to the respective cloud modules61A-61n, and keep tracks of the migration control plane state, thus facilitating an atomic migration action for workload assets and services specified by rules engine56. Atomicity facilitates completion of a migration/transition for a given computing system8A-8n, or a return to the pre-migration/pre-transition state, but typically not partial execution of a migration/transition of a workload.

Alerting module64may manage an alerting function of system50. Alerting module64may generate an alert message or cause an indication, either visual, audio, textual, and the like, to be provided to user interface54. Alert module64may cause an alert at user interface54such that a user of the interface is either aware that a transfer from a donor public computing system to a recipient computing system will automatically occur, has automatically occurred, or that a manual confirmation from the user is required to effectuate the transfer from the donor public computing system to the recipient public computing system. Thus, DevOps and FinOps personnel can receive alerts of critical and emergency actions, as well as alerts based on the individual alerts/notification preferences of the personnel.

History and logging module66may comprise functionality to captures a history of activity in system50, for example in a time-series database. A time-stamp record consisting of uniform log messages may be managed by module66. A log may be forwarded to an external control-plane system, another log-capture service (e.g., a SIEM), etc. Management of logging settings, logging preferences, logging expectations, and other logging rules may be conducted by module66.

History and logging module66may keep track of, log, or otherwise record transactions, such as, for example, cost information received by monitor module52, input information provided to rules engine56, user responses to alert messages, latency values associated with effectuating a transfer from one public computing resource to another upon a determination being made to do such a transfer, another transaction and information related to transferring a workload from one public computing system to another public computing system. History and logging information may be provided from history and logging module66to artificial intelligence module68. Artificial intelligence module68may use information received from history and logging module66to revise a learning model that may automatically revise rules and criteria of rules engine56without further human intervention or without input received at rules engine56from user interface module54. Storage70, which may comprise a database, may keep track of overall operation of system50, or module51, and may comprise history and logging information generated by history and logging module66, criteria in transactions related thereto received by rules engine module56from user interface54, cost information received at monitor module52and provided to rules engine56or execution engine58, or information related to migration, or transfer, of a workload, such as, for example, latency, time of day, time of week. Storage70may comprise relational and NoSQL database management systems, and provide atomicity, consistency, isolation, and durability (“ACID”) assurances. Storage70may maintain state information of the execution of system51with respect to transferring a workload from one public computing system to another, as well as provide failover, disaster recovery, and business continuity functionality.

Turning now toFIG.5, the figure illustrates a flow diagram of an embodiment method500to migrate a workload from a first public computing system to a second public computing system. Method500begins at step505. At step510a module monitors cost metrics received from multiple computing systems. The cost metrics may correspond to pricing for computer resources offered by one or more public computing service providers. In addition, costs metrics may include pricing for use of public computing resources that have been sold by the operator of the public computing resources and are being resold. At step515migration criteria may be received from a user interface module. For example, a user, such as an Information Technology (“IT”) employee of an enterprise, may enter a criterion or criteria for when a workload that the enterprise is currently being facilitated by a public computing system should be migrated to a different public computing system. The criteria may be used to evaluate cost metrics received from one or more public computing systems or from a reseller market as described in reference to step510. The criteria may also be used to evaluate latency relative to the public computing system that is currently being used to run the workload and a latency of another public computing system to which the workload may be migrated. The criteria may also be used to evaluate information relative to a transition period that may be associated with migrating the workload from the currently used public computing system to a different public computing system.

At step520migration rules are generated based on the migration criteria. At step525the rules that are based on the migration criteria received at step515are applied to the metrics monitored at step510to evaluate whether to migrate the workload from a public computing system currently being used to run the workload or to a different public computing system. At step530, if a result of the evaluation made it step525is that the workload is not to be migrated method500returns to step510and metrics continued to be monitored.

If, however, a determination is made at step530that a result of the evaluation performed at step525is that the workload should be migrated from a public computing system that is currently being used to operate the workload to a different public computing system, method500advances to step535. At step535a subnet layer connection may be established between the public computing system that is currently being used to operate the workload and a different public computing system to which the workload is to be migrated based on the evaluation made at step525.

At step537state information corresponding to the workload may be stored to a remote storage that may be operated at a data center that is geographically located proximate to the public computing system that is currently being used to operate the workload or proximate a public computing system to which the workload is to be migrated. Although the remote storage may be located graphically close to the current, donor, public computing system or geographically close to the recipient computing system the remote storage may nevertheless be a separate storage and may be operated by a different service provider then a service provider of the donor computing system or the recipient computing system. In an embodiment, the state information that may be stored to the remote storage may comprise state information corresponding to the workload such that if a new virtual machine that is to continue the workload at the recipient computing system is to be established the state information may be used to revive the new workload virtual machine at the recipient computing system. In an embodiment, the state information that may be stored to the remote storage may comprise an image, or snapshot, of the current workload as it existed at the time performing step537such that instead of beginning a new virtual machine and then reviving the workload by applying stored state information to the new virtual machine, an image of the workload, which may include an application and its associated executable code, parameters, data, coefficients, algorithms, factors, and other aspects of the workload may be used to revive the workload at the recipient public computing system when the public computing system retrieves the image of the workload that is stored at step537.

At step540a migration type, which may also be based on the evaluation performed at step525, may be retrieved. At step545the workload may be migrated according to the migration type and according to a determination at step525of a public computing system to which the workload is to be migrated. A more detailed discussion of the migrating of the workload is provided in reference toFIG.6. At step550the workload is revived as described above in reference to step537and is further described in reference toFIG.6. Method500ends at step555

turning now toFIG.6, the figure illustrates a flow diagram of an embodiment method545to migrate a workload from a donor computing system to a recipient computing system, and to revive the workload at the recipient computing system. Method545begins at step545as described in reference toFIG.5. At step605method545an execution module receives an instruction to migrate a workload from the donor computing system to the recipient computing system. The execution module may also receive a migration type, which may be to transfer the workload directly via a subnet layer between the donor computing system and the recipient computing system. The migration type may be a ‘dirty state’ transfer in which state information relative to the workload to be migrated has been stored to a remote storage but where a new virtual machine starts at the recipient computing system and is revived using the state information that is retrieved by the recipient computing system from the remote storage. The migration type may be an image migration where a snapshot is taken of the workload at the donor computing system before the migration begins and is stored to remote storage and then the recipient computing system retrieves the snapshot image of the workload and revives the workload based on the retrieved workload image.

At step610the execution module determines whether the migration type is to be a migration of a full virtual machine directly from the donor computing system to the recipient computing system. If the determination at step610is that a full VM migration is to be made, the workload is directly transferred via a subnet layer from the donor computing system to the recipient computing system in method545returns to step545as shown and described in reference toFIG.5.

If a determination made it step610is that a full virtual machine migration is not to be made method545advances to step620. At step620a determination is made whether a partial, or dirty state, migration is to be made. If a determination made at step620is that a dirty state migration is to be made method545advances to step625. At step625the recipient computing system is instructed to retrieve from a remote storage state information corresponding to the workload that was stored at step537as described in reference toFIG.5. At step630the recipient computing system may be instructed to begin a new virtual machine, retrieve dirty state information from the remote storage and revive the workload by executing the new virtual machine using the dirty state information retrieved from the remote storage. After step630method545returns to step545as described in reference toFIG.5.

If a determination made at step620is that a dirty state migration is not to be made, method545advances to step635. At step635the recipient computing system is instructed to read an image of the workload to be migrated from a remote storage. At step640the recipient computing system is instructed to revive the workload by running the workload based on the workload image retrieved at step635and method545returns to step545as described in reference toFIG.5. A migration type, or a migration type instruction, may be determined manually by a user using interface54, or a migration type may be determined automatically based complexity of a workload, how many connections or resources it may have to, or share with, other workloads, instances, application, processors, virtual machines, storage, and the like. If a workload is relatively simple with few external connections or interactions a determination may be made to perform a direct migration. For a complex workload with more connections a determination may be made that a migration type is to be a dirty state migration. A system such as system50, or a module, such as module52may determine a given migration type according to information and data acquired from monitoring the current workload and interconnections therewith.

With reference again toFIG.7, the example environment700for implementing various embodiments of the aspects described herein includes a computer702, the computer702including a processing unit704, a system memory706and a system bus708. The system bus708couples system components including, but not limited to, the system memory706to the processing unit704. The processing unit704can be any of various commercially available processors and may include a cache memory. Dual microprocessors and other multi-processor architectures can also be employed as the processing unit704.

Computer702further includes an internal hard disk drive (HDD)714(e.g., EIDE, SATA), one or more external storage devices716(e.g., a magnetic floppy disk drive (FDD)716, a memory stick or flash drive reader, a memory card reader, etc.) and an optical disk drive720(e.g., which can read or write from a CD-ROM disc, a DVD, a BD, etc.). While the internal HDD714is illustrated as located within the computer702, the internal HDD714can also be configured for external use in a suitable chassis (not shown). Additionally, while not shown in environment700, a solid-state drive (SSD) could be used in addition to, or in place of, an HDD714. The HDD714, external storage device(s)716and optical disk drive720can be connected to the system bus708by an HDD interface724, an external storage interface726and an optical drive interface728, respectively. The interface724for external drive implementations can include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are within contemplation of the embodiments described herein.

A number of program modules can be stored in the drives and RAM712, including an operating system730, one or more application programs732, other program modules734and program data736. All or portions of the operating system, applications, modules, and/or data can also be cached in the RAM712. The systems and methods described herein can be implemented utilizing various commercially available operating systems or combinations of operating systems.

Computer702can optionally comprise emulation technologies. For example, a hypervisor (not shown) or other intermediary can emulate a hardware environment for operating system730, and the emulated hardware can optionally be different from the hardware illustrated inFIG.7. In such an embodiment, operating system730can comprise one virtual machine (VM) of multiple VMs hosted at computer702. Furthermore, operating system730can provide runtime environments, such as the Java runtime environment or the .NET framework, for applications732. Runtime environments are consistent execution environments that allow applications732to run on any operating system that includes the runtime environment. Similarly, operating system730can support containers, and applications732can be in the form of containers, which are lightweight, standalone, executable packages of software that include, e.g., code, runtime, system tools, system libraries and settings for an application.

A monitor746or other type of display device can be also connected to the system bus608via an interface, such as a video adapter748. In addition to the monitor746, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

When used in a LAN networking environment, the computer702can be connected to the local network754through a wired and/or wireless communication network interface or adapter758. The adapter758can facilitate wired or wireless communication to the LAN754, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter758in a wireless mode.

When used in a WAN networking environment, the computer702can include a modem760or can be connected to a communications server on the WAN756via other means for establishing communications over the WAN756, such as by way of the internet. The modem760, which can be internal or external and a wired or wireless device, can be connected to the system bus708via the input device interface744. In a networked environment, program modules depicted relative to the computer702or portions thereof, can be stored in the remote memory/storage device752. It will be appreciated that the network connections shown are example and other means of establishing a communications link between the computers can be used.

When used in either a LAN or WAN networking environment, the computer702can access cloud storage systems or other network-based storage systems in addition to, or in place of, external storage devices716as described above. Generally, a connection between the computer702and a cloud storage system can be established over a LAN754or WAN756e.g., by the adapter758or modem760, respectively. Upon connecting the computer702to an associated cloud storage system, the external storage interface726can, with the aid of the adapter758and/or modem760, manage storage provided by the cloud storage system as it would other types of external storage. For instance, the external storage interface726can be configured to provide access to cloud storage sources as if those sources were physically connected to the computer702.

Turning now toFIG.8, the figure illustrates an example method800comprising at block805a method, comprising monitoring, via a communication network, by a first computing system comprising a processor, at least one metric corresponding to at least one of a group of computing systems that are coupled with, and that provide computing services via, the communication network; at block810analyzing, by the first computing system, the at least one metric with respect to a determined migration criterion; and at block815in response to the at least one metric being determined to satisfy the determined migration criterion based on a result of the analyzing, initiating, by the first computing system, a migrating of a computing workload from a second computing system of the group of computing systems to a third computing system of the group of computing systems. Embodiment method800may further comprise at block820storing pre-migration state information corresponding to a state of the workload at the initiating of the migrating of the workload to a remote computing system storage that is remote from the first computing system, the second computing system, and the third computing system; and at block825causing the reviving of the workload at the second computing system with the pre-migration state information. Embodiment method800may further comprise at block830wherein the computing workload is configured to access, and perform operations on, data that is stored by a storage of a fourth computing system that is coupled with the communication network, and wherein the fourth computing system comprises the remote computing storage on which the pre-migration state information is stored.

Turning now toFIG.9, the figure illustrates am example system900, comprising at block905a computing system comprising a processor configured to at block905monitor, via the communication network, at least one metric corresponding to at least one computing system of computing systems that are coupled to, and that enable computing services via, the communication network; at block910analyze the at least one metric with respect to a determined migration criterion; and at block915in response to the at least one metric being determined to satisfy the determined migration criterion based on a result of the analyzing, initiate migration of a computing workload from a second computing system of the computing systems to a third computing system of the computing systems. The processor may be further configured to at block920store pre-migration state information corresponding to a state of the workload at the initiating of the migrating of the workload to a remote computing system storage that is remote from the first computing system, the second computing system, and the third computing system; and at block925cause the reviving of the workload at the second computing system with the pre-migration state information. In addition, at block930the workload may be configured to access, and perform operations on, data that is stored by a storage of a fourth computing system that is communicatively coupled with the communication network, and the fourth computing system may comprise the remote computing storage on which the pre-migration state information is stored.

Turning now toFIG.10, the figure illustrates a method1000comprising at block1005a non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a computing device coupled with a communication network, facilitate performance of operations, comprising at block1005monitor, via the communication network, at least one metric corresponding to at least one of a set of computing systems that are coupled with the communication network; at block1010analyze the at least one metric with respect to a determined migration criterion; and at block1015in response to the at least one metric being determined to satisfy the determined migration criterion based on a result of the analyzing, initiate migration of a computing workload from a second computing system of the set of computing systems to a third computing system of the set of computing systems. The executable instructions may facilitate performance of operations by the processor further comprising at block1020store pre-migration state information corresponding to a state of the workload at the initiating of the migrating of the workload to a remote computing system storage that is remote from the first computing system, the second computing system, and the third computing system; and at block1025cause the reviving of the workload at the second computing system with the pre-migration state information. The executable instructions may facilitate performance of operations by the processor wherein at block1030the workload is configured to access, and perform operations on, data that is stored by a storage of a fourth computing system that is coupled with the communication network and that is geographically located within a defined distance of the second computing system, and wherein the fourth computing system comprises the remote computing storage on which the pre-migration state information is stored.